Conformal Thinking: Risk Control for Reasoning on a Compute Budget – synesis

A risk-control framework sets reasoning stopping thresholds with finite-sample guarantees, adding a lower threshold that gives up early on likely-unsolvable problems.

Conformal Certification of Reasoning Trace Prefixes – synesis

A calibration layer turns any step-level risk score into the longest reasoning-trace prefix that provably contains no annotated error, with finite-sample marginal risk control under exchangeability.

Optimal prediction sets for plant identification: an interactive guide – Joseph Salmon

Insights and visualization of conformal prediction in the context of long-tailed classification. Challenges from citizen sciences platforms like Pl@ntNet

Uncertainty Estimation with Conformal Prediction – Michael Clark

More options for uncertainty estimation

17  Conformal prediction – Model to Meaning

Online conformal inference for multi-step time series forecasting – Rob J Hyndman

Week #1: Getting Started With Conformal Prediction For Classification

Using conformal prediction in just 3 lines of code

A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification

Black-box machine learning models are now routinely used in high-risk settings, like medical diagnostics, which demand uncertainty quantification to avoid consequential model failures. Conformal prediction is a user-friendly paradigm for creating statistically rigorous uncertainty sets/intervals for the predictions of such models. Critically, the sets are valid in a distribution-free sense: they possess explicit, non-asymptotic guarantees even without distributional assumptions or model assumptions. One can use conformal prediction with any pre-trained model, such as a neural network, to produce sets that are guaranteed to contain the ground truth with a user-specified probability, such as 90%. It is easy-to-understand, easy-to-use, and general, applying naturally to problems arising in the fields of computer vision, natural language processing, deep reinforcement learning, and so on. This hands-on introduction is aimed to provide the reader a working understanding of conformal prediction and related distribution-free uncertainty quantification techniques with one self-contained document. We lead the reader through practical theory for and examples of conformal prediction and describe its extensions to complex machine learning tasks involving structured outputs, distribution shift, time-series, outliers, models that abstain, and more. Throughout, there are many explanatory illustrations, examples, and code samples in Python. With each code sample comes a Jupyter notebook implementing the method on a real-data example; the notebooks can be accessed and easily run using our codebase.