Given a sequence of p-values, conformal test martingales can be used for signaling that the exchangeability assumption is violated, while the false alarm rate is controlled by a user-specified significance level. In some scenarios, multiple p-values are observed at each time step, e.g., p-values may be received from multiple conformal predictors for a single target, or p-values are obtained for multiple targets. In such cases, signaling whenever a violation is detected for any of the sequences, leads to an increased risk of false alarms. Bonferroni correction, which is a standard approach to controlling the error rate when testing multiple hypotheses, is shown to be dominated by the straightforward approach of forming a single conformal test martingale from the martingales generated from the individual sequences of p-values. In addition to testing exchangeability for the individual sequences, approaches for testing them jointly are also investigated. For the latter, the use of aggregation operators to transform multiple sequences of p-values into a single sequence is investigated, as well as a previously proposed approach for detecting covariate shift. Experimental results are presented, highlighting the potential strengths and weaknesses of the different approaches.
Distribution-free uncertainty quantification is an emerging field, which encompasses risk control techniques in finite sample settings with minimal distributional assumptions, making it suitable for high-stakes applications. In particular, high-probability risk control methods, namely the learn then test (LTT) framework, use a calibration set to control multiple risks with high confidence. However, these methods rely on the assumption that the calibration and target distributions are identical, which can pose challenges, for example, when controlling label-dependent risks under the absence of labeled target data. In this work, we propose a novel extension of LTT that handles covariate shifts by directly weighting calibration losses with importance weights. We validate our method on a synthetic fraud detection task, aiming to control the false positive rate while minimizing false negatives, and on an image classification task, to control the miscoverage of a set predictor while minimizing the average set size. The results show that our approach consistently yields less conservative risk control than existing baselines based on rejection sampling, which results in overall lower false negative rates and smaller prediction sets.
Conformal prediction (CP) has gained increasing attention in machine learning owing to its ability to provide reliable prediction sets with well-calibrated uncertainty estimates. While most existing CP implementations focus on inductive conformal prediction (ICP), full conformal prediction-also known as online or transductive CP-offers the strongest validity guarantees but has been largely absent from open-source software due to its computational complexity. In this paper, we introduce online-cp, a Python package designed for online conformal prediction, conformal predictive systems (CPS), and conformal test martingales. The package implements several online CP algorithms, enabling efficient and principled uncertainty quantification in streaming data scenarios. Additionally, it includes tools for testing the exchangeability assumption by using conformal test martingales. We demonstrate the functionality of online-cp through classification and regression examples as well as applications to predictive systems and exchangeability testing. By making online CP methods accessible, online-cp provides a foundation for the broader adoption and further development of conformal prediction in real-time machine learning applications.
We address the problem of conformal prediction (CP) in the challenging setting of source-free domain adaptation (SFDA), where models must be calibrated using only unlabeled data from the target domain. Existing CP methods for domain shift rely heavily on labeled source data and importance weighting (IW), but we demonstrate that these approaches perform poorly in practice, even when source labels are available. As an alternative, we propose Source-Free Conformal Prediction (SFCP), a simple and effective method that replaces the unavailable target labels with pseudo-labels generated by the source model. We show both theoretically and empirically that, despite their inherent noise, these pseudo-labels can be reliably used to estimate conformal thresholds. Our method requires no access to source data and no hyperparameter tuning, making it particularly suitable for real-world SFDA scenarios. Experiments across more than 100 domain shifts demonstrate that SFCP achieves coverage levels comparable to oracle CP while consistently outperforming IW-based methods.
A recurrent problem in many domains is the accurate and rapid detection of a change in the distribution of observed variables. This is important since our algorithms have been trained for a certain data distribution, and if the distribution has changed, the results will not be accurate and/or valid any longer. Instances of this problem, which are generally referred to as change-point detection, are found in fault detection in vehicle control systems, detection of the onset of an epidemic, and many other applications. Recently, new methods based on reliable machine learning have shown important advantages of this statistical task. Conformal Test Martingales (CTM) allow one to avoid this limitation and obtain valid results without information about used distributions. This is done with the assumption that the data are i.i.d. (or exchangeable) in online mode, and the corresponding martingale accumulates evidence against this assumption. This work aims to extend the conformal framework and consider the other family of reliable machine learning methods, the Venn-Abers method of probabilistic prediction, to test the data for change points. This work shows how Venn-Abers testing of exchangeability (VATE) can be founded on the ground of e-value theory, including recently developed e-pseudomartingales, and studies its advantages and drawbacks, compared to CTM. Our conclusion is that the efficiency of this approach is related to the type of causality in the data set.
We propose Conformal Good-Turing Classification (CGTC), a novel conformal inference method for classification tasks where the true label space is unknown or potentially infinite. Traditional conformal classification methods rely on the assumption of a finite and fully known set of labels (Vovk et al., 2005; Romano et al., 2020; Angelopoulos et al., 2021). However, this assumption is sometimes violated in real-world applications, such as image classification tasks with dynamic datasets, where new classes can continually emerge (Bendale and Boult, 2015; Scheirer et al., 2012). Consider, for example, a sequence (X-1, Y-1),..., (X-n, Y-n), (Xn+1, Yn+1) of facial images (X) with associated discrete identity labels (Y). In dynamic environments, new identities regularly appear, making it infeasible to predefine the entire label space. Consequently, applying traditional conformal methods in such contexts may lead to invalid coverage guarantees, as these methods fail to account for the possibility of encountering new, previously unseen labels. The fundamental challenge in classification tasks with unknown and potentially infinite label spaces lies in accurately determining whether a new observation corresponds to an existing class or represents a new class. To address this, we integrate conformal inference with hypothesis testing for novelties. Specifically, we formalize two null hypotheses: the "new-label" hypothesis H-0(new), testing whether an observation represents a previously unseen label, and the "old-label" hypothesis H-0(old), testing whether it belongs to an existing class. These two hypotheses form a mutually exclusive and exhaustive partition-every test observation must either belong to a new class or to one of the existing classes, but not both. Our decision rule operates as follows: If we reject H-0(new), we apply standard conformal classification methods and simply output the resulting prediction set. If we reject H-0(old), we conclude the observation belongs to an unseen label and output a special catch-all symbol indicating the presence of the new label explicitly. If we fail to reject both hypotheses at the chosen significance levels, we output the union of the standard prediction set and the catch-all symbol. This procedure maintains valid coverage guarantees. To test these hypotheses, we introduce novel conformal p-values based on the classical Good-Turing frequency estimator (Good, 1953). Thus, we name our framework Conformal Good-Turing Classification. We develop multiple variants of these conformal p-values, including feature-enhanced versions that leverage additional covariate information beyond mere frequency counts. Beyond the primary contributions, our framework addresses several practical challenges. First, we introduce a principled hyperparameter tuning strategy to optimally allocate the significance level between the classification and hypothesis testing components, enhancing prediction efficiency. Additionally, in settings with many rare labels appearing only once or a few times, random train-calibration splitting may result in some label classes appearing exclusively in the calibration set, rendering the conformal prediction set uninformative. We propose a selective splitting strategy to ensure each observed class is represented in the training set. While this breaks exchangeability, we design appropriate weights and prove that valid coverage is maintained. Empirically, we evaluate our method using both synthetic experiments and the CelebA dataset. For synthetic data, we generate labels from a Dirichlet Process (DP) with a concentration parameter. controlling the likelihood of new classes, and a uniform base distribution P-0 over [0, 1]. The Dirichlet Process is a canonical example of species sampling models, where labels are generated sequentially with predictive probabilities. This directly models our setting where new labels emerge dynamically. For each generated label Y-i = y, the corresponding feature vector Xi is sampled from a shifted multivariate normal distribution. For real-world evaluation, we use the CelebA dataset containing 202,599 face images of various celebrities representing 10,177 unique identities. We apply the MTCNN for face detection on the raw images and FaceNet to extract 128-dimensional feature embeddings. We then subsample a smaller group from the dataset and apply CGTC. Our experiments demonstrate that standard conformal classification methods fail to maintain the nominal coverage level when new labels appear. In contrast, CGTC successfully corrects the coverage to the target level while outputting more efficient prediction sets.
Large Language Models (LLMs) are increasingly deployed in real-world Natural Language Processing (NLP) systems to perform multi-label classification tasks, such as identifying multiple forms of toxicity in online content. However, most models output raw probabilities without an exact way to quantify uncertainty, increasing the risk of misclassification in high-stakes applications. In this work, we integrate Inductive Conformal Prediction (ICP) with the Binary Relevance (BR) approach to produce statistically valid prediction sets, label-wise. Using a modified Wikipedia Toxic Comments dataset, we evaluate this framework across varying significance levels (epsilon), incorporating calibration-set-aware thresholds to address label imbalances. Our results show that BR-based conformal prediction maintains valid marginal coverage while enabling flexible control over prediction set size (efficiency). Even in the presence of rare labels, the framework provides practical uncertainty estimates and where the prediction can be abstained in uncertain cases via empty sets. These findings support the feasibility of BR-ICP-based uncertainty calibration for scalable, interpretable automation in multi-label NLP systems.
Conformal prediction offers a principled framework for uncertainty quantification in classification tasks by outputting prediction sets with guaranteed error control. However, the interpretability of these set-valued predictions, and consequently their practical usefulness, remains underexplored. In this paper, we introduce a method for explaining conformal classification outputs using SHAP (SHapley Additive exPlanations), enabling model-agnostic local and global feature attributions for the p-values associated with individual class labels. This approach allows for rich, class-specific explanations in which feature effects need not be symmetrically distributed across classes. The resulting flexibility supports the detection of ambiguous predictions and potential out-of-distribution instances in a transparent and structured way. While our primary focus is on explaining p-values, we also outline how the same framework can be applied to related targets, including label inclusion, set predictions, and the derived confidence and credibility measures. We demonstrate the method on several benchmark datasets and show that SHAP-enhanced conformal predictors offer improved interpretability by revealing the drivers behind set predictions, thereby providing actionable insights in high-stakes decision-making contexts.
Robust conformal prediction is a model-agnostic technique designed to construct predictive sets with guaranteed coverage, assuming data exchangeability, even under adversarial attacks. Two primary strategies have been explored to address vulnerabilities to these attacks. The first strategy employs randomization, which is computationally efficient but fails to provide formal performance guarantees without resulting in overly conservative predictive sets. The second strategy involves formal verification, which restores coverage guarantees but leads to excessively conservative predictive sets and prohibitive computational overhead. Indeed, verification generally becomes NP-hard as it attempts to cope with attacks that are practically impossible, rendering some security claims unfalsifiable. In this paper, we propose a novel, provably efficient robust conformal prediction method by clearly defining a realistic threat model. Specifically, we assume explicit knowledge of the set of potential adversarial attacks, aligning our approach with standard certification procedures designed to certify against specific, identified threats. We demonstrate that attacks targeting the model can effectively be reframed as attacks on the score function, allowing us to recalibrate the score quantile to account for these known attacks and thereby restore desired coverage guarantees. It is worth noting that our approach allows to easily incorporate unknown or emerging (zero-day) attacks upon discovery, thus reestablishing coverage guarantees. By avoiding computationally intensive verification and operating under realistic threat assumptions, our approach achieves both efficiency and provable robustness. Empirical evaluations on real-world classification datasets and comparisons with state-of-the-art methods support the effectiveness and practicality of our proposed solution.
Ensuring reliable safety monitoring in autonomous driving systems (ADS) under uncertainty is essential for deployment in real-world scenarios. We propose the Temporal Multimodal Probabilistic Transformer (TMPT), a novel deep learning framework that integrates uncertainty quantification (UQ) into lane-keeping safety monitoring. TMPT forecasts lane deviation metrics along with calibrated aleatoric and epistemic uncertainties by processing sequences of multimodal sensor and control data. Our framework combines Transformer-based temporal fusion with deep ensembles and post-hoc calibration to improve predictive accuracy and uncertainty estimation. We evaluate 24 model variants in the CARLA simulator, analyzing the impact of architecture, calibration, and ensembling on both prediction and uncertainty. Calibrated models achieve near-perfect uncertainty reliability (ENCE < 0.03), while uncalibrated models show sharper predictions but overconfident errors. Ensemble methods further improve robustness but incur significant computational cost. Our findings show that aligning model selection with application context-balancing precision, calibration, and efficiency-is critical for safe and practical ADS deployment.
In many real-world settings, machine learning predictions serve as intermediate outputs used to inform decision-making. However, quantifying and accounting for uncertainty in these decisions remains a fundamental challenge. Conformal Predictive Decision Making is a framework for decision-making under uncertainty that leverages Conformal Predictive Distributions to optimize outcomes over a specified utility function. In this work, we evaluate Conformal Predictive Decision Making on synthetic datasets in both online and inductive settings, and compare its performance to two alternative approaches: Bayesian Decision Theory and Point Predictive Decision Making. Online Conformal Predictive Decision Making showed signs of greater robustness than Bayesian Decision Theory and Point Predictive Decision Making in scenarios involving noisy data and skewed utility functions, suggesting it may be a suitable option in more complex settings. However, it generally performed worse than the two alternative methods. In contrast, inductive Conformal Predictive Decision Making consistently outperformed the alternatives. This, combined with its computational advantages, makes it a promising approach for larger real-world decision-making applications where well-calibrated uncertainty quantification is needed for robustness.
yKnowledge graphs (KGs) have seen an increasing use in application domains where information may be deemed proprietary, protected, or sensitive, such as enterprise, medical, or security applications. For such systems, incorporating uncertainty quantification (UQ) is critically necessary when KG information is passed to others for any downstream usage. Moreover, such systems often have constraints on data availability due to safety or legal restrictions, and as such full access to well-labeled training data may be unavailable. Conformal prediction is a distribution-free UQ strategy which is well-equipped to handle both of these concerns, as it produces prediction sets with statistically valid guarantees and is highly compatible with black-box models, which may be shared more easily than training data. In this work, we develop a novel conformal framework for simultaneously detecting and classifying multi-hop relations between entities in a KG, which only assumes access to a pre-trained KG model over triples and does not require multi-hop training data. Our framework utilizes a greedy approach, wherein we use successive conformal predictors to build a sparsely-supported scoring function in the high-dimensional multi-hop relation space. In numerical experiments on publicly available benchmark KGs with variable size and multi-hop length, our conformal multi-hop relation sets offer substantial reduction relative to the multi-hop relation space.
When dealing with conformal prediction for real-world artificial intelligence applications, it is necessary to ensure its physical feasibility. In this work, we propose to tackle this problem for an autonomous wheelchair guided by a deep neural network for local navigation. We adapt the conformal sets to be compliant with the wheelchair's kinematics, enhancing their efficiency while preserving coverage guarantees.
Given the increasing usage of black-box Machine Learning models in high-risk scenarios such as clinical trials and fraud detection, a need for safe, robust and trustworthy machine learning solutions with reliable outcomes becomes all the more paramount. Uncertainty quantification in anomaly detection applications helps the cause of trustworthiness in non-parametric models used in One-Class classification. While ensembles and the sampling approaches can quantify uncertainty by learning on varied distributions of data and aggregating multiple predictions on test data, making the results more robust, statistical guarantees for Type-I Errors are not provided by ensembling and sampling techniques. This is where conformal prediction comes into play, providing statistical guarantees for controlling Type-I errors (false positives) below a user-specified error threshold, whilst not compromising on the Type-II errors (false negatives). This work proposes BaKC+, a novel approach for cross-conformal anomaly detection by combining K-fold cross-validation based cross-conformal prediction with ensembles and sampling techniques. BaKC+ proves to be a model-agnostic, distribution-free uncertainty quantification technique for highly imbalanced datasets, providing conformal guarantees for Type-I errors whilst showcasing high statistical power. Without additional post-hoc operations for Type-I error control needed, BaKC+ outperforms existing cross-conformal frameworks on benchmark anomaly detection datasets, and demonstrates itself to be a robust and reliable conformal anomaly detection framework, providing highly certain outcomes to the data analyst.
A major challenge in quantitative finance is not just predicting which stocks will outperform but quantifying the uncertainty and reliability of those predictions. This is critical because financial markets are inherently noisy, volatile, and affected by countless unpredictable factors, meaning that even the best models can be dramatically wrong (Virgilio and Paz Lopez (2024)). Reliable measures of uncertainty are essential for risk-aware investment decisions: they help portfolio managers judge when to trust a prediction, size positions appropriately, and avoid overconfidence that can lead to costly losses. Currently, most machine learning approaches for stock selection produce only point predictions (Gu et al. (2020)), offering no meaningful measure of confidence, which limits their practical value for investors who need to manage risk. Thus, in this paper, we benchmark classical and deep learning models for US stock selection (Fu et al. (2018)), and apply conformal prediction (CP) to generate well-calibrated prediction sets. Across all models, CP achieves empirical coverage closely matching the nominal confidence level, with most prediction sets being singletons.
Empirical, Archimedean and vine copulas have been repeatedly investigated and leveraged to infer conformal prediction regions for multivariate predictions, but they do not provide finite-size guarantees when the estimated copula is biased or misspecified. To address this limitation, we start with copula-based conformal prediction regions that are always nested and we leverage this property to counteract this copula-estimation bias, via an additional conformal re-calibration step. Furthermore, we introduce a simpler class of semi-parametric copulas (i.e., hierarchical Archimedean copulas) as an alternative to the more complex vine copulas for which incorporating prior knowledge is difficult. Using synthetic data sets, we compare biased and debiased copula-based conformal prediction methods, and we report the impact of the data size and the impact of the number of output dimensions. Using real data, we leverage prior knowledge via this simpler class of copulas. In these experiments, we observe that this additional re-calibration step effectively eliminates the estimation bias of empirical and semi-parametric copulas when its computations are precise (enough) and the data size is large enough. The debiased hierarchical Archimedean copulas yield performances that are comparable to the results of debiased vine copulas.
Conformal Prediction (CP) constructs prediction intervals with marginal coverage guarantees under the assumption of exchangeability, yet it has also been widely applied to non-exchangeable settings such as time series, where temporal dependence and distribution shifts often violate this assumption. Despite this, CP methods are typically evaluated using descriptive metrics like empirical coverage and average interval width, without formal statistical testing. This lack of hypothesis-driven evaluation makes it unclear whether deviations are meaningful or due to random variation. We address this gap by establishing a formal equivalence between CP and Value at Risk (VaR), enabling the use of VaR-style backtesting methods to statistically assess both marginal and conditional coverage. Additionally, we incorporate Diebold-Mariano tests with interval scores to compare predictive performance. Applied to synthetic, electricity, and financial time series, our framework uncovers violation and adaptation issues overlooked by standard metrics. The Dynamic Binary Test and Geometric Conformal Backtesting, in particular, identify covariate- and drift-induced dependence and miscalibration, offering a sharper lens for evaluating CP methods in non-stationary settings.
