Given a sequence of p-values, conformal test martingales can be used for signaling that the exchangeability assumption is violated, while the false alarm rate is controlled by a user-specified significance level. In some scenarios, multiple p-values are observed at each time step, e.g., p-values may be received from multiple conformal predictors for a single target, or p-values are obtained for multiple targets. In such cases, signaling whenever a violation is detected for any of the sequences, leads to an increased risk of false alarms. Bonferroni correction, which is a standard approach to controlling the error rate when testing multiple hypotheses, is shown to be dominated by the straightforward approach of forming a single conformal test martingale from the martingales generated from the individual sequences of p-values. In addition to testing exchangeability for the individual sequences, approaches for testing them jointly are also investigated. For the latter, the use of aggregation operators to transform multiple sequences of p-values into a single sequence is investigated, as well as a previously proposed approach for detecting covariate shift. Experimental results are presented, highlighting the potential strengths and weaknesses of the different approaches.
Distribution-free uncertainty quantification is an emerging field, which encompasses risk control techniques in finite sample settings with minimal distributional assumptions, making it suitable for high-stakes applications. In particular, high-probability risk control methods, namely the learn then test (LTT) framework, use a calibration set to control multiple risks with high confidence. However, these methods rely on the assumption that the calibration and target distributions are identical, which can pose challenges, for example, when controlling label-dependent risks under the absence of labeled target data. In this work, we propose a novel extension of LTT that handles covariate shifts by directly weighting calibration losses with importance weights. We validate our method on a synthetic fraud detection task, aiming to control the false positive rate while minimizing false negatives, and on an image classification task, to control the miscoverage of a set predictor while minimizing the average set size. The results show that our approach consistently yields less conservative risk control than existing baselines based on rejection sampling, which results in overall lower false negative rates and smaller prediction sets.
Conformal prediction (CP) has gained increasing attention in machine learning owing to its ability to provide reliable prediction sets with well-calibrated uncertainty estimates. While most existing CP implementations focus on inductive conformal prediction (ICP), full conformal prediction-also known as online or transductive CP-offers the strongest validity guarantees but has been largely absent from open-source software due to its computational complexity. In this paper, we introduce online-cp, a Python package designed for online conformal prediction, conformal predictive systems (CPS), and conformal test martingales. The package implements several online CP algorithms, enabling efficient and principled uncertainty quantification in streaming data scenarios. Additionally, it includes tools for testing the exchangeability assumption by using conformal test martingales. We demonstrate the functionality of online-cp through classification and regression examples as well as applications to predictive systems and exchangeability testing. By making online CP methods accessible, online-cp provides a foundation for the broader adoption and further development of conformal prediction in real-time machine learning applications.
We address the problem of conformal prediction (CP) in the challenging setting of source-free domain adaptation (SFDA), where models must be calibrated using only unlabeled data from the target domain. Existing CP methods for domain shift rely heavily on labeled source data and importance weighting (IW), but we demonstrate that these approaches perform poorly in practice, even when source labels are available. As an alternative, we propose Source-Free Conformal Prediction (SFCP), a simple and effective method that replaces the unavailable target labels with pseudo-labels generated by the source model. We show both theoretically and empirically that, despite their inherent noise, these pseudo-labels can be reliably used to estimate conformal thresholds. Our method requires no access to source data and no hyperparameter tuning, making it particularly suitable for real-world SFDA scenarios. Experiments across more than 100 domain shifts demonstrate that SFCP achieves coverage levels comparable to oracle CP while consistently outperforming IW-based methods.
