A recurrent problem in many domains is the accurate and rapid detection of a change in the distribution of observed variables. This is important since our algorithms have been trained for a certain data distribution, and if the distribution has changed, the results will not be accurate and/or valid any longer. Instances of this problem, which are generally referred to as change-point detection, are found in fault detection in vehicle control systems, detection of the onset of an epidemic, and many other applications. Recently, new methods based on reliable machine learning have shown important advantages of this statistical task. Conformal Test Martingales (CTM) allow one to avoid this limitation and obtain valid results without information about used distributions. This is done with the assumption that the data are i.i.d. (or exchangeable) in online mode, and the corresponding martingale accumulates evidence against this assumption. This work aims to extend the conformal framework and consider the other family of reliable machine learning methods, the Venn-Abers method of probabilistic prediction, to test the data for change points. This work shows how Venn-Abers testing of exchangeability (VATE) can be founded on the ground of e-value theory, including recently developed e-pseudomartingales, and studies its advantages and drawbacks, compared to CTM. Our conclusion is that the efficiency of this approach is related to the type of causality in the data set.

There is a useful counterpart of conformal prediction for e-values, called conformal e-prediction. Conformal prediction can serve as basis for testing the assumption of exchangeability, leading to conformal testing. Similarly, conformal e-prediction can also serve as basis for testing exchangeability. The resulting conformal e-testing looks very different from but inherits some strengths of conformal testing; it even has some advantages over conformal testing. In this paper we discuss systematically both strengths and limitations of conformal e-testing.

In this paper we study the validity and efficiency of a conformal version of the CUSUM procedure for change detection both experimentally and theoretically.

The main objective of ensemble learning is to aggregate multiple models to better capture complex data distributions. Various ensemble techniques, including bagging and boosting, have been investigated and widely embraced in both research and practical applications. In this work, we enhance ensemble learning by incorporating distribution-free uncertainty inspired by conformal prediction. Conformal prediction allows us to quantify any model's uncertainty rigorously with valid coverage guarantees under lenient assumptions of the data distribution. We propose a novel ensemble learning framework called Distribution-Free Uncertainty-Aware Ensemble Framework (DUEn) for regression tasks which uses the information from distribution-free uncertainty in the form of intervals to benefit final point predictions and makes outputs more accurate and robust. Moreover, we propose a weighted interval agreement approach that aggregates base learners considering the degrees of uncertainty of their predictions. Experiments conducted on multiple data sets from different domains illustrate that DUEn is capable of enhancing the accuracy of regression by effectively using data while considering each base learner's distribution-free uncertainty.

Venn-Abers predictors offer a distribution-free probabilistic framework that generates calibrated predictions from the outputs of scoring classifiers, relying on minimal assumptions about the data distribution. This paper explores the extension of this framework from classification to regression, producing predictive distributions. We show how to evaluate the efficacy of the framework by comparing various metrics that assess the accuracy and informativeness of the predictions. We also show that the framework can be used for real-time prediction, using datasets from predictive maintenance and energy consumption forecasting.

In this work, we study applications of the Conformal Prediction machine learning framework to the questions of statistical data testing. This technique is also known as Conformal Test Martingales. Earlier works on this topic used it to detect deviations from exchangeability assumptions (such as change points). Here we move to test popular hypergraphical models. We adopt and compare two versions of Conformal Testing Martingales. First: testing the data against exchangeability assumption, but using the elements of hypergraphical model for setting its parameters. Second: combining Conformal Testing Martingale with Hypergraphical On-Line Compression Models. The latter is an extension of the Conformal Prediction technique beyond exchangeability. We show how these approaches help to accelerate the detection of data deviation from i.i.d. by making use of the knowledge about relations between the features embedded into a hypergraphical model.

A recurrent problem in many domains is the accurate and rapid detection of a change in the distribution of observed variables. This is important since the algorithms have been trained for a certain data distribution and if the distribution has changed, the results will not be accurate and/or valid any longer. Instances of this problem, which are generally referred to as change-point detection, are found in fault detection in vehicle control systems, detection of the onset of an epidemic and many other applications. It has been a subject of intensive research, with many publications in the statistical literature. Among well-known methods, there are CumulativeSum (CUSUM) and Shiryaev-Roberts procedures for the detection of changes. However, many of the methods would require complete or partial knowledge of the distribution of observed variables before and after the distribution has changed. Recent work in Conformal Testing and the introduction of Conformal Test Martingales (CTM) allows us to avoid this limitation and obtained valid results without information about used distributions Vovk et al. (2022). This is done in online mode with the assumption that data are exchangeable and the corresponding martingale accumulates evidence against this assumption. This paper considers an approach to the problem using the Venn-ABERS testing. It

Conformal prediction (CP) is a modern framework for reliable machine learning. It is most commonly used in the context of supervised learning, where in combination with an underlying algorithm it generates predicted labels for new, unlabelled examples and complements each of them with an individual measure of confidence. Conversely, association rule mining (ARM) is an unsupervised learning technique for discovering interesting relationships in large datasets in the form of rules. In this work, we integrate CP and ARM to develop a novel technique termed Conformal Association Rule Mining (CARM). The technique enables the identification of probable errors within a set of binary labels. Subsequently, these probable errors are analysed using another modern framework called Venn-ABERS prediction to correct the value in a probabilistic way.

Extended Abstract This poster presents the current state of our work in progress on development and application of Inductive Venn-Abers Predictive Distribution framework. As a sample task, we consider real-time household energy consumption forecasting problem. Concretely, the machine learning problem is to predict evening consumption (at 18:00) based on the morning consumption (0:00–12:00) on the same day. We use UCI public dataset on household power consumption (ECP) (1) to make prediction for the first 300 days. Only information from one particular household is used but in a fair real-time mode: in fair on-line mode: training on past days, not on the future. The advantages of the method are following. First, outputting well-calibrated predictions that are valid in weak assumptions. Second, that this way of regression gives a rich prediction in the form of the whole predictive distribution. It can be converted to a confidence interval of any probability, with possibility of flexible choice of its location (lower or upper ray, the interval with a given centre, or of the smallest length). Third, easy adaption to non-linear dependencies. In the previous poster (2) we also developed and demonstrated advantage of combining two ways of prediction by means of using bivariate isotonic regression as a merging tool. The results are shown in Tab. 1. We merge two versions of k-Nearest-Neighbours algorithms with different value of k . Our evaluation criterion is Continuous Ranked Probability Score (CRPS) that is the integrated difference between CDF of the true and the predicted distributions. Like earlier in

We continue study of conformal testing in binary model situations. In this note we consider Markov alternatives to the null hypothesis of exchangeability. We propose two new classes of conformal test martingales; one class is statistically efficient in our experiments, and the other class partially sacrifices statistical efficiency to gain computational efficiency.

The paradigm of Learning Under Privileged Information (LUPI) was used in various practical applications, including its combination with Conformal Prediction (CP) framework. In this note, we discuss possible sources and limitations of its efficiency. We try to argue that accuracy improvement coming from using privileged information is not occasional. For this goal, we consider some minimalistic models of LUPI where the contribution of the privileged information appears in its noise-free essence. Then, we discuss connection of LUPI paradigm and CP framework in relation with the models.

We argue for supplementing the process of training a prediction algorithm by setting up a scheme for detecting the moment when the distribution of the data changes and the algorithm needs to be retrained. Our proposed schemes are based on exchangeability martingales, i.e., processes that are martingales under any exchangeable distribution for the data. Our method, based on conformal prediction, is general and can be applied on top of any modern prediction algorithm. Its validity is guaranteed, and in this paper we make first steps in exploring its efficiency.

Conformal prediction provides a way of testing the IID assumption, which is the standard assumption in machine learning. A natural question is whether this way of testing is efficient. A typical situation where the IID assumption is broken is the existence of a changepoint at which the distribution of the data changes. We study the case of a change from one continuous distribution to another with both distributions belonging to standard parametric families. Our conclusion is that the conformal approach to testing the IID assumption is efficient, at least to some degree.

In this work we present a clustering technique called multi-level conformal clustering (MLCC). The technique is hierarchical in nature because it can be performed at multiple significance levels which yields greater insight into the data than performing it at just one level. We describe the theoretical underpinnings of MLCC, compare and contrast it with the hierarchical clustering algorithm, and then apply it to real world datasets to assess its performance. There are several advantages to using MLCC over more classical clustering techniques: Once a significance level has been set, MLCC is able to automatically select the number of clusters. Furthermore, thanks to the conformal prediction framework the resulting clustering model has a clear statistical meaning without any assumptions about the distribution of the data. This statistical robustness also allows us to perform clustering and anomaly detection simultaneously. Moreover, due to the flexibility of the conformal prediction framework, our algorithm can be used on top of many other machine learning algorithms.

Conformal predictive systems are a recent modification of conformal predictors that output, in regression problems, probability distributions for labels of test observations rather than set predictions. The extra information provided by conformal predictive systems may be useful, e.g., in decision making problems. Conformal predictive systems inherit the relative computational inefficiency of conformal predictors. In this paper we discuss two computationally efficient versions of conformal predictive systems, which we call split conformal predictive systems and cross-conformal predictive systems. The main advantage of split conformal predictive systems is their guaranteed validity, whereas for cross-conformal predictive systems validity only holds empirically and in the absence of excessive randomization. The main advantage of cross-conformal predictive systems is their greater predictive efficiency.

In this paper, we combine a modern machine learning technique called conformal predictors (CP) with elements of gestalt detection and apply them to the problem of visual perception in digital images. Our main task is to quantify several gestalt principles of visual reconstruction. We interpret an image/shape as being perceivable (meaningful) if it sufficiently deviates from randomness - in other words, the image could hardly happen by chance. These deviations from randomness are measured by using conformal prediction technique that can guarantee the validity under certain assumptions. The technique describes the detection of perceivable images that allows to bound the number of false alarms, i.e. the proportion of non-perceivable images wrongly detected as perceivable.

Some type of privacy-preserving transformation must be applied to any data record from Industrial Internet of Things (IIoT) before it is disclosed to the researchers or analysts. Based on the existing privacy models such as Differential Privacy (DP) and k-anonymity, we extend the DP model to explicitly incorporate feature dependencies, and to produce guarantees of privacy in a probabilistic form that generalize k-anonymity. We assume that additional (external) knowledge of these relations and models can be represented in the form of joint probability distributions, such as Mutual Information (MI). We propose an enhanced definition of DP in conjunction with a realisation for non-randomizing anonymizing strategies such as binning, reducing the extent of binning required and preserving more valuable information for researchers. This allows the formulation of privacy conditions over the evolving set of features such that each feature can be associated its own allowance for privacy budget. As a case study, we consider an example from the Industrial Medical Internet of Things (IMIoT). We have identified some challenges that are not completely addressed by existing privacy models. Unlike physiological measurements in conventional medical environments, IMIoT is likely to result in duplicate and overlapping measurements, which can be associated with different personally identifiable items of information. As an example, we present a model of sequential feature collection.

This paper reviews the checkered history of predictive distributions in statistics and discusses two developments, one from recent literature and the other new. The first development is bringing predictive distributions into machine learning, whose early development was so deeply influenced by two remarkable groups at the Institute of Automation and Remote Control. The second development is combining predictive distributions with kernel methods, which were originated by one of those groups, including Emmanuel Braverman.

Medical and general health-related measurements can increasingly be performed via IoT components and protocols, whilst inexpensive sensors allow the capturing of a wider range of parameters in clinical, care, and general health monitoring domains. Measurements must typically be combined to allow e.g. differential diagnosis, and in many cases it is highly desirable to track progression over time or to detect anomalies in care and general monitoring contexts. However, the sensitive nature of such data requires safeguarding, particularly where data is retained by different third parties such as medical device manufacturers for extended periods. This appears to be very challenging especially when standards-based interoperability (i.e using IoT standards like HyperCAT or Web of Things-WoT) is to be achieved. This is because open meta-data of those standards can facilitate inference and source linkage if compiled or analysed by adversaries. Therefore, we propose an architecture of pseudonimyised distributed storage including a dynamic query analyser to protect the privacy of information being released.

The aim of this work is to discuss abnormality detection and explanation challenges motivated by Medical Internet of Things. First, any feature is a measurement taken by a sensor at a time moment, so abnormality detection also becomes a sequential process. Second, an anomaly detection process could not rely on having a large collection of data records, but instead there is a knowledge provided by the experts.