Deep active learning (AL) selects batches of instances for annotation to avoid retraining deep neural networks (DNNs) after each new label. Employing a naive top-b selection can result in a batch of redundant (similar) instances. To address this, various AL strategies employ clustering techniques that ensure diversity within a batch. We approach this issue by substituting the costly retraining with an efficient Bayesian update. Our proposed update represents a second-order optimization step using the Gaussian posterior from a last-layer Laplace approximation. Thereby, we achieve low computational complexity by computing the inverse Hessian in closed form. We demonstrate that in typical AL settings, our update closely approximates retraining while being considerably faster. Leveraging our update, we introduce a new framework for batch selection through sequential construction, updating the DNN after each label acquisition. Furthermore, we incorporate our update into a look-ahead selection strategy as a feasible upper baseline approximating optimal batch selection. Our results highlight the potential of efficient updates to advance deep AL research.

Every new material needs to be assessed and qualified for an envisaged application. A steadily increasing number of new alloys, designed to address challenges in terms of reliability and sustainability, poses significant demands on well-known analysis methods in terms of their efficiency, e.g., in X-ray diffraction analysis. Particularly in laboratory measurements, where the intensities in diffraction experiments tend to be low, a possibility to adapt the exposure time to the prevailing boundary conditions, i.e., the investigated microstructure, is seen to be a very effective approach. The counting time is decisive for, e.g., complex texture, phase, and residual stress measurements. Traditionally, more measurement points and, thus, longer data collection times lead to more accurate information. Here, too short counting times result in poor signal-to-background ratios and dominant signal noise, respectively, rendering subsequent evaluation more difficult or even impossible. Then, it is necessary to repeat experiments with adjusted, usually significantly longer counting time. To prevent redundant measurements, it is state-of-the-art to always consider the entire measurement range, regardless of whether the investigated points are relevant and contribute to the subsequent materials characterization, respectively. Obviously, this kind of approach is extremely time-consuming and, eventually, not efficient. The present study highlights that specific selection strategies, taking into account the prevailing microstructure of the alloy in focus, can decrease counting times in X-ray energy dispersive diffraction experiments without any detrimental effect on data quality for the subsequent analysis. All relevant data, including the code, are carefully assessed and will be the basis for a widely adapted strategy enabling efficient measurements not only in lab environments but also in large-scale facilities.

Retraining deep neural networks when new data arrives is typically computationally expensive. Moreover, certain applications do not allow such costly retraining due to time or computational constraints. Fast Bayesian updates are a possible solution to this issue. Therefore, we propose a Bayesian update based on Monte-Carlo samples and a last-layer Laplace approximation for different Bayesian neural network types, i.e., Dropout, Ensemble, and Spectral Normalized Neural Gaussian Process (SNGP). In a large-scale evaluation study, we show that our updates combined with SNGP represent a fast and competitive alternative to costly retraining. As a use case, we combine the Bayesian updates for SNGP with different sequential query strategies to exemplarily demonstrate their improved selection performance in active learning.

Active learning aims to reduce the amount of labeled data while maximizing machine learning models’ performances. Currently, there is sparse research on the potential of an optimal active learning strategy. Therefore, we propose a non-myopic oracle policy that accesses the true labels of the data pool to approximate an optimal active learning strategy. We evaluate how the hyperparameters of this oracle policy influence its performance and empirically demonstrate that it is an upper baseline for common active learning strategies while being faster than a state-of-the-art oracle policy. For the sake of reproducibility, all the code related to our research is publicly available on our GitHub repository at https://github.com/ies-research/non-myopic-oracle-policy .

Remote sensing through semantic segmentation of satellite images contributes to the understanding and utilisation of the earth's surface. For this purpose, semantic segmentation networks are typically trained on large sets of labelled satellite images. However, obtaining expert labels for these images is costly. Therefore, we propose to rely on a low-cost approach, e.g. crowdsourcing or pretrained networks, to label the images in the first step. Since these initial labels are partially erroneous, we use active learning strategies to cost-efficiently refine the labels in the second step. We evaluate the active learning strategies using satellite images of Bengaluru in India, labelled with land cover and land use labels. Our experimental results suggest that an active label refinement to improve the semantic segmentation network's performance is beneficial.

In transductive active learning, the goal is to determine the correct labels for an unlabeled, known dataset. Therefore, we can either ask an oracle to provide the right label at some cost or use the prediction of a classifier which we train on the labels acquired so far. In contrast, the commonly used (inductive) active learning aims to select instances for labeling out of the unlabeled set to create a generalized classifier, which will be deployed on unknown data. This article formally defines the transductive setting and shows that it requires new solutions. Additionally, we formalize the theoretically cost-optimal stopping point for the transductive scenario. Building upon the probabilistic active learning framework, we propose a new transductive selection strategy that includes a stopping criterion and show its superiority.

Support Vector Data Description (SVDD) is a popular one-class classifier for anomaly and novelty detection. But despite its effectiveness, SVDD does not scale well with data size. To avoid prohibitive training times, sampling methods select small subsets of the training data on which SVDD trains a decision boundary hopefully equivalent to the one obtained on the full data set. According to the literature, a good sample should therefore contain so-called boundary observations that SVDD would select as support vectors on the full data set. However, non-boundary observations also are essential to not fragment contiguous inlier regions and avoid poor classification accuracy. Other aspects, such as selecting a sufficiently representative sample, are important as well. But existing sampling methods largely overlook them, resulting in poor classification accuracy. In this article, we study how to select a sample considering these points. Our approach is to frame SVDD sampling as an optimization problem, where constraints guarantee that sampling indeed approximates the original decision boundary. We then propose RAPID, an efficient algorithm to solve this optimization problem. RAPID does not require any tuning of parameters, is easy to implement and scales well to large data sets. We evaluate our approach on real-world and synthetic data. Our evaluation is the most comprehensive one for SVDD sampling so far. Our results show that RAPID outperforms its competitors in classification accuracy, in sample size, and in runtime.

Analysing correlations between streams of events is an important problem. It arises for example in Neurosciences, when the connectivity of neurons should be inferred from spike trains that record neurons’ individual spiking activity. While recently some approaches for inferring delayed synaptic connections have been proposed, they are limited in the types of connectivities and delays they are able to handle, or require computation-intensive procedures. This paper proposes a faster and more flexible approach for analysing such delayed correlated activity: a statistical A nalysis of the C onnectivity of spiking E vents (ACE), based on the idea of hypothesis testing. It first computes for any pair of a source and a target neuron the inter-spike delays between subsequent source- and target-spikes. Then, it derives a null model for the distribution of inter-spike delays for uncorrelated neurons. Finally, it compares the observed distribution of inter-spike delays to this null model and infers pairwise connectivity based on the Pearson’s χ 2 test statistic. Thus, ACE is capable to detect connections with a priori unknown, non-discrete (and potentially large) inter-spike delays, which might vary between pairs of neurons. Since ACE works incrementally, it has potential for being used in online processing. In an experimental evaluation, ACE is faster and performs comparable or better than four baseline approaches, in terms of AUPRC (reported here), F1, and AUROC (reported on our website), for the majority of the 11 evaluated scenarios.

Machine learning applications often need large amounts of training data to perform well. Whereas unlabeled data can be easily gathered, the labeling process is difficult, time-consuming, or expensive in most applications. Active learning can help solve this problem by querying labels for those data points that will improve the performance the most. Thereby, the goal is that the learning algorithm performs sufficiently well with fewer labels. We provide a library called scikit-activeml that covers the most relevant query strategies and implements tools to work with partially labeled data. It is programmed in Python and builds on top of scikit-learn.

Stream-based active learning (AL) strategies minimize the labeling effort by querying labels that improve the classifier’s performance the most. So far, these strategies neglect the fact that an oracle or expert requires time to provide a queried label. We show that existing AL methods deteriorate or even fail under the influence of such verification latency. The problem with these methods is that they estimate a label’s utility on the currently available labeled data. However, when this label would arrive, some of the current data may have gotten outdated and new labels have arrived. In this article, we propose to simulate the available data at the time when the label would arrive. Therefore, our method Forgetting and Simulating (FS) forgets outdated information and simulates the delayed labels to get more realistic utility estimates. We assume to know the label’s arrival date a priori and the classifier’s training data to be bounded by a sliding window. Our extensive experiments show that FS improves stream-based AL strategies in settings with both, constant and variable verification latency.

Gathering labeled data to train well-performing machine learning models is one of the critical challenges in many applications. Active learning aims at reducing the labeling costs by an efficient and effective allocation of costly labeling resources. In this article, we propose a decision-theoretic selection strategy that (1) directly optimizes the gain in misclassification error, and (2) uses a Bayesian approach by introducing a conjugate prior distribution to determine the class posterior to deal with uncertainties. By reformulating existing selection strategies within our proposed model, we can explain which aspects are not covered in current state-of-the-art and why this leads to the superior performance of our approach. Extensive experiments on a large variety of datasets and different kernels validate our claims.

Despite the success of deep neural networks (DNN) in many applications, their ability to model uncertainty is still significantly limited. For example, in safety-critical applications such as autonomous driving, it is crucial to obtain a prediction that reflects different types of uncertainty to address life-threatening situations appropriately. In such cases, it is essential to be aware of the risk (i.e., aleatoric uncertainty) and the reliability (i.e., epistemic uncertainty) that comes with a prediction. We present AE-DNN, a model allowing the separation of aleatoric and epistemic uncertainty while maintaining a proper generalization capability. AE-DNN is based on deterministic DNN, which can determine the respective uncertainty measures in a single forward pass. In analyses with synthetic and image data, we show that our method improves the modeling of epistemic uncertainty while providing an intuitively understandable separation of risk and reliability.

In machine learning, active class selection (ACS) algorithms aim to actively select a class and ask the oracle to provide an instance for that class to optimize a classifier’s performance while minimizing the number of requests. In this paper, we propose a new algorithm (PAL-ACS) that transforms the ACS problem into an active learning task by introducing pseudo instances. These are used to estimate the usefulness of an upcoming instance for each class using the performance gain model from probabilistic active learning. Our experimental evaluation (on synthetic and real data) shows the advantages of our algorithm compared to stateof-the-art algorithms. It effectively prefers the sampling of difficult classes and thereby improves the classification performance.

Classifiers require annotations of instances, i.e., class labels, for training. An annotation process is often costly due to its manual execution through human annotators. Active learning (AL) aims at reducing the annotation costs by selecting instances from which the classifier is expected to learn the most. Many AL strategies assume the availability of a single omniscient annotator. In this article, we overcome this limitation by considering multiple error-prone annotators. We propose the novel AL strategy multi-annotator probabilistic active learning (MaPAL). Due to the nature of learning with error-prone annotators, it must not only select instances but annotators, too. MaPAL builds on a decision-theoretic framework and selects instance-annotator pairs maximizing the classifier's expected performance. Experiments on a variety of data sets demonstrate MaPAL's superior performance compared to five related AL strategies.

Heterogeneous domain adaptation adapts a machine learning model, here classification model, from a source domain to a target domain to leverage data from both domains. Thereby, supervised heterogeneous domain adaptation expects labeled data from the target domain, while unsupervised heterogeneous domain adaptation does not. In this article, we study the inclusion of active learning to bridge unsupervised and supervised domain adaptation. The active learning approach iteratively queries the most useful instances from the target domain, which are then labeled and used to improve the classification model. Using active learning, the selection of training instances can focus on areas where ambiguity in the source domain resolves in the target domain. Hence, we achieve the same performance with fewer labels. Experiments on real activity recognition data confirm our claims.

Classification algorithms aim to predict an unknown label (e.g., a quality class) for a new instance (e.g., a product). Therefore, training samples (instances and labels) are used to deduct classification hypotheses. Often, it is relatively easy to capture instances but the acquisition of the corresponding labels remain difficult or expensive. Active learning algorithms select the most beneficial instances to be labeled to reduce cost. In research, this labeling procedure is simulated and therefore a ground truth is available. But during deployment, active learning is a one-shot problem and an evaluation set is not available. Hence, it is not possible to reliably estimate the performance of the classification system during learning and it is difficult to decide when the system fulfills the quality requirements (stopping criteria). In this article, we formalize the task and review existing strategies to assess the performance of an actively trained classifier during training. Furthermore, we identified three major challenges: 1)~to derive a performance distribution, 2)~to preserve representativeness of the labeled subset, and 3) to correct against sampling bias induced by an intelligent selection strategy. In a qualitative analysis, we evaluate different existing approaches and show that none of them reliably estimates active learning performance stating a major challenge for future research for such systems. All plots and experiments are provided in a Jupyter notebook that is available for download.

Class assignment (label) is not the only information that can be queried from annotators. Additional feedback, in form of confidences, quantifies the reliability of the provided label. Multiple annotators classifying the same sample with different levels of expertise posses different levels of confidences. In this article, we discuss different types of confidence inspired by real-world application and discuss how they relate to each other. The α-confidence is inspired by humans, the β-confidence works with probabilistic outputs from intelligent machines (e.g., robots), and γ-confidence models non-normalized confidences. We consider uncertain, benevolent annotators, thus the provided labels can be contradictory. To overcome this problem, we propose two fusion strategies that combine the confidences from multiple uncertain annotators to a single one. Numerical and graphical evaluation indicates superior performance of our strategy compared to related strategies, namely, confidence weighted majority vote, c-Certainty, and a maximum likelihood estimation.

Nowadays, information and communication technology (ICT) has become a key driver for future health-enabling and ambient assisted living technologies. These future health-enabling living environments proactively anticipate the inhabitants' needs and adapt their behaviour accordingly. They further continuously monitor the behaviour of the inhabitants and may call in support in suspicious cases. In this article, we present an architectural blueprint for such a proactive living environment and highlight the corresponding challenges for research in the field. Afterwards, we present a simulation as experimental platform for learning the daily routine of inhabitants of a flat-sharing community of senior citizens. The experimental evaluation highlights that probably unusual behaviour of persons can be detected using a probabilistic approach, which may serve as an indicator for external support.

In the field of automated processes in industry, a major goal is for robots to solve new tasks without costly adaptions. Therefore, it is of advantage if the robot can perform new tasks independently while the learning process is intuitively understandable for humans. In this article, we present a highly automated and intuitive active learning algorithm for robots. It learns new classification tasks by asking questions to a human teacher and automatically decides when to stop the learning process by self-assessing its confidence. This so-called stopping criterion is required to guarantee a fully automated procedure. Our approach is highly interactive as we use speech for communication and a graphical visualization tool. The latter provides information about the learning progress and the stopping criterion, which helps the human teacher in understanding the training process better. The applicability of our approach is shown and evaluated on a real Baxter robot.

Machine learning systems learn from data. It is not rare that those data are grouped into categories (also called classes). Thus, one of the goals of a learning algorithm is to find out and understand how to assign new, previously unknown data to the correct class. But, in many cases data is available or can be gathered at low costs, whereas acquiring the corresponding category (or class) involves high costs. This is precisely where active learning (AL) comes into its own: In order to reduce the annotation costs, it allows the learning system to deliberately select the samples which should be annotated. Subsequently, the selected samples are presented to an entity (e.g., humans, simulation systems, etc.), generally addressed under the term oracle, that provides the corresponding classes. Afterwards, the knowledge base of the learner is updated and, depending on a stopping criterion, new labels are queried or not. Such a system is self-aware of its own imperfection, thus, it uses a selection strategy to determine the next most informative sample. Hitherto, it has been shown that AL is a powerful paradigm that evinces the desired results, provided that the oracles are omniscient. But, human oracles are prone to error, so for that reason we can ask ourself: Does AL still work with error prone and uncertain human annotators? In this article we present the results of an active learning case study conducted on 30 000 images labeled by two humans and propose a new type of labeling for better solving AL problems with error-prone annotators.