The increasing availability of Artificial Intelligence systems in everyday life has drawn growing attention to the right to erasure, as defined by the General Data Protection Regulation. This right raises questions about how learned information can be removed from trained Artificial Intelligence models. In this paper, we examine the problem of machine unlearning from legal, philosophical and technical perspectives. We highlight the misalignment in terminology, as well as between legal expectations, which require data erasure at the individual level, and current technical methods, which typically operate at the class level. In addition, we explore the practical difficulties in evaluating the effectiveness of unlearning, particularly the limitations of using privacy attacks, such as Membership Inference Attack, as evaluation tools. Considering the several challenges in this setting, the lack of standardized metrics for unlearning, and the difficulty of assessing whether a model has truly forgotten specific data, we see the need for developing a unified evaluation framework at the European level. Such a framework should define robust, transparent, and reproducible criteria for assessing unlearning performance.
Learning coordinated behaviour in decentralised multi-agent reinforcement learning with sparse rewards presents significant exploration challenges. Whilst novelty bonuses encourage exploration, we identify a critical failure mode they create in sequential coordination tasks: coordination de-synchronisation, where agents repeatedly traversing earlier coordination points gradually exhaust their intrinsic motivation to revisit these critical locations. We hypothesize that the effectiveness of exploration strategies depends on two key task factors: coordination complexity and geometric revisit pressure. Our preliminary experiments confirm this: lifelong novelty bonuses deteriorate with increasing task complexity, while augmenting with episodic bonuses substantially improves performance. These findings motivate further theoretical investigation into coordination-aware intrinsic motivation for decentralized agents.
As the aging population grows, coupled with a shortage of healthcare personnel, the demand for innovative solutions becomes imperative. Digital tools, such as medicine dispensers, offer promising avenues for remote healthcare delivery, alleviating the workload on professionals. Nonetheless, home care organizations encounter challenges in implementing and scaling these tools, ranging from a lack of awareness about available options to difficulties in selecting the most suitable tool for specific situations. This study investigates a recommendation methodology for a medicine dispenser based on Omaha profiles from Electronic Patient Dossier (EPD). Using the CRISP-DM methodology, we designed a Positive-Unlabeled learning-based algorithm. We added Explainable Artificial Intelligence (XAI) techniques, showing a feature importance representation based on Shapley values, to enrich the transparency and reliability of suggested interventions. The solution was evaluated with healthcare professionals from two healthcare organizations. Although the technical performance of the algorithm was decent (recall: 0.9), they stated the data is not detailed enough to conclude whether a medicine dispenser could be used, showing the need for human evaluation during the process. This study addresses challenges like a sparse dataset lacking detailed data and iteratively involving users during development when performing research in a real life situation.
Causal effects vary across subgroups in a population. This is especially true for treatment effects in the medical setting where context information like family history, medical history and symptom presentation influence and explain differences between individuals. Estimating these local, causal treatment effects is not straightforward, especially since the covariate (context) information may i) interact with the outcome and/or treatment variable, ii) be measured at a lower hierarchical level than the level of the individual, and iii) contain high dimensional observations. In this paper, we perform two synthetic data experiments to further analyze and provide insights into the behavior of two types of non-confounding covariates: effect modifiers influence the treatment effect and prognostic factors influence the outcome variable, but not the treatment effect. We temporarily disregard the fundamental problem of causal inference that factual and counterfactual outcomes cannot be observed together, and generate synthetic data where treatment assignment variable W is considered a time-variant variable: the outcome value is observed for every occasion (treatment) for every patient. Consequently, we obtain a hierarchical dataset where outcomes (and possibly, covariates) reside at a lower hierarchical level than the treatment effects, which reside at the level of the individual. Our findings from controlled experiments provide 1) a demonstration of existing knowledge that prognostic factors reduce left-over variance and improve precision, whereas effect modifiers determine and explain individual-specific deviations, and 2) an additional insight that in order to unravel population heterogeneity, aggregations of lower-level, high-dimensional observations should align with (possibly unknown) expectations about whether these observations display prognostic or effect modification behavior.
We present a prototype real-time pipeline for anomaly detection and explanation in streaming time series data. The system consists of four core modules: time series storage, anomaly detection, model-agnostic explanation, and interactive visualisation. It supports multiple detection methods and produces variable-level contribution scores to explain real-time anomalies. An explainability score is introduced to quantify interpretability, aiding domain experts in understanding alerts. The pipeline updates both detection and explanation modules incrementally using a window-based approach. Preliminary results show reliable detection and intuitive explanations. Our pipeline is fully containerised and distributed to facilitate demonstration and deployment, enabling quick setup and reproducibility. Future work will focus on continuous learning to adapt to new data without forgetting past patterns.
Dealing with changes in data distributions is an important aspect of deploying machine learning based systems, including network intrusion detectors. While traditional drift detectors identify changes in feature distributions, they cannot reveal whether the decision boundaries of the model remain valid. This paper proposes leveraging Shapley Additive Explanation attributions to monitor shifts in the underlying feature-to-classification mapping. Using two NetFlow-based datasets (CIC-IDS2018 and UNSW-NB15), gradual multiplicative drift and an abrupt cross-dataset shift are simulated. For each drift scenario, raw feature distributions and Shapley Additive Explanations value distributions are compared via the Kolmogorov–Smirnov test and normalised Wasserstein distance. Results show that under pure scale drift, classification performance degrades and traditional detectors fire alarms; yet, the TreeSHAP explainability distributions remain stable. By differentiating drift types, the proposed explainability-based approach informs whether simple preprocessing adjustments suffice or if costly retraining is required.
Maritime vessel monitoring is vital for ensuring navigational safety, protecting marine ecosystems, and enforcing regulations. This work presents a framework to support expert analysis and monitoring of vessel activities using AIS trajectory data. By extracting rotation- and scale-invariant shape signatures through a relative Hough transform, our system clusters and organizes subtrajectory patterns, enabling intuitive visual exploration. Experts interactively associate representative shapes with maritime events such as trawling or port visits, creating an event-to-shape map used for real-time detection in new trajectories. The framework efficiently captures geometric and motion dynamics, supports interpretable, trustworthy AI through human-in-the-loop validation, and facilitates labeled data creation to enhance future automated analysis. We demonstrate its effectiveness with a case study using real-world data of fishing vessels collected from the ports of Britannia, highlighting its potential for scalable, human-in-the-loop maritime surveillance.
Satellite telemetry data consists of information transmitted from spacecraft providing their location, status, health and the functioning of the instruments. Due to the data volume, signal quality issues, resolution and orbital constraints, detecting anomalies in satellite telemetry data is challenging. One of the main challenges concerned with the existing methods is the large number of false alarms, contributing to a waste of operational resources and decreased trust in the system. In this paper, we tackle this issue and present Morphological Leave-One-Out Kernel Density Estimates (Mo-LOO-KDE)—an unsupervised anomaly detection method tailored for satellite telemetry data. It combines Extreme Value Theory, a statistical framework for modeling extremes, with computer-vision morphological operations. Here, Extreme Value Theory is used to model rare, extreme events, whereas morphological operations are effectively used for noise reduction. The unusual combination of Extreme Value Theory and morphological operations drive Mo-LOO-KDE’s low false positive rate. The proposed morphological filtering method has no formal requirement for regular sampling, so the method is applicable for irregular time series as well. Furthermore, the employment of a lightweight, unsupervised method such as Mo-LOO-KDE is advantageous due to its practical utility, as annotated and representative ground-truth datasets are expensive to generate and rarely available. Utilizing the 42-month telemetry dataset from the ESA-ADB benchmark, we perform a comparison with the SOTA approach and show that Mo-LOO-KDE outperforms other unsupervised methods, demonstrating its applicability for satellite telemetry.
Publicly available datasets play a crucial role in reliable, reproducible and responsible Machine Learning (ML) research and application. However, common benchmark datasets often suffer from data quality and other issues that affect model behavior. Auditing public datasets is, thus, a crucial step in understanding these data, allowing better models to be developed and improving understanding of their results. In this paper, we characterize popular healthcare datasets to illustrate how this can help practitioners select benchmarking data that align with their needs for responsible development of ML models. We conduct our study using 15 popular healthcare benchmarking datasets available on the UCI Machine Learning Repository. To assess their quality, we consider measures of accuracy, completeness and redundancy. To assess their complexity, we consider feature overlap, linearity, neighborhood, network, dimensionality, and class balance measures. We also inspect the datasets for imbalanced representation of different demographic groups. We found several data quality and demographic representation imbalance issues that indicate that careful analysis of public datasets remains needed.
As the number of satellites in orbit continues to increase, automated health monitoring of satellites has become increasingly critical. Given the complexity and diversity of satellite telemetry, it is essential to employ methods specifically tailored to these characteristics. A method named Quantile Gaussian Process Regression (Q-GPR) has been developed by the authors based on operational insights of actual satellite missions, but the method had not yet been evaluated using publicly available datasets. In this study, Q-GPR is evaluated by participating in the Spacecraft Anomaly Challenge, a widely open competition focused on algorithmic satellite anomaly detection held in early 2025. The competition utilized the ESA-ADB dataset, which comprises real satellite telemetry provided by the European Space Agency, enabling a realistic and practical evaluation environment. To accommodate the competition and dataset-specific requirements, Q-GPR was extended through adaptive learning and ensemble learning techniques. The proposed method outperformed all baseline methods included with the dataset. Notably, it demonstrated superior performance across diverse telemetry channels, highlighting its effectiveness and practicality for real-world satellite operations.
While AI has found its way into many areas of our daily lives, the black-box nature of most Machine Learning (ML) models raises concerns about trust and the identification of model limitations. Although refining ML models is essential for developing applications, improving data quality and model performance via explainable AI remains underexplored. This empirical comparative study addresses these gaps by diagnosing data quality and model performance issues using the Contrastive Explanations Method. We (1) developed a diagnostic framework for the analysis of explanations, (2) conducted a user study to evaluate the extent to which humans can identify these issues with the help of explanations, and (3) evaluated ML-based diagnosis of explanations using deep learning models. Humans interpreted explanations effectively, achieving mean diagnostic accuracies of up to 96.00
Active learning is a strategy designed to reduce annotation costs by allowing models to select the most informative samples from an unlabeled dataset, particularly in tasks like image classification. Integrated within the broader Human-in-the-loop framework, active learning creates an interactive process where human annotators play a role in labeling data. However, this interaction introduces variability in annotation quality, since annotators may differ in domain knowledge, experience, and reliability. Traditional active learning approaches often overlook these differences, assuming constant performance across all annotators. This assumption can lead to suboptimal model updates, especially when labels come from less reliable sources. To tackle this limitation, this work proposes a reputation-based framework that captures annotator performance over time and across domains. Reputation scores are computed based on past annotation accuracy and feedback from other annotators, and these scores are then used to weigh individual contributions through a voting mechanism. In addition, the study explores how annotator expertise and oracle size influence the effectiveness of this approach, evaluating their impact on model performance in controlled, simulated settings with imperfect annotators.
Sequence alignment is a powerful technique to compare and analyze sequential data, with applications in bioinformatics and signal processing. Traditional multiple sequence alignment (MSA) methods have a number of limitations that make them unsuitable for applications such as apnea detection in sleep medicine. First, the sequences need to be numerical or they are one-dimensional sequences of events. Apnea detection requires patterns across multiple dimensions of event sequences. Second, existing approaches are based on pairwise alignments. This puts too much focus on the alignment of all possible events in a pair of sequences. In apnea detection, however, the sequences contain many extraneous events, leading to a sparse set of relevant events. Furthermore, relevant events only manifest themselves when considering all sequences jointly. We present a novel multiple sequence alignment method that lifts these limitations. Our approach iteratively moves events across all sequences simultaneously to group identical events together. The direction in which to move events is based on the insight that event occurrences represent a density function and that a set of aligned sequences represents a sharply peaked density function. New sequences can be warped to a reference pattern using this density function. The efficacy of our new alignment method is shown in the apnea and hypopnea application.
Glaucoma is a leading cause of irreversible blindness worldwide, often progressing undiagnosed due to asymptomatic early stages and limited access to specialist care. To address these barriers, we present a feasibility study of Glaucoma-PAIR (Privacy-preserving AI-based Referral), a computer-aided system developed using federated learning (FL). The study was conducted across a multi-centric network of three heterogeneous clinical sites in Portugal, two tertiary large public hospitals and one private clinic, each with distinct patient demographics, imaging equipment, and data distributions. The system leverages color fundus photography and expert-labeled cases to train a glaucoma classification model, without transferring sensitive patient data across institutions by employing FL, ensuring compliance with institutional governance and data protection regulations. Our work addresses major challenges in clinical AI, including privacy, generalizability, and integration into real-world workflows. Through close collaboration with ophthalmologists, we identified key constraints in existing referral pathways and incorporated those insights into the study design. Notably, the federated global model achieved performance comparable to a centralized model trained on pooled data, improved the average sensitivity which is a critical metric for a screening tool, and showed significant performance gains at the most clinically diverse site. This study provides a practical demonstration of responsible machine learning, combining privacy-preserving operations with clinical feasibility. Our findings highlight the potential of federated learning to enable the development of scalable and equitable AI tools, to support patient triage for a glaucoma specialist, particularly in settings with limited ophthalmology accessibility, promoting access to earlier diagnosis and care. We discuss the implications for future deployment and integration into national screening workflows.
The labeling process for supervised learning is costly and time-consuming, and is often impractical to scale due to real-world constraints. Active learning (AL) addresses this challenge by strategically selecting representative and informative data points to reduce labeling efforts. This paper focuses on an AL scenario in which only a very limited number of labels can be acquired. We propose an algorithm operating in two phases: (1) an exploration phase that prioritizes representative and diverse data points using density-driven criteria, and (2) an exploitation phase that combines predictive uncertainty with density weighting to select informative samples from densely populated regions. This enhances both representativeness and informativeness. Our results demonstrate significant improvements in model quality compared to other algorithms typically employed for this scenario, across various scenarios involving imbalanced data in classification tasks and skewness in regression tasks. Through this work, we aim to provide a new algorithm for this scenario and investigate general principles for AL. While most AL studies focus on either classification or regression, our work applies the algorithms to both. Therefore, we can analyze the differences between classification and regression problems and their effects on AL strategies. Furthermore, we explore different categories of AL criteria and their effectiveness in the low-budget regime. These results also provide insight into the cold-start problem, which involves selecting an initial labeled set and is faced by many model-based AL methods.
Anomaly detection is critical in industrial domains such as quality control and predictive maintenance, where combining it with visual inspection and explainability enhances trust and reduces errors. This study evaluates a pre-trained multimodal foundation model for visual anomaly detection and explanation on the MVTec dataset, using a post hoc fusion strategy that integrates outputs from independent models. The setup includes comparisons with PatchCore and extended configurations incorporating metadata such as heatmaps, segmentation masks, and patches. Results show that the multimodal model outperforms PatchCore on texture categories (mean F1: 0.960 vs. 0.947), but underperforms on object categories. It offers interpretable explanations in simple cases, but limited classification accuracy, marginal benefits from metadata augmentation, and reduced specificity in complex scenes indicate that further refinement is needed. These findings highlight the potential of multimodal models for explainable anomaly detection while underscoring current limitations in handling complex object structures.
Concept drift detection is a critical component of most online learning algorithms. While the primary goal of concept drift detection techniques is to trigger the retraining of machine learning models, in high-risk domains such as industry and healthcare “where models serve as decision-support systems, with final decisions made by humans” it is essential to understand the nature of the drift, its potential causes, and its impact on model performance. In this paper, we propose a 3P (three-perspective) approach for efficiently detecting concept drifts and analyzing their nature from three perspectives: data characteristics, model performance, and explainability. Unlike traditional approaches that merely identify the point in time where drift occurs, our approach not only detects drift but also provides deeper context by integrating insights from data, model behaviour, and XAI (explainable AI). This enriched perspective enables a more comprehensive understanding of drift dynamics, facilitating informed decision-making. Additionally, by incorporating explainability-driven analysis, our approach allows for detecting drifts that may not be identifiable through model performance alone, as with most conventional detectors. We integrate this three-perspective analysis with visualization techniques and demonstrate its effectiveness on benchmark datasets and two real-world use cases from healthcare and industry.
Federated learning (FL) has emerged as a promising approach to facilitate collaborative model training while ensuring data privacy. FL faces challenges related to heterogeneity in data distributions, interpretability of model decisions, and optimization of model architectures across decentralized nodes. This paper proposes a framework that combines FL with neural architecture search (NAS) and explainable large language model (XLLM) to overcome these issues and improve clinical outcomes. We test this approach in three medical areas. NAS is used to discover optimized model architectures tailored to heterogeneous medical data across decentralized hospitals. XLLMs are employed to interpret and communicate complex decision-making processes. Experimental validation on benchmark datasets for each clinical use case indicates improvements in predictive accuracy and clinical relevance compared to conventional federated approaches.
The paper aims to use fundus images to determine the patient’s sex. Admittedly, such a possibility has been confirmed by previous studies, but the paper will consider how transfer learning from the RETFound foundation model for retinal images dedicated to the diagnosis of diabetic retinopathy can be used to build such a system and whether the quality of the transfer is sufficient when analyzing data from a different population than the one used to build the fundus model. The study presented here is based on a non-public dataset of fundus images collected from individuals of Polish ancestry. Our study shows that transfer learning, a proven tool for applying and achieving better results when reusing models tuned for different tasks, has some limitations when applied to medical data. In addition, we address emerging concerns about data leakage in medical imaging, as such data may contain overt patient metadata and hidden patterns invisible to humans, which may encode sensitive information that is impossible to capture even by medical experts aided by Explainable artificial intelligence (XAI) techniques.
Recent advances in Large Language Models (LLMs) have significantly transformed conversational AI, enabling their integration into sensitive domains such as healthcare. The growing integration of LLMs with search engines has accelerated this shift by replacing traditional symptom lookups with conversational queries. A notable development is the increasing trend in medical self-diagnosis by the general public, which raises critical concerns regarding accuracy, fairness, and responsible deployment. Unlike professional diagnoses conducted by clinical experts, self-diagnosis often depends on incomplete or subjective symptom descriptions, further complicated by unrestricted access to LLM-generated outputs. This study evaluates the feasibility and risks associated with public LLMs for self-diagnosis, focusing on prompt engineering strategies, demographic bias, and the application of Retrieval Augmented Generation (RAG) to enhance reliability. Through a comparative analysis of 10,000 synthetic patient cases across LLMs, we discuss significant inconsistencies and demographic disparities. Our findings underscore the limitations of current public LLMs for unsupervised medical use and demonstrate how RAG can improve diagnostic accuracy while mitigating bias-driven errors. To the best of our knowledge, this is the first study to systematically explore extrinsic bias and responsible AI considerations in the context of LLM-powered self-diagnosis, highlighting the urgent need for safeguards and deployment frameworks in consumer-facing health AI tools.
