This position paper argues that current insurance markets are fundamentally misaligned with AI risk, creating significant coverage, control, and measurement gaps that threaten both organizations and insurers. Through analysis of insurance policy coverages, risk controls, and measurement approaches across 15 AI risk categories, we demonstrate that conventional insurance structures are inadequately addressing the unique challenges presented by AI systems. This misalignment stems from AI's autonomous nature, probabilistic operations, opacity, and rapid development cycles, which conflict with insurance assumptions about human control, causality, deterministic failures, and stable risk environments. While some argue that existing policies sufficiently cover AI risks, our evidence shows that even the most relevant cyber and technology liability insurance products leave organizations exposed to significant AI-specific harms. This qualitative analysis establishes a foundational framework for future quantitative studies and proposes measurement approaches that could enable more rigorous empirical analysis as AI incident data matures. Without deliberate evolution in AI risk transfer mechanisms, organizations face a protection gap while insurers confront potentially catastrophic unpriced exposure, creating an urgent need for market-based risk transfer solutions that can drive adoption of technical AI security and safety standards.
AI has shown remarkable potential in healthcare, but faces accessibility challenges due to high computational and expertise demands, especially in medical image analysis. Vector embeddings, compact representations of medical images achieved from foundation models in zero-shot inference, offer a potential solution. Recently, an equivalent vector embeddings dataset of existing large publicly available medical images has been released, for which training an AI model requires significantly lower computing infrastructure and storage needs. Such data sets provide greater accessibility to AI in medical imaging for those who do not have access to large computing resources. The burning question remains: What is the gain or loss in using vector embedding to replace medical images, particularly from a fairness and utility point of view? In this work, we compare AI models trained in vector embeddings (Emb) with raw chest radiograph images for disease diagnosis, focusing on both performance and fairness. Our results show that Emb-based models match or exceed image-based models in diagnostic performance while improving fairness. Crucially, Emb achieve this with far less computational cost. These findings position Emb as a powerful, scalable alternative to image-based AI, especially valuable for low-resource settings where access to GPUs and expert infrastructure is limited.
Time series (TS) analysis is an active application area for Artificial Intelligence (AI) methods where the objective is to analyze numeric quantities indexed by time for tasks like classification, forecasting, and abnormality detection. In health, TS manifests as biosignals like the electroencephalogram (EEG), where electrical signals from the brain are analyzed. AI and health communities can tremendously benefit each other in TS, with the former offering advanced analytical methods while the latter provides complex data sets and trust-sensitive use cases. But the communities also need to overcome confusing terminologies, hidden assumptions, and a lack of necessary domain contexts for result evaluation and interpretation. In this paper, we attempt to bridge the gap using the problem of channel selection in EEG. We outline challenges in working with EEG data, demonstrate via two experiments how simple explainable AI (XAI) methods can be quite effective for channel selection irrespective of EEG tasks/paradigms, and argue that recent TS trends in AI, like LLMs and XAI methods, can benefit health as well. We hope that this work will bring researchers working on TS problems at the intersection of AI and health closer to work in AI trustworthiness so that they can better leverage results from their respective areas to overcome common challenges. All code and resources are released on GitHub to help others replicate.
The generation of structured medical reports using large language models (LLMs) presents unique challenges, particularly in maintaining clinical relevance and adhering to strict formatting requirements. In this work, we investigate the effectiveness of fine-tuning LLMs for structured report generation using DeepSeek R1 models. We conduct experiments with two model variants: DeepSeek R1 8B and DeepSeek R1 14B. For both models, we apply Group Relative Policy Optimization (GRPO) using the Medical Information Mart for Intensive Care (MIMIC-IV) dataset, leveraging Low-Rank Adaptation (LoRA) for parameter-efficient fine-tuning. Our results show that the GRPO fine-tuned DeepSeek-R1 8B and 14B models outperformed all baseline models, including the larger 32B DeepSeek-R1 model, demonstrating the effectiveness of parameter-efficient tuning. These findings underscore the potential of reinforcement learning-based fine-tuning of LLMs for generating structured reports in the medical domain.
Deep learning models have demonstrated impressive accuracy in predicting acute kidney injury (AKI), a condition affecting up to 20% of ICU patients, yet their black-box nature prevents clinical adoption in high-stakes critical care settings. While existing interpretability methods like SHAP, LIME, and attention mechanisms can identify important features, they fail to capture the temporal dynamics essential for clinical decision-making, and are unable to communicate when specific risk factors become critical in a patient's trajectory. This limitation is particularly problematic in the ICU, where the timing of interventions can significantly impact patient outcomes. We present a novel interpretable framework that brings temporal awareness to deep learning predictions for AKI. Our approach introduces three key innovations: (1) a latent convolutional concept bottleneck that learns clinically meaningful patterns from ICU time-series without requiring manual concept annotation, leveraging Conv1D layers to capture localized temporal patterns like sudden physiological changes; (2) Temporal Concept Tracing (TCT), a gradient-based method that identifies not only which risk factors matter but precisely when they become critical addressing the fundamental question of temporal relevance missing from current XAI techniques; and (3) integration with MedAlpaca to generate structured, time-aware clinical explanations that translate model insights into actionable bedside guidance. We evaluate our framework on MIMIC-IV data, demonstrating that our approach performs better than existing explainability frameworks, Occlusion and LIME, in terms of the comprehensiveness score, sufficiency score, and processing time. The proposed method also better captures risk factors inflection points for patients timelines compared to conventional concept bottleneck methods, including dense layer and attention mechanism. This work represents the first comprehensive solution for interpretable temporal deep learning in critical care that addresses both the what and when of clinical risk factors. By making AKI predictions transparent and temporally contextualized, our framework bridges the gap between model accuracy and clinical utility, offering a path toward trustworthy AI deployment in time-sensitive healthcare settings.
Clinical AI models increasingly inform care decisions, yet implicit assumptions about data timing, label semantics, calibration, and operating thresholds are rarely specified or monitored, causing subtle failures with standard metrics. We present executable safety contracts, lightweight, task-level specifications enforced as runtime checks for hospital length-of-stay prediction. The specifications capture preconditions (data integrity, index-time alignment, censoring), postconditions (admissible outputs, alert-budget bounds), and invariants (coverage/calibration targets, subgroup equity). We implement these checks in a Python pipeline and evaluate them on a single-center MIMIC-IV cohort and a multi-center eICU-style cohort using simple baselines (logistic regression, gradient boosting) with conformal intervals and post-hoc calibration. The contracts exposed hazards that MAE (Mean Absolute Error), AUC (Area Under the ROC Curve), or ECE (Expected Calibration Error) alone missed, for example, acceptable point error with severe under-coverage in eICU, well-calibrated probabilities that nonetheless violated alert-rate constraints, and dataset-specific fairness gaps. Lightweight remedies such as conformal radius tuning, threshold/alert-scope selection, and calibration often restored compliance without degrading point performance, while clarifying when deeper modeling or policy changes were needed. Overall, the case study shows that Design by Contract principles extend beyond APIs to system-level specifications for clinical ML, providing a practical way to state safety expectations, check them with minimal compute, and make violations actionable.
A safe and interpretable visual method for prosthetic alignment assessment is proposed, suitable for sensorless scenarios such as home rehabilitation and telemedicine. The method collects human skeletal data based on a depth camera and extracts the motion difference characteristics of the left and right legs through gait symmetry analysis. Three types of clearly structured evaluation indicators are designed, including differences in joint range of motion, differences in swing phase duration, and angular trajectory similarity, to construct an interpretable alignment scoring function. This system is designed as a front-end module of a digital twin system. The scoring results can intuitively reflect differences in wearing status, facilitating real-time evaluation and adjustment of prosthetic alignment quality. Preliminary experiments have verified the stability and practicality of this method under visual recognition conditions, laying the foundation for personalized prosthetic optimization based on digital twins.
Retrieval-Augmented Generation (RAG) has become the go-to fix for LLM hallucinations. But its most common form, built on Vector Databases, is like a confident consultant who has only read the executive summaries. It's fluent, convincing, and adept at finding information that sounds right, but critically lacks the deep, verifiable connections between the facts. In high-stakes domains like medicine, this creates a dangerous new form of AI: one that is wrong with conviction. This paper provides a comparative experiment for distinguishing between answers that merely sound correct and those that are verifiably true. Our head-to-head evaluation of Vector-based versus Knowledge Graph-based RAG reveals a stark architectural choice. Our findings demonstrate that while Vector RAG produces a convincing but untraceable story, the Knowledge Graph approach delivers a factually correct answer with a verifiable evidence trail. This is the blueprint for building RAG systems that don't ask for your trust - they earn it by showing their work.
After a brief introduction of the Knowledge Graph (KG) technology, a subfield of symbolic AI used to represent and manage semantic information, this article is devoted to quality assessment, emphasizing the importance of developing trustworthy AI in such knowledge-based systems, particularly in safety-critical applications. In this context, we remind several metrics or methods that can be applied to KGs, along with examples of their implementation in the context of digital NOTAMs (Notice To AirMen) illustrated by the HLIF2024 Hackathon.
In today's age of information, unstructured information can become overwhelming and difficult to interpret, particularly in safety critical domains such as healthcare where the volume and complexity of unstructured textual notes is required to be interpretable, insightful, and easily automated for processing. This paper introduces Hermes, a modular agentic system that transforms unstructured clinical text into a modified version of the Subjective-Objective-Assessment-Plan (SOAP) format and generates a knowledge graph offering a high-level, distilled view that facilitates downstream clinical reasoning and decision-making. Hermes employs a multi-agent architecture consisting of four specialized components: Hermes-R (report generation), Hermes-G (knowledge graph generation), Hermes-Q (question-answer pair generation), and Hermes-A (answer generation). These agents operate sequentially with validation to generate structured medical information using iterative refinement. Preliminary evaluations on a few samples demonstrate that Hermes is able to generate structured clinical reports and knowledge graphs according to provided specifications from unstructured discharge summaries with good consistency, accuracy, and reward score. Hermes offers a unified framework that advances clinical natural language processing, bridging structured representation, question answering, and semantic validation.
Quantum generative models (QGMs), including Variational Quantum Circuits (VQCs) and Quantum GANs, hold significant potential in generating complex data distributions beyond the capabilities of classical generative approaches. However, robust monitoring and evaluation of QGMs remain underdeveloped due to hardware constraints, stochastic quantum behavior, and reproducibility limitations. This paper proposes a scalable and modular framework using Apache Spark and MLflow to monitor, evaluate, and track the performance of QGMs. The framework enables ingestion of quantum-generated data, distributed computation of performance metrics such as fidelity, entanglement entropy, and distributional divergence, and experiment tracking via MLflow. I validate our methodology using Qiskit-based simulated QGMs and demonstrate the effectiveness of classical big data tools in bridging the evaluation gap in quantum ML research.
Recent advances in AI, especially those of LLMs, bring the prospect of increased adoption of AI in medicine and medical education. In particular, many institutions responsible for medical treatment and education are rapidly aiming to increase AI use in practice and curricula. However, the potential downsides of overuse of AI in these fields are under-discussed. In the rush to AI adoption, sources of healthcare risk such as LLM reliability, patient privacy, financial and environmental costs, vendor dependencies, and AI over-reliance are often not deeply considered. This paper discusses these recent trends and makes recommendations for healthcare institutions considering further adoption of AI.
Reliable anomaly detection is crucial for safe AI deployment in clinical imaging, yet most systems offer limited insight into prediction uncertainty or failure modes-key factors in medical decision-making. We analyze the uncertainty characteristics of a patch-level Deep Q-Network anomaly detection framework (DQN_AD) for brain MRI, trained with few annotated abnormal cases and designed to generalize to highly imbalanced clinical datasets. Our study links uncertainty to model errors, calibration, anomaly scores, spatial correspondence with ground truth, and selective evaluation. Results show that high-uncertainty predictions consistently coincide with error-prone regions, providing a strong signal for identifying potential failures. This study establishes the foundation for uncertainty-aware, reinforcement learning-based anomaly detection models that enhance reliability, interpretability, and clinical usability in large-scale MRI analysis.
Principles-based frameworks for AI assurance have been proposed for various AI/ML use cases, focusing on aspects such as ethical design, trustworthiness, and safety. However, translating these high-level principles into actionable, objective criteria for auditing, particularly by third parties, remains challenging. Our analysis shows this is due to the inherent subjectivity of principles, the need for vertical frameworks tailored to specific AI/ML applications, and the unreliability of information gathered during the assurance process. In this paper, we present a case study on how to develop and operationalise a principles-based framework for AI assurance aimed at assessing the ‘accuracy’ of child sexual exploitation (CSEA) and terrorism detection technologies in the context of online safety. The proposed assurance framework addresses a requirement in the UK's 2023 Online Safety Act to create an 'accreditation' scheme specifically for CSEA and terrorism detection technologies. We discuss the critical challenges for operationalising such principles-based frameworks for assurance, particularly in relation to ensuring transparency, reliability, and consistency in audits. We also map potential issues which remain for effectively assessing and auditing AI/ML technologies, informing the development of future research agendas which further research and development of robust standards for assurance, particularly in sociotechnical contexts.
Robots that can physically interact with humans in a safe, comfortable, and intuitive manner can help in a variety of settings. However, perceptions of the users greatly affect the acceptability of such robots. Ability of the system to understand user's perception of the physical interaction as well as adapting robot's behaviors based on user perception and interaction context can facilitate acceptability of these robots. In this paper we propose a perception-based interaction adaptation framework. One main component of this framework is a multi-modal perception model which is grounded on the existing literature and is intended to provide a quantitative estimation of the human state- defined as the perceptions of the physical interaction- by using human, robot, and context information. This model is intended to be comprehensive in many physical Human-Robot Interaction (pHRI) scenarios. The estimated human state is fed to a context-aware behavior adaptation framework which recommends robot behaviors to improve human state using a learned behavior cost model and an optimization formulation. We show the potential and feasibility of such a human state estimation model by evaluating a reduced model, with data collected through a user study. Additionally, through some feature analysis, we aimed to shed light on future interaction designs for pHRI.
Value chains for AI systems are becoming increasingly complex and can consists of multiple actors that contribute services, tools, data, models and code. An efficient risk management along this value chain requires all actors to communicate potential risk sources and recommendations for mitigation. The Software Bill of Materials (SBOM) is a method from cybersecurity, that enables organizations to communicate information like licences, security vulnerabilities and dependencies of software components. SBOM raises increasing interest in the AI community to share information about AI components, like data and models. In this paper we discuss the suitability of SBOM for AI risk management along a value chain and show the potential but also gaps in current approaches.
Alzheimer's disease (AD) is the most common cause of dementia, yet many cases go undiagnosed due to limited access to expensive brain scans and lab tests. This study investigated whether medication data could help identify AD. Using data from 1,785 participants in the US-representative National Health and Nutrition Examination Survey 2013-2014, we identified 105 individuals (5.9%) with memory test scores suggesting possible AD. We evaluated seven machine learning models using medication features. Models that incorporated contextual prescription information, including the reasons for medication use and conditions being treated, achieved the best performance (AUC 0.61-0.63). In contrast, models using only basic drug names or provider information performed poorly (AUC 0.46-0.51). This performance difference was statistically significant (t = 14.98, p < 0.0001). Our findings suggest that medications data, when analyzed with attention to clinical context, could serve as a low-cost tool for identifying individuals at risk of AD. This approach may help address diagnostic disparities in settings with limited access to advanced testing.
Autonomous tiny drones face significant challenges in navigation due to strict constraints on size, weight, power, and onboard computational capacity. This paper presents a lightweight navigation framework that integrates basic multi-sensor perception with deep reinforcement learning (DRL) to enable safe, mapless flight in cluttered environments. We employ the Crazyflie 2.1 nano-drone, equipped with a grayscale camera and a multi-ranger deck, a laser-based distance sensor, for real-time obstacle detection and avoidance. A Proximal Policy Optimization (PPO) agent is trained within a ROS and Gazebo simulation environment to generate collision-free trajectories using fused visual and range data. The system is evaluated in two environments: a simple obstacle field, where the drone achieves a 100% success rate (112/112 episodes), and a densely cluttered map, where it reaches the target in 35% of trials (7/20). These results demonstrate that effective autonomous navigation is achievable using minimal sensing and low-computation models, making it well-suited for resource-constrained aerial platforms.
Understanding public perception towards water consumption is crucial for promoting sustainable water management, shaping effective policies, and enhancing the awareness of water conservation in diverse communities. Although previous works have studied water consumption from self-reported data and motivations, the topic has not been analyzed from the perspective of cognition from social media. Given the significance of social media's broad reach, real-time engagement, and the diverse demographic representation it offers, understanding how public perception is reflected in online discussions can provide valuable insights into societal attitudes, concerns, and behavioral trends related to water consumption. In this work, we performed a cognitive analysis, based on Reddit discussions about water consumption. Our approach includes both sentiment analysis, representing conscious attitudes, and concept mapping analysis, which captures subconscious cognitive frameworks. Sentiment analysis shows overall positive polarity on Reddit with key aspects of water consumption, while concept mapping reveals cognitive frameworks shaping perceptions. Together, these insights inform communication strategies and policy on water conservation.
Accurate prediction of the transmission fitness of emerging SARS-CoV-2 variants is vital for timely public health responses. In this study, we present a deep learning framework that predicts variant fitness from raw genomic sequences using a convolutional neural network (CNN) trained to regress Differential Population Growth Rate (DPGR) values. Our approach achieves high predictive accuracy (R-2 = 0.9168, MSE approximate to 1.94 x 10(-4)) on genomic sequences sampled from the USA and Europe. To interpret the model's predictions, we apply SHapley Additive exPlanations (SHAP) to identify nucleotide-level contributions to predicted fitness. Our analysis highlights key mutations in ORF9 (nucleocapsid), ORF2 (spike), ORF5 (membrane), and ORF8 that either enhance or reduce predicted DPGR. Notably, we identify amino acid-altering mutations such as D3L, E484K, N501Y, and V97I as strong positive contributors to fitness, while synonymous or non-coding mutations had more subtle or regulatory effects. These findings validate the potential of sequencebased modeling and interpretable AI to support early detection and prioritization of high-risk variants.
