Large language models (LLMs) are increasingly used to answer health-related questions and support healthcare workflows, yet evidence for their safety still relies heavily on static benchmarks that can rapidly become obsolete or be optimized against. Here we introduce a Dynamic, Automatic and Systematic (DAS) red-teaming audit framework that continuously stress-tests LLMs for health across four safety-critical axes: robustness, privacy, bias and hallucination. Validated against board-certified clinicians, a suite of adversarial agents autonomously mutates health-related test cases to uncover vulnerabilities in real time. Applying DAS to 15 state-of-the-art LLMs revealed a profound gap between high static benchmark performance and low dynamic reliability—the ‘benchmarking gap’. Despite median MedQA accuracy exceeding 80%, 94% of previously correct answers failed under dynamic robustness testing. This brittleness generalized to the realistic, open-ended HealthBench dataset, where top-tier models exhibited failure rates exceeding 70%, suggesting that high scores on established static benchmarks may reflect superficial memorization. We observed similarly high failure rates across other domains: privacy leaks were elicited in 86% of scenarios, cognitive bias priming altered recommendations in 81% of fairness tests and hallucination rates exceeded 74% in widely used models. By converting LLM safety evaluation for health from a static checklist into a living adversarial audit, DAS provides a scalable framework for surfacing latent risks before such systems are deployed in consumer-facing health assistants and broader clinical workflows. Using a suite of adversarial agents designed to test large language models in interactive conversations in a range of safety-related metrics, this study shows strong gaps between static performance and reliability in multi-turn conversations and high failure rates with respect to privacy, fairness and accuracy.

Background: Mental disorders are one of the leading causes of disease burden worldwide. Large language models (LLMs) offer a promising avenue for improving mental healthcare. However, whether LLMs can match human therapists on key psychotherapeutic competencies in standardized assessments remains unclear. To inform this discussion, we applied the human psychotherapist qualification standard to LLMs and examined the performance of contemporary commercial and open-source LLMs against human performance in official state examinations. Methods: In this pre-registered observational study, 14 widely used commercial (e.g., ChatGPT-4o, Claude 3.5 Sonnet) and open-source (e.g., Llama-4-Maverick, DeepSeek-V3) LLMs were evaluated on three official psychotherapist state exams from Germany. Questions and official solutions were independently extracted from a secure training platform restricted to registered trainees. Models produced concise final answers via standardized prompts. Two blinded raters scored all responses against the official solutions. Performance was compared with pass thresholds and human cohort averages. Sensitivity analyses focused on domain-specific performance across 10 domains (i.e., psychological, medical, pharmacological, guideline, ICD, healthcare system and legal, behavioural therapy, psychodynamic therapy, and other therapies such as systemic). Findings: All commercial models and 7 out of 11 open-source models exceeded the pass threshold on all exams. Five models (Claude 3.5 Sonnet, ChatGPT-4o, ChatGPT-o1-preview, Llama-4-Maverick, and DeepSeek-V3) surpassed the human cohort average across all exams, in some cases by more than one standard deviation. Domain-level analyses across all LLMs showed excellent performance in medical (M=95·3%, SD=9·58) and pharmacological (M=89·1%, SD=16·10) questions but substantial deficits in psychotherapy-related domains, including psychodynamic therapy (M=54·12%, SD=22·81) and healthcare system/legal knowledge (M=62·44%, SD=21·28). Interpretation: Several commercial and leading open-source LLMs match or exceed average trainee performance on written qualification exams, even without specialized mental-health fine-tuning. These results highlight the potential of LLMs and identify promising commercial and open-source LLMs as candidates for future clinical applications. However, further research, clear qualification standards, safety benchmarks, crisis-response safeguards, and regulatory evaluation frameworks are needed before LLM-based systems can be responsibly integrated into mental healthcare.

A bstract Human disease risk emerges from the shared influences of genetics, environment, lifestyle, and concurrent diseases over time, resulting in recurring patterns of susceptibility across conditions. However, most risk prediction models treat diseases as independent outcomes or rely on limited input variables, restricting their ability to capture these shared patterns. Here we present RisQ, a framework that learns a unified representation of human health across diseases, modalities, and time. This representation is queried with natural language to estimate disease risk for arbitrary diseases and prediction horizons. Generalization to unseen disease groups and prediction horizons indicates that information is shared across diseases and time, revealing a common structure of disease risk that is learnable. Trained and validated in 488,170 participants from the UK Biobank and evaluated without retraining in 257,538 participants from the independent All of Us cohort, RisQ leverages this shared structure to outperform disease-specific models, multi-disease frameworks, and tabular foundation models in risk prediction. We show that jointly modeling increasing numbers of diseases, input modalities, and prediction horizons improves performance, indicating that scaling these axes increases information transfer and enriches the learned structure. We then show this structure is multi-scale: it captures demographic determinants of disease susceptibility, while also organizing individuals into reproducible cross-disease risk clusters within demographically restricted subgroups. Genetic analyses further support the biological grounding of the structure by linking gene-level loss of function to cross-disease risk profiles. This surfaces known relationships of HBB , SLC22A12 , CASR , and LDLR , while also highlighting less characterized associations. Together, these results indicate that human disease risk exhibits a shared structure that can be learned from multimodal data to improve risk prediction, stratify individuals by cross-disease susceptibility, and support the discovery of relationships across diseases.

To drive progress in science and engineering, large language models (LLMs) must be able to process large amounts of numerical data and solve long calculations efficiently. This is currently only possible through the use of external tools or extensive reasoning chains, either weakening the numerical representations of LLMs or limiting the length of problems they can solve. We show that frontier LLMs require excessive amounts of reasoning tokens to solve even basic calculations, which is exacerbated by their tokenization strategies that split single numbers into multiple tokens. This motivates the need for efficient and effective single-token number encodings. We introduce a set of desiderata for such encodings and show that existing approaches fail to fulfill them. To address these shortcomings, we propose BitTokens, a novel encoding strategy that represents any number as a single token using its IEEE 754 binary floating-point representation. Through extensive experiments we show that our BitTokens allow even small language models to learn algorithms that solve basic arithmetic operations nearly perfectly. This newly gained efficiency could expand the length and complexity of problems language models can solve.

Simulating realistic patient interactions is a key requirement to testing clinical applications of LLMs at scale without time-consuming and expensive user studies. However, existing approaches often lack realism and controllability, often oversharing information unprompted, and failing to capture the wide variability of patient behavior. Here, we introduce PatientsWithPersonality (PWP), a patient simulation framework that generates realistic yet diverse virtual patient responses through explicit personality parametrization over a latent patient state. Grounded in HEXACO, a six-dimensional personality space used to quantify and parameterize human behavioral traits, our approach enables fine-grained control over conversational style, cooperativeness, and information disclosure within a unified framework. In a clinician evaluation, PWP is judged nearly as realistic as recorded human actors and clearly ahead of prior simulators, while being flagged as "too informative" far less often. Conditioning on HEXACO axes yields personas whose configured traits are recoverable by both clinicians and an autorater, span a substantially wider behavioral footprint than the closest baseline, and prevent oversharing. Altogether, our framework paves the way for more accurate and informative LLM benchmarking through our realistic and steerable patient simulator.

Survival analysis is crucial for many medical applications, but remains challenging for modern machine learning due to limited data, censoring, and the heterogeneity of tabular covariates. While the prior-fitted paradigm, which relies on pretraining models on large collections of synthetic datasets, has recently facilitated tabular foundation models for classification and regression, its suitability for time-to-event modeling remains unclear. We propose a flexible survival data generation framework that defines a rich survival prior with explicit control over covariates and time-event distributions. Building on this prior, we introduce Survival In-Context (SIC), a prior-fitted in-context learning model for survival analysis that is pretrained exclusively on synthetic data. SIC is trained to approximate Bayesian posterior predictive inference under the synthetic survival prior, enabling individualized survival prediction in a single forward pass, requiring no task-specific training or hyperparameter tuning. Across a broad evaluation on real-world survival datasets, SIC achieves competitive or superior performance compared to classical and deep survival models, particularly in small and medium-sized data regimes, highlighting the promise of a prior-fitted paradigm for survival analysis. The code and pretrained models will be made available upon publication.

Clinicians spend significant time reviewing medical images and transcribing findings. By integrating visual and textual data, foundation models have the potential to reduce workloads and boost efficiency, yet their practical clinical value remains uncertain. In this study, we find that OpenAI’s ChatGPT-4o and two medical vision-language models (VLMs) significantly underperform ophthalmologists in key tasks for age-related macular degeneration (AMD). To address this, we developed a dedicated training curriculum, designed by domain specialists, to optimize VLMs for tasks related to clinical decision making. The resulting model, RetinaVLM-Specialist, significantly outperforms foundation medical VLMs and ChatGPT-4o in AMD disease staging (F1: 0.63 vs. 0.33) and referral (0.67 vs. 0.50), achieving performance comparable to junior ophthalmologists. In a reader study, two senior ophthalmologists confirmed that RetinaVLM’s reports were substantially more accurate than those written by ChatGPT-4o (64.3% vs. 14.3%). Overall, our curriculum-based approach offers a blueprint for adapting foundation models to real-world medical applications.

Supervised contrastive learning (SupCon) has proven to be a powerful alternative to the standard cross-entropy loss for classification of multi-class balanced datasets. However, it struggles to learn well-conditioned representations of datasets with long-tailed class distributions. This problem is potentially exacerbated for binary imbalanced distributions, which are commonly encountered during many real-world problems such as medical diagnosis. In experiments on seven binary datasets of natural and medical images, we show that the performance of SupCon decreases with increasing class imbalance. To substantiate these findings, we introduce two novel metrics that evaluate the quality of the learned representation space. By measuring the class distribution in local neighborhoods, we are able to uncover structural deficiencies of the representation space that classical metrics cannot detect. Informed by these insights, we propose two new supervised contrastive learning strategies tailored to binary imbalanced datasets that improve the structure of the representation space and increase downstream classification accuracy over standard SupCon by up to 35%. We make our code available.1

Cardiac magnetic resonance (CMR) imaging is the gold standard for non-invasive cardiac assessment, offering rich spatio-temporal views of the heart's anatomy and physiology. Patient-level health factors, such as demographics, metabolic, and lifestyle, are known to substantially influence cardiovascular health and disease risk, yet remain uncaptured by CMR alone. To holistically understand cardiac health and to enable the best possible interpretation of an individual's disease risk, CMR and patient-level factors must be jointly exploited within an integrated framework. Recent multi-modal approaches have begun to bridge this gap, yet they often rely on limited spatio-temporal data and focus on isolated clinical tasks, thereby hindering the development of a comprehensive representation for cardiac/health evaluation. To overcome these limitations, we introduce ViTa, a step toward foundation models that delivers a comprehensive representation of the heart and a precise interpretation of individual disease risk. Leveraging data from 42,000 UK Biobank participants, ViTa integrates 3D+T cine stacks from short-axis and long-axis views, enabling a complete capture of the cardiac cycle. These imaging data are then fused with detailed tabular patient-level factors, enabling context-aware insights. This multi-modal paradigm supports a wide spectrum of downstream tasks, including cardiac phenotype and physiological feature prediction, segmentation, and classification of cardiac/metabolic diseases within a single unified framework. By learning a shared latent representation that bridges rich imaging features and patient context, ViTa moves beyond traditional, task-specific models toward a universal, patient-specific understanding of cardiac health, highlighting its potential to advance clinical utility and scalability in cardiac analysis. 2.

Cardiovascular diseases (CVD) can be diagnosed using various diagnostic modalities. The electrocardiogram (ECG) is a cost-effective and widely available diagnostic aid that provides functional information of the heart. However, its ability to classify and spatially localise CVD is limited. In contrast, cardiac magnetic resonance (CMR) imaging provides detailed structural information of the heart and thus enables evidence-based diagnosis of CVD, but long scan times and high costs limit its use in clinical routine. In this work, we present a deep learning strategy for cost-effective and comprehensive cardiac screening solely from ECG. Our approach combines multimodal contrastive learning with masked data modelling to transfer domain-specific information from CMR imaging to ECG representations. In extensive experiments using data from 40,044 UK Biobank subjects, we demonstrate the utility and generalisability of our method for subject-specific risk prediction of CVD and the prediction of cardiac phenotypes using only ECG data. Specifically, our novel multimodal pre-training paradigm improves performance by up to 12.19% for risk prediction and 27.59% for phenotype prediction. In a qualitative analysis, we demonstrate that our learned ECG representations incorporate information from CMR image regions of interest. Our entire pipeline is publicly available at https://github.com/oetu/MMCL-ECG-CMR.

Modern deep learning-based clinical imaging workflows rely on accurate labels of the examined anatomical region. Knowing the anatomical region is required to select applicable downstream models and to effectively generate cohorts of high quality data for future medical and machine learning research efforts. However, this information may not be available in externally sourced data or generally contain data entry errors. To address this problem, we show the effectiveness of self-supervised methods such as SimCLR and BYOL as well as supervised contrastive deep learning methods in assigning one of 14 anatomical region classes in our in-house dataset of 48,434 skeletal radiographs. We achieve a strong linear evaluation accuracy of 96.6 an ensemble approach. Furthermore, only a few labeled instances (1 training set) suffice to achieve an accuracy of 92.2 low-label and thus low-resource scenarios. Our model can be used to correct data entry mistakes: a follow-up analysis of the test set errors of our best-performing single model by an expert radiologist identified 35 labels and 11 anatomical region labelling performance increased – without and with an ensemble, respectively – to a theoretical accuracy of 98.0

Clinicians spend a significant amount of time reviewing medical images and transcribing their findings regarding patient diagnosis, referral and treatment in text form. Vision-language models (VLMs), which automatically interpret images and summarize their findings as text, have enormous potential to alleviate clinical workloads and increase patient access to high-quality medical care. While foundational models have stirred considerable interest in the medical community, it is unclear whether their general capabilities translate to real-world clinical utility. In this work, we demonstrate that OpenAI's ChatGPT-4o model, in addition to two foundation VLMs designed for medical use, markedly underperform compared to practicing ophthalmologists on specialist tasks crucial to the care of patients with age-related macular degeneration (AMD). To address this, we initially identified the essential capabilities required for image-based clinical decision-making, and then developed a curriculum to selectively train VLMs in these skills. The resulting model, RetinaVLM, can be instructed to write reports that significantly outperform those written by leading foundation medical VLMs and ChatGPT-4o in disease staging (F1 score of 0.63 vs. 0.33) and patient referral (0.67 vs. 0.50), and approaches the diagnostic performance of junior ophthalmologists (who achieve 0.77 and 0.78 on the respective tasks). Furthermore, in a single-blind reader study two senior ophthalmologists with up to 32 years of experience found RetinaVLM's reports were found to be substantially more accurate than those by ChatGPT-4o (64.3 that our curriculum-based approach provides a blueprint towards specializing foundation medical VLMs for real-world clinical tasks.

Analyzing temporal developments is crucial for the accurate prognosis of many medical conditions. Temporal changes that occur over short time scales are key to assessing the health of physiological functions, such as the cardiac cycle. Moreover, tracking longer term developments that occur over months or years in evolving processes, such as age-related macular degeneration (AMD), is essential for accurate prognosis. Despite the importance of both short and long term analysis to clinical decision making, they remain understudied in medical deep learning. State of the art methods for spatiotemporal representation learning, developed for short natural videos, prioritize the detection of temporal constants rather than temporal developments. Moreover, they do not account for varying time intervals between acquisitions, which are essential for contextualizing observed changes. To address these issues, we propose two approaches. First, we combine clip-level contrastive learning with a novel temporal embedding to adapt to irregular time series. Second, we propose masking and predicting latent frame representations of the temporal sequence. Our two approaches outperform all prior methods on temporally-dependent tasks including cardiac output estimation and three prognostic AMD tasks. Overall, this enables the automated analysis of temporal patterns which are typically overlooked in applications of deep learning to medicine.

Clinical decision-making is one of the most impactful parts of a physician’s responsibilities and stands to benefit greatly from artificial intelligence solutions and large language models (LLMs) in particular. However, while LLMs have achieved excellent performance on medical licensing exams, these tests fail to assess many skills necessary for deployment in a realistic clinical decision-making environment, including gathering information, adhering to guidelines, and integrating into clinical workflows. Here we have created a curated dataset based on the Medical Information Mart for Intensive Care database spanning 2,400 real patient cases and four common abdominal pathologies as well as a framework to simulate a realistic clinical setting. We show that current state-of-the-art LLMs do not accurately diagnose patients across all pathologies (performing significantly worse than physicians), follow neither diagnostic nor treatment guidelines, and cannot interpret laboratory results, thus posing a serious risk to the health of patients. Furthermore, we move beyond diagnostic accuracy and demonstrate that they cannot be easily integrated into existing workflows because they often fail to follow instructions and are sensitive to both the quantity and order of information. Overall, our analysis reveals that LLMs are currently not ready for autonomous clinical decision-making while providing a dataset and framework to guide future studies. Using a curated dataset of 2,400 cases and a framework to simulate a realistic clinical setting, current large language models are shown to incur substantial pitfalls when used for autonomous clinical decision-making.

Microglial activation occurs early in Alzheimer's disease (AD) and previous studies reported both detrimental and protective effects of microglia on AD progression. Here, we used CSF sTREM2 to investigate disease stage‐dependent drivers of microglial activation and to determine downstream consequences on AD progression. We included 402 patients with measures of earliest beta‐amyloid (CSF Aβ1‐42) and late‐stage fibrillary Aβ pathology (amyloid‐PET centiloid), as well as sTREM2, p‐tau181, and FDG‐PET. To determine disease stage, we stratified participants into early Aβ‐accumulators (Aβ CSF+/PET−; n = 70) or late Aβ‐accumulators (Aβ CSF+/PET+; n = 201) plus 131 controls. In early Aβ‐accumulators, higher centiloid was associated with cross‐sectional/longitudinal sTREM2 and p‐tau181 increases. Further, higher sTREM2 mediated the association between centiloid and cross‐sectional/longitudinal p‐tau181 increases and higher sTREM2 was associated with FDG‐PET hypermetabolism. In late Aβ‐accumulators, we found no association between centiloid and sTREM2 but a cross‐sectional association between higher sTREM2, higher p‐tau181 and glucose hypometabolism. Our findings suggest that a TREM2‐related microglial response follows earliest Aβ fibrillization, manifests in inflammatory glucose hypermetabolism and may facilitate subsequent p‐tau181 increases in earliest AD. In Alzheimer's disease (AD), microglial activation has been linked to both detrimental and protective effects on disease progression. This study used CSF sTREM2 to investigate disease stage‐dependent drivers of microglial activation and to determine downstream consequences on AD progression. In Alzheimer's disease (AD), microglial activation has been linked to both detrimental and protective effects on disease progression. This study used CSF sTREM2 to investigate disease stage‐dependent drivers of microglial activation and to determine downstream consequences on AD progression.

Medical datasets and especially biobanks, often contain extensive tabular data with rich clinical information in addition to images. In practice, clinicians typically have less data, both in terms of diversity and scale, but still wish to deploy deep learning solutions. Combined with increasing medical dataset sizes and expensive annotation costs, the necessity for unsupervised methods that can pretrain multimodally and predict unimodally has risen. To address these needs, we propose the first self-supervised contrastive learning framework that takes advantage of images and tabular data to train unimodal encoders. Our solution combines SimCLR and SCARF, two leading contrastive learning strategies, and is simple and effective. In our experiments, we demonstrate the strength of our framework by predicting risks of myocardial infarction and coronary artery disease (CAD) using cardiac MR images and 120 clinical features from 40,000 UK Biobank subjects. Furthermore, we show the generalizability of our approach to natural images using the DVM car advertisement dataset. We take advantage of the high interpretability of tabular data and through attribution and ablation experiments find that morphometric tabular features, describing size and shape, have outsized importance during the contrastive learning process and improve the quality of the learned embeddings. Finally, we introduce a novel form of supervised contrastive learning, label as a feature (LaaF), by appending the ground truth label as a tabular feature during multimodal pretraining, outperforming all supervised contrastive baselines.

Abstract Alzheimer’s disease and cerebral small vessel disease are the two leading causes of cognitive decline and dementia and coexist in most memory clinic patients. White matter damage as assessed by diffusion MRI is a key feature in both Alzheimer’s and cerebral small vessel disease. However, disease-specific biomarkers of white matter alterations are missing. Recent advances in diffusion MRI operating on the fixel level (fibre population within a voxel) promise to advance our understanding of disease-related white matter alterations. Fixel-based analysis allows derivation of measures of both white matter microstructure, measured by fibre density, and macrostructure, measured by fibre-bundle cross-section. Here, we evaluated the capacity of these state-of-the-art fixel metrics to disentangle the effects of cerebral small vessel disease and Alzheimer’s disease on white matter integrity. We included three independent samples (total n = 387) covering genetically defined cerebral small vessel disease and age-matched controls, the full spectrum of biomarker-confirmed Alzheimer’s disease including amyloid- and tau-PET negative controls and a validation sample with presumed mixed pathology. In this cross-sectional analysis, we performed group comparisons between patients and controls and assessed associations between fixel metrics within main white matter tracts and imaging hallmarks of cerebral small vessel disease (white matter hyperintensity volume, lacune and cerebral microbleed count) and Alzheimer’s disease (amyloid- and tau-PET), age and a measure of neurodegeneration (brain volume). Our results showed that (i) fibre density was reduced in genetically defined cerebral small vessel disease and strongly associated with cerebral small vessel disease imaging hallmarks; (ii) fibre-bundle cross-section was mainly associated with brain volume; and (iii) both fibre density and fibre-bundle cross-section were reduced in the presence of amyloid, but not further exacerbated by abnormal tau deposition. Fixel metrics were only weakly associated with amyloid- and tau-PET. Taken together, our results in three independent samples suggest that fibre density captures the effect of cerebral small vessel disease, while fibre-bundle cross-section is largely determined by neurodegeneration. The ability of fixel-based imaging markers to capture distinct effects on white matter integrity can propel future applications in the context of precision medicine.

Uncertainty estimation, which provides a means of building explainable neural networks for medical imaging applications, have mostly been studied for single deep learning models that focus on a specific task. In this paper, we propose a method to propagate uncertainty through cascades of deep learning models in medical imaging pipelines. This allows us to aggregate the uncertainty in later stages of the pipeline and to obtain a joint uncertainty measure for the predictions of later models. Additionally, we can separately report contributions of the aleatoric, data-based, uncertainty of every component in the pipeline. We demonstrate the utility of our method on a realistic imaging pipeline that reconstructs undersampled brain and knee magnetic resonance (MR) images and subsequently predicts quantitative information from the images, such as the brain volume, or knee side or patient's sex. We quantitatively show that the propagated uncertainty is correlated with input uncertainty and compare the proportions of contributions of pipeline stages to the joint uncertainty measure.

In Alzheimer’s disease (AD), younger symptom onset is associated with accelerated disease progression and tau spreading, yet the mechanisms underlying faster disease manifestation are unknown. To address this, we combined resting-state fMRI and longitudinal tau-PET in two independent samples of controls and biomarker-confirmed AD patients (ADNI/BioFINDER, n = 240/57). Consistent across both samples, we found that younger symptomatic AD patients showed stronger tau-PET in globally connected fronto-parietal hubs, i.e., regions that are critical for maintaining cognition in AD. Stronger tau-PET in hubs predicted faster subsequent tau accumulation, suggesting that tau in globally connected regions facilitates connectivity-mediated tau spreading. Further, stronger tau-PET in hubs mediated the association between younger age and faster tau accumulation in symptomatic AD patients, which predicted faster cognitive decline. These independently validated findings suggest that younger AD symptom onset is associated with stronger tau pathology in brain hubs, and accelerated tau spreading throughout connected brain regions and cognitive decline.