Accurate interpretation of chest radiograph images and generation of narrative reports is essential for patient care but places a heavy burden on radiologists and clinical experts. While AI models for automated report generation show promise, standardized evaluation frameworks remain limited. Here we present the ReXrank Challenge V1.0, a competition in the generation of chest radiograph reports utilizing ReXGradient, the largest test dataset consisting of 10,000 studies across 67 sites. The challenge attracted diverse participants from academic institutions, industry, and independent research teams, resulting in 8 new submissions alongside 16 state-of-the-art models previously benchmarked. Through comprehensive evaluation using multiple metrics, we analyzed model performance across various dimensions: differences between normal and abnormal studies, generalization capabilities across healthcare sites, and error rates in identifying clinical findings. This benchmark reveals that automated chest X-ray report generation remains fundamentally unsolved, with significant performance gaps between normal and abnormal studies, where even top-performing models achieve less than 45% error-free reporting on abnormal cases, and substantial variability across healthcare institutions, indicating that robust, clinically-ready systems require continued development before widespread deployment.
Biologically informed deep neural networks, which connect input layer to hidden layers based on genepathway relationship have gained popularity in recent years. However, most existing methods do not incorporate protein-protein interactions (PPI) and protein-DNA interactions (PDI) in their designs. In this study, we introduce BioLM-NET, a deep learning-based framework that fuses single cell or bulk gene expression data and DNA methylation data with prior biological knowledge including Protein- Protein Interactions (PPI), Protein-DNA Interactions (PDI). BioLM-NET also aggregates latent representation of omics signals at pathway-level through an attention-based pathway layer where a pretrained large language model (LLM) was incorporated to generate context-specific gene embeddings. We evaluated BioLM-NET on single cell colorectal cancer data from scTrioseq2 platform to predict primary and metastatic cancer cells, on TCGA-BRCA, TCGA-GBM, TCGA-COAD to predict cancer subtypes and ROSMAP data to predict Alzheimer's disease patient. Our results showed that BioLMNET outperformed baseline and state-of-the-art (SOTA) methods, P-NET and PASNet with statistical significance on scTrioseq2 data, TCGA-COAD and ROSMAP data and ties with SVM and Dense neural network on TCGA-BRCA data. Our ablation studies demonstrated the importance of incorporating PPI, PDI data and attention-based pathway layer. We also interpret our models and found out that our important input features are significantly enriched in GO terms and KEGG pathways and can serve as potential biomarkers or therapeutic targets for the corresponding disease.
Ecological momentary assessment (EMA) and wearable sensors offer unprecedented opportunities to capture the dynamics of substance use through real-time, high-resolution behavioral and physiological data. These data streams are increasingly used to train AI/ML models for digital phenotyping and predictive intervention, raising critical questions about fairness, bias, and inclusivity in model development. However, the adoption of these technologies, or the lack thereof, among diverse and historically marginalized groups raises questions and challenges of equity, cultural relevance, and participant trust. In this study, we conducted a four-week observational study with adults in Hawai.i where we combined continuous Fitbit monitoring and daily EMA surveys to document substance use patterns and cravings. Through semi-structured interviews and grounded theory analysis, we identified six primary barriers to study participation and adherence: (1) disruptions to daily routines, (2) physical and psychosocial discomfort associated with wearing the Fitbit device, (3) concerns about aesthetic compatibility and professional appearance, (4) phonerelated issues, (5) challenges related to substance use and cravings, and (6) socially sensitive contexts. We also highlight participant-identified facilitators, such as the value of participant-driven scheduling, motivational feedback, and contextually adaptive protocols. Drawing on these collective findings, we propose a set of design guidelines aimed at advancing the inclusivity, engagement, and fairness of wearable-based EMA research.
This feasibility study assesses using Large Language Models (LLMs) to automate the extraction of Multiple Sclerosis (MS) progression metrics from unstructured clinical notes. While the Expanded Disability Status Scale (EDSS) is the standard for measuring disability, calculating it during routine care is burdensome, often leaving progression details locked in narrative text. Using OpenAI’s GPT-4.1 in a zero-shot setting, the study processed 288 progress notes from 239 patients. The model extracted scores for eight Functional Systems (FS) and derived a total EDSS score. Performance was evaluated against expert annotations using a strict and lenient agreement criteria. Results showed strong performance for FS extraction, with accuracy ranging from 0.782 to 0.971 across domains. While exact EDSS agreement was variable due to documentation ambiguity, applying clinically meaningful tolerance bands yielded high consistency, with most disagreements being minor adjacent errors unlikely to alter therapy. The study confirms that LLMs can feasibly derive detailed disability measures, potentially reducing documentation burden enabling the automated, population-wide tracking of MS progression over time.
MedAgentBench is the first benchmark for evaluating LLM agents on clinical tasks in a FHIR-compliant EHR. In this paper, we present significant prompt engineering and tool design improvements over the original agent implementation and introduce a memory component that enables the agent to learn from prior failures. We added new tools for the agent to properly format its output for tasks, interact with an EHR without constructing explicit HTTP requests, which were prone to syntax errors, and make math calculations. We also wrote a new system prompt that asked the agent to outline its plan before making any tool calls and think step by step using chain of thought reasoning, and provided few shot examples of good vs. bad outputs. Using GPT-4.1 as the base model, our agent achieved a success rate of 91.0% without memory and 98.0% with memory. A surprising consequence is that the agent performed better on a different task that had no associated memory entry, possibly demonstrating that LLMs can adapt to the style of tasks presented by users. To contribute to the benchmark and evaluate the generalization of our agent, we developed 300 new multi-step clinically-driven tasks in collaboration with a physician. Lastly, we show the current limitations of these benchmarks and highlight the necessary next steps and challenges for the responsible deployment of AI agents in real-world healthcare settings. We hope that this paper leads to further development of EHR agents and benchmarks.
