Automated diagnostic report generation lies at the core of clinical diagnosis and can alleviate clinician shortages. Existing diagnostic report generation methods have two major limitations: they rely on unimodal inputs (e.g., images), ignoring textual biomarkers like medical history, and lack proactive dialogue capabilities to elicit personalized clinical information. To address those issues, we propose a ProActive Multimodal Agentic (PAMA) system, which performs comprehensive disease analysis by examining biomarkers in diverse sources, including multi-view medical images, medical histories, and diagnostic conversations. Built upon a knowledge graph and recommendation-based dialogue architecture, PAMA actively initiates adaptive, multi-turn conversations with patients, which is integrated with visual data for robust and reliable report generation. Specifically, PAMA actively generates adaptive multi-turn questions to collect clinically relevant background information, and then fuses the resulting dialogue context with visual representations for robust diagnostic report generation. We validate our approach on two real-world benchmark datasets, MIMIC-CXR and IU-Xray, through extensive quantitative evaluations and comparisons with state-of-the-art baselines. Furthermore, we conduct user studies to assess the realism and clinical quality of the generated reports. Finally, we present real-world case studies to examine the performance of our system across diverse scenarios, demonstrating its robustness on scalability, complexity, and data variability.

Accurately monitoring the screen exposure of young children is important for research related to screen use such as childhood obesity, physical activity, and social interaction. Most existing studies rely upon self-report or manual measures from bulky wearable sensors, thus lacking efficiency and accuracy in capturing quantitative screen exposure data. In this work, we developed a novel screen detection framework that utilizes egocentric images from a wearable sensor, named the screen time tracker (STT), and a vision language model (VLM). In particular, we devised a multi-view VLM that takes multiple views from egocentric image streams and interprets screen exposure dynamically. We validated our approach by using a dataset of children's free-living activities, demonstrating significant improvement over existing methods in conventional vision language models and object detection models. The combination of vision language model and lightweight hardware design provides a novel solution in screen detection for children. The proposed framework has great potential to benefit children's behavioral study.

At the core of modern prompting techniques is contextual sensitivity, the ability of large language models to adapt their predictions based on inference-time context. Despite its central role, inference behavior under strong contextual influence remains poorly understood, particularly at the level of internal inference dynamics. To bridge this gap, we introduce a theoretical framework for analyzing contextual influence through inference dynamics, enabling quantitative characterization of inference behavior beyond output-level answer changes. Our analysis shows that inference dynamics do not exhibit unbounded drift under repeated contextual assertions. Instead, predictive representations converge to stable, query-dependent regimes that fundamentally constrain whether contextual signals can alter a model’s prediction. This leads to a surprising finding: Repeated contextual assertions do not act as accumulating evidence during inference and may therefore fail to alter a model’s prediction even under unbounded repetition, while in other cases a prediction change becomes inevitable. We empirically validate our theoretical predictions across diverse models and tasks, demonstrating strong alignment between theory and observed inference behavior. These contributions offer a principled pathway toward characterizing the limits of contextual influence during inference, and provide practical implications for designing and evaluating repetition-based prompting methods.

Visual evidence selection is a critical component of multimodal retrieval-augmented generation (RAG), yet existing methods typically rely on semantic relevance or surface-level similarity, which are often misaligned with the actual utility of visual evidence for downstream reasoning. We reformulate multimodal evidence selection from an information-theoretic perspective by defining evidence utility as the information gain induced on a model’s output distribution. To overcome the intractability of answer-space optimization, we introduce a latent notion of evidence helpfulness and theoretically show that, under mild assumptions, ranking evidence by information gain on this latent variable is equivalent to answer-space utility. We further propose a training-free, surrogate-accelerated framework that efficiently estimates evidence utility using lightweight multimodal models. Experiments on MRAG-Bench and Visual-RAG across multiple model families demonstrate that our method consistently outperforms state-of-the-art RAG baselines while achieving substantial reductions in computational cost. We release our code at https://github.com/Hcnaeg/utility-mrag.

Efficient patient-doctor interaction is among the key factors for a successful disease diagnosis. During the conversation, the doctor could query complementary diagnostic information, such as the patient's symptoms, previous surgery, and other related information that goes beyond medical evidence data (test results) to enhance disease diagnosis. However, this procedure is usually time-consuming and less-efficient, which can be potentially optimized through computer-assisted systems. As such, we propose a diagnostic dialogue system to automate the patient information collection procedure. By exploiting medical history and conversation logic, our conversation agents, particularly the doctor agent, can pose multi-round clinical queries to effectively collect the most relevant disease diagnostic information. Moreover, benefiting from our two-stage recommendation structure, carefully designed ranking criteria, and interactive patient agent, our model is able to overcome the under-exploration and non-flexible challenges in dialogue generation. Our experiment results in a real-world medical conversation dataset with better performance than biomedical language model-based and Deepseek-based approaches. Our experiments show that we can generate clinical queries that mimic the conversation style of real doctors, with efficient fluency, professionalism, and safety, while effectively collecting relevant disease information.

Recent advancements in large language models (LLMs) have demonstrated extraordinary comprehension capabilities with remarkable breakthroughs on various vision-language tasks. However, the application of LLMs in generating reliable medical diagnostic reports remains in the early stages. Currently, medical LLMs typically feature a passive interaction model where doctors respond to patient queries with little or no involvement in analyzing medical images. In contrast, some ChatBots simply respond to predefined queries based on visual inputs, lacking interactive dialogue or consideration of medical history. As such, there is a gap between LLM-generated patient-ChatBot interactions and those occurring in actual patient-doctor consultations. To bridge this gap, we develop an LLM-based dialogue system, namely proactive multi-round vision-language interactions for computer-aided diagnosis (ProMRVL-CAD), to generate patient-friendly disease diagnostic reports. The proposed ProMRVL-CAD system allows proactive dialogue to provide patients with constant and reliable medical access via an integration of knowledge graph into a recommendation system. Specifically, we devise two generators: a Proactive Question Generator (Pro-Q Gen) to generate proactive questions that guide the diagnostic procedure and a Multi-Vision Patient-Text Diagnostic Report Generator (MVP-DR Gen) to produce high-quality diagnostic reports. Evaluating two real-world publicly available datasets, MIMIC-CXR and IU-Xray, our model has better quality in generating medical reports. We further demonstrate the performance of ProMRVL achieves robust under the scenarios with low image quality. Moreover, we have created a synthetic medical dialogue dataset that simulates proactive diagnostic interactions between patients and doctors, serving as a valuable resource for training LLM.

Near-infrared spectroscopy (NIRS) is a non-invasive monitoring technique that utilizes light transmission and absorption to continuously evaluate regional tissue oxygen saturation, delivery, and metabolism. Widely adopted in modern clinical practice, NIRS is particularly effective in assessing cerebral tissue oxygenation, enabling the early detection of impaired tissue perfusion. Grounded in the Beer–Lambert law, NIRS relies on the absorption characteristics of oxyhemoglobin and deoxyhemoglobin as the primary chromophores in biological tissues. This chapter provides a comprehensive overview of NIRS technology and its applications in biomedical and clinical fields. It begins by discussing the fundamental assumptions, advantages, and limitations of NIRS, along with the typical structure of a NIRS system. Following this, the principles and properties of NIRS are explored in depth. The chapter then delves into NIRS applications in brain monitoring, focusing on cerebral oxygenation and hemodynamics. Additionally, it examines the use of NIRS in cardiac applications, highlighting both its potential and the challenges involved. The role of machine learning in NIRS signal processing and clinical applications is also demonstrated. Finally, the integration of NIRS with other imaging modalities, including optical coherence tomography, electroencephalography, and ultrasound, is introduced to illustrate the comprehensive capabilities of NIRS in modern multi-modality clinical systems.

Few-shot multimodal dialogue intention recognition is a critical challenge in the e-commerce domainn. Previous methods have primarily enhanced model classification capabilities through post-training techniques. However, our analysis reveals that training for few-shot multimodal dialogue intention recognition involves two interconnected tasks, leading to a seesaw effect in multi-task learning. This phenomenon is attributed to knowledge interference stemming from the superposition of weight matrix updates during the training process. To address these challenges, we propose Knowledge-Decoupled Synergetic Learning (KDSL), which mitigates these issues by utilizing smaller models to transform knowledge into interpretable rules, while applying the post-training of larger models. By facilitating collaboration between the large and small multimodal large language models for prediction, our approach demonstrates significant improvements. Notably, we achieve outstanding results on two real Taobao datasets, with enhancements of 6.37% and 6.28% in online weighted F1 scores compared to the state-of-the-art method, thereby validating the efficacy of our framework.

Automatically collecting patients' health status information and medical images is of great practical importance in alleviating clinical shortages and facilitating patient-doctor communications. Despite increasing interest, few studies have investigated building reliable interactive medical dialogue systems to perform disease-related queries. To address this need, we propose a novel dialogue agent to automatically collect patients' health status information. Building upon foundation models, our model has language-reasoning and logic-performing capabilities to mimic real-world dialogues. Additionally, we fine-tune our model on real-world medical datasets to grant it clinical knowledge on disease diagnosis, ensuring its professional knowledge on posing diagnosis-related queries. We further deploy the reinforcement learning strategy to boost the model's information collection capability on necessary medical evidence, including medical visuals. Lastly, we conduct experiments on real-world datasets to validate the effectiveness of our model on health status information collection.

INTRODUCTION:Ablation of scar-related reentrant atrial tachycardia (SRRAT) involves identification and ablation of a critical isthmus. A graph convolutional network (GCN) is a machine learning structure that is well-suited to analyze the irregularly-structured data obtained in mapping procedures and may be used to identify potential isthmuses. METHODS:Electroanatomic maps from 29 SRRATs were collected, and custom electrogram features assessing key tissue and wavefront properties were calculated for each point. Isthmuses were labeled off-line. Training data was used to determine the optimal GCN parameters and train the final model. Putative isthmus points were predicted in the training and test populations and grouped into proposed isthmus areas based on density and distance thresholds. The primary outcome was the distance between the centroids of the true and closest proposed isthmus areas. RESULTS:A total of 193 821 points were collected. Thirty isthmuses were detected in 29 tachycardias among 25 patients (median age 65.0, 5 women). The median (IQR) distance between true and the closest proposed isthmus area centroids was 8.2 (3.5, 14.4) mm in the training and 7.3 (2.8, 16.1) mm in the test group. The mean overlap in areas, measured by the Dice coefficient, was 11.5 ± 3.2% in the training group and 13.9 ± 4.6% in the test group. CONCLUSION:A GCN can be trained to identify isthmus areas in SRRATs and may help identify critical ablation targets.

Being able to accurately monitor the screen exposure of young children is important for research on phenomena linked to screen use such as childhood obesity, physical activity, and social interaction. Most existing studies rely upon self-report or manual measures from bulky wearable sensors, thus lacking efficiency and accuracy in capturing quantitative screen exposure data. In this work, we developed a novel sensor informatics framework that utilizes egocentric images from a wearable sensor, termed the screen time tracker (STT), and a vision language model (VLM). In particular, we devised a multi-view VLM that takes multiple views from egocentric image sequences and interprets screen exposure dynamically. We validated our approach by using a dataset of children's free-living activities, demonstrating significant improvement over existing methods in plain vision language models and object detection models. Results supported the promise of this monitoring approach, which could optimize behavioral research on screen exposure in children's naturalistic settings.

Significance Radiofrequency ablation (RFA) procedures for atrial fibrillation frequently fail to prevent recurrence, partially due to limitations in assessing extent of ablation. Optical spectroscopy shows promise in assessing RFA lesion formation but has not been validated in conditions resembling those in vivo. Aim Catheter-based near-infrared spectroscopy (NIRS) was applied to porcine hearts to demonstrate that spectrally derived optical indices remain accurate in blood and at oblique incidence angles. Approach Porcine left atria were ablated and mapped using a custom-fabricated NIRS catheter. Each atrium was mapped first in phosphate-buffered saline (PBS) then in porcine blood. Results NIRS measurements showed little angle dependence up to 60 deg. A trained random forest model predicted lesions with a sensitivity of 81.7%, a specificity of 86.1%, and a receiver operating characteristic curve area of 0.921. Predicted lesion maps achieved a mean structural similarity index of 0.749 and a mean normalized inner product of 0.867 when comparing maps obtained in PBS and blood. Conclusions Catheter-based NIRS can precisely detect RFA lesions on left atria submerged in blood. Optical parameters are reliable in blood and without perpendicular contact, confirming their ability to provide useful feedback during in vivo RFA procedures.

Bayesian bandit algorithms with approximate Bayesian inference have been widely used in real-world applications. Despite the superior practical performance, their theoretical justification is less investigated in the literature, especially for contextual bandit problems. To fill this gap, we propose a theoretical framework to analyze the impact of approximate inference in stochastic linear bandits and conduct regret analysis on two Bayesian bandit algorithms, Linear Thompson sampling (LinTS) and the extension of Bayesian Upper Confidence Bound, namely Linear Bayesian Upper Confidence Bound (LinBUCB). We demonstrate that when applied in the presence of approximate inference, LinTS and LinBUCB can preserve their original rates of regret upper bound but with a sacrifice of larger constant terms. These results hold for general Bayesian inference approaches, assuming the inference error measured by two different α-divergences is bounded. Additionally, by introducing a new definition of well-behaved distributions, we show that LinBUCB expedites the regret rate of LinTS from Õ(d^3/2√(T)) to Õ(d√(T)), matching the minimax optimal rate. To our knowledge, this work provides the first regret bounds in the setting of stochastic linear bandits with bounded approximate inference errors.

Deep learning has revolutionized medical image analysis, promising to significantly improve the precision of diagnoses and therapies through advanced segmentation methods. However, the efficacy of deep neural networks is often compromised by the prevalence of imperfect medical labels, while acquiring large-scale, accurately labeled data remains a prohibitive challenge. To address the imperfect label issue, we introduce a novel learning framework that iteratively optimizes both a neural network and its label set to enhance segmentation accuracy. This framework operates through two steps: initially, it robustly trains on a dataset with label noise, distinguishing between clean and noisy labels, and subsequently, it refines noisy labels based on high-confidence predictions from the robust network. By applying this method, not only is the network trained more effectively on imperfect data, but the dataset is progressively cleaned and expanded. Our evaluations are conducted on retina Optical Coherence Tomography datasets using U-Net and SegNet architectures, and demonstrate substantial improvements in segmentation accuracy and data quality, advancing the capabilities of weakly supervised segmentation in medical imaging.

Atrial fibrillation is a common and potentially lethal arrhythmia, yet catheter radiofrequency ablation (RFA), a mainstay of treatment, frequently fails to provide long-term remission. We present a catheter capable of near-infrared diffuse reflectance spectroscopy, with a source fiber delivering broadband light and a detection fiber whose light is sent to a spectrometer. Separate catheters have been fabricated with different source-detection separations, yielding spectra sensitive to different optical properties of the underlying tissue. Optical indices have been developed from benchtop measurements to distinguish the spectral signatures of different cardiac substrates. These measurements will equip clinicians with intraprocedural feedback to improve RFA effectiveness.

Segment anything model (SAM) has emerged as the leading approach for zero-shot learning in segmentation tasks, offering the advantage of avoiding pixel-wise annotations. It is particularly appealing in medical image segmentation, where the annotation process is laborious and expertise-demanding. However, the direct application of SAM often yields inferior results compared to conventional fully supervised segmentation networks. An alternative approach is to use SAM as the initial stage to generate pseudo labels for further network training. However, the performance is limited by the quality of pseudo labels. In this paper, we propose a novel label correction framework to push the boundary of SAM-based segmentation. Our model utilizes a label quality evaluation module to distinguish between noisy labels and clean labels. This enables the correction of the noisy labels using an uncertainty-based self-correction module, thereby enriching the clean training set. Finally, we retrain the segmentation network with updated labels to optimize its weights for future predictions. One key advantage of our model is its ability to train deep networks using SAM-generated pseudo labels without relying on a set of expert-level annotations while attaining good segmentation performance. We demonstrate the effectiveness of our proposed model on three public datasets, indicating its ability to improve segmentation accuracy and outperform baseline methods in label correction.

Automatically identifying the structural substrates underlying cardiac abnormalities can potentially provide real-time guidance for interventional procedures. With the knowledge of cardiac tissue substrates, the treatment of complex arrhythmias such as atrial fibrillation and ventricular tachycardia can be further optimized by detecting arrhythmia substrates to target for treatment (i.e., adipose) and identifying critical structures to avoid. Optical coherence tomography (OCT) is a real-time imaging modality that aids in addressing this need. Existing approaches for cardiac image analysis mainly rely on fully supervised learning techniques, which suffer from the drawback of workload on labor-intensive annotation process of pixel-wise labeling. To lessen the need for pixel-wise labeling, we develop a two-stage deep learning framework for cardiac adipose tissue segmentation using image-level annotations on OCT images of human cardiac substrates. In particular, we integrate class activation mapping with superpixel segmentation to solve the sparse tissue seed challenge raised in cardiac tissue segmentation. Our study bridges the gap between the demand on automatic tissue analysis and the lack of high-quality pixel-wise annotations. To the best of our knowledge, this is the first study that attempts to address cardiac tissue segmentation on OCT images via weakly supervised learning techniques. Within an in-vitro human cardiac OCT dataset, we demonstrate that our weakly supervised approach on image-level annotations achieves comparable performance as fully supervised methods trained on pixel-wise annotations.

Radiofrequency ablation (RFA) is a minimally invasive procedure that is commonly used for the treatment of atrial fibrillation. However, it is associated with a significant risk of arrhythmia recurrence and complications owing to the lack of direct visualization of cardiac substrates and real-time feedback on ablation lesion transmurality. Within this manuscript, we present an automated deep learning framework for in vivo intracardiac optical coherence tomography (OCT) analysis of swine left atria. Our model can accurately identify cardiac substrates, monitor catheter-tissue contact stability, and assess lesion transmurality on both OCT intensity and polarization-sensitive OCT data. To the best of our knowledge, we have developed the first automatic framework for in vivo cardiac OCT analysis, which holds promise for real-time monitoring and guidance of cardiac RFA therapy..

Ablation of scar-related reentrant VT is challenging, as the rhythm is hemodynamically unstable. Durable freedom from VT requires ablation of protected isthmus areas. Although substrate-based mapping strategies are commonly used, ablation is still done during tachycardia in some institutions. Recurrence is common and improved strategies to identify ablation targets are needed. Recently, machine learning (ML) has been more commonly used in medicine. However common ML techniques use input that is ordered in a regularly spaced matrix, unlike the data collected during ablations.

Deep learning has been recently studied to generate high-quality prediction intervals(PIs)for uncertainty quantification in regression tasks,including recent applications in simulation metamodeling.The high-quality criterion requires PIs to be as narrow as possible,whilst maintaining a pre-specified level of data(marginal)coverage.However,most existing works for high-quality PIs lack accurate information on conditional coverage,which may cause unreliable predictions if it is significantly smaller than the marginal coverage.To address this problem,we propose an end-to-end framework which could output high-quality PIs and simultaneously provide their conditional coverage estimation.In doing so,we design a new loss function that is both easy-to-implement and theoretically justified via an exponential concentration bound.Our evaluation on real-world benchmark datasets and synthetic examples shows that our approach not only achieves competitive results on high-quality PIs in terms of average PI width,but also accurately estimates conditional coverage information that is useful in assessing model uncertainty.