Reliable recognition and localization of surgical instruments in endoscopic video recordings are foundational for a wide range of applications in computer- and robot-assisted minimally invasive surgery (RAMIS), including surgical training, skill assessment, and autonomous assistance. However, robust performance under real-world conditions remains a significant challenge. Incorporating surgical context - such as the current procedural phase - has emerged as a promising strategy to improve robustness and interpretability. To address these challenges, we organized the Surgical Procedure Phase, Keypoint, and Instrument Recognition (PhaKIR) sub-challenge as part of the Endoscopic Vision (EndoVis) challenge at MICCAI 2024. We introduced a novel, multi-center dataset comprising thirteen full-length laparoscopic cholecystectomy videos collected from three distinct medical institutions, with unified annotations for three interrelated tasks: surgical phase recognition, instrument keypoint estimation, and instrument instance segmentation. Unlike existing datasets, ours enables joint investigation of instrument localization and procedural context within the same data while supporting the integration of temporal information across entire procedures. We report results and findings in accordance with the BIAS guidelines for biomedical image analysis challenges. The PhaKIR sub-challenge advances the field by providing a unique benchmark for developing temporally aware, context-driven methods in RAMIS and offers a high-quality resource to support future research in surgical scene understanding.

Recent multimodal large language models (MLLMs) have substantially advanced video understanding, yet long-form video QA remains challenging under fixed input token budgets, where uniform sampling can be inefficient for evidence localization. We propose ReQuest , an uncertainty-driven, question-adaptive keyframe selection pipeline that aligns question intent with relevant video content through selective computation. ReQuest integrates (i) a lightweight question-aware selector distilled from MLLM-generated supervision, (ii) Re-thinking Routing that triggers additional inference only when the model is uncertain with a length-adaptive criterion, and (iii) uncertainty-guided adaptive non-maximum suppression that selects temporally diverse frames while adjusting spacing based on question difficulty. As a plug-andplay method, ReQuest improves long-video QA without modifying or fine-tuning the underlying MLLM. Experiments on Video-MME, MLVU, and LongVideoBench demonstrate consistent accuracy gains with competitive computational cost, with particularly strong improvements in medium and long video regimes.

The presence of adversarial examples in the physical world poses significant challenges to the deployment of Deep Neural Networks in safety-critical applications such as autonomous driving. Most existing methods for crafting physical-world adversarial examples are ad-hoc, relying on temporary modifications like shadows, laser beams, or stickers that are tailored to specific scenarios. In this paper, we introduce a new class of physical-world adversarial examples, AdvWT, which draws inspiration from the naturally occurring phenomenon of `wear and tear', an inherent property of physical objects. Unlike manually crafted perturbations, `wear and tear' emerges organically over time due to environmental degradation, as seen in the gradual deterioration of outdoor signboards. To achieve this, AdvWT follows a two-step approach. First, a GAN-based, unsupervised image-to-image translation network is employed to model these naturally occurring damages, particularly in the context of outdoor signboards. The translation network encodes the characteristics of damaged signs into a latent `damage style code'. In the second step, we introduce adversarial perturbations into the style code, strategically optimizing its transformation process. This manipulation subtly alters the damage style representation, guiding the network to generate adversarial images where the appearance of damages remains perceptually realistic, while simultaneously ensuring their effectiveness in misleading neural networks. Through comprehensive experiments on two traffic sign datasets, we show that AdvWT effectively misleads DNNs in both digital and physical domains. AdvWT achieves an effective attack success rate, greater robustness, and a more natural appearance compared to existing physical-world adversarial examples. Additionally, integrating AdvWT into training enhances a model's generalizability to real-world damaged signs.

Bimanual robot manipulation for long-horizon (LH) tasks is crucial for the practical use of humanoids, but it struggles with robust planning and generalization. Approaches based on Task and Motion Planning (TAMP), transformers, and Large Language Models (LLMs) suffer from critical limitations, including costly human demonstrations, task planner hallucination, and unsatisfactory generalization performance. To address these challenges, this paper introduces the Multi-modal Affordance Planner with Temporal-Context Action Policy (MAP-TCA), a novel hierarchical framework that learns and performs diverse bimanual long-horizon (LH) tasks by generating action plans from MAP. The MAP-TCA consists of a planner based on Bimanual Robot Manipulation Retrieval-Augmented Generation (Bi-RAG)-enhanced Large-Language Model (LLM) and a low-level Temporal Context Action Policy (TCA). With multimodal inputs including vision, language, and affordance for primitive action demonstration, Bi-RAG generates a Primitive Action (PA)-specific embedded space. Then, MAP generates LH plans, LH demonstrations, and reward functions within the PA-specific embedded space, thereby mitigating hallucinations and reducing training cost. The generated plan, demos, and rewards then guide TCA, which learns the LH tasks via behavior cloning (BC) and online fine-tuning. We demonstrate that the proposed MAP-TCA achieves an average success rate of 86.75%, comparable to a baseline model, TCA, which is trained extensively on direct human demonstrations and manually designed rewards. Our work presents a scalable and generalizable solution for complex bimanual LH manipulation, significantly reducing the dependency on human supervision.

Branch retinal vein occlusion (BRVO) can cause persistent visual impairment, and predicting long-term best-corrected visual acuity (BCVA) after anti-vascular endothelial growth factor treatment remains clinically challenging. This retrospective proof-of-concept study developed multimodal neural networks to predict 12-month BCVA classes using retinal images and clinical metadata from treatment-naive BRVO eyes. The best internal model used OCT-horizontal scans, OCTA images, baseline BCVA, central subfield thickness, age, and sex. We evaluated performance using adjacent accuracy, exact accuracy, and mean absolute error under five-fold cross-validation, and we analyzed attribution localization using Pathway Attribution. We additionally performed a supplementary exploratory cross-disease feasibility analysis using an available diabetic macular edema (DME) OCT cohort with 24-month visual acuity outcomes. This analysis was interpreted only as supplementary exploratory feasibility evidence and not as disease-matched BRVO external validation. Overall, the results suggest that multimodal imaging and clinical metadata may provide complementary prognostic information after BRVO, but the modest predictive performance, retrospective single-center design, lack of disease-matched external validation, and limited attribution reliability require cautious interpretation and further prospective validation.

Uncertainty estimation is an important area of research in the deep learning field by offering numerous potential applications. Selective prediction is one of the possible applications that focuses on rejecting less confident samples to enhance the overall performance and reliability of accepted samples. In the context of segmentation, selective segmentation aims to allow the model to abstain from assigning labels to uncertain pixels. While selective classification has received considerable attention, only a few studies have explored selective segmentation. In this paper, we introduce a novel approach that addresses this gap by incorporating uncertainty learning during model training. Specifically, we propose a new training scheme that encourages the model to identify and flag uncertain pixels in a real-time manner. Comparative experiments on three public medical segmentation datasets demonstrate the effectiveness of our method. Our approach achieves higher Dice scores across multiple coverage levels, surpassing the performance of existing approaches in the field.

Purpose: Vision-language models (VLMs) have shown promising performance in surgical visual question answering (VQA). However, existing surgical VQA datasets often contain linguistic shortcuts, where question phrasing implicitly constrains the answer space. In safety-critical surgical settings, it remains unclear whether reported performance reflects visual understanding or reliance on such linguistic shortcuts. Methods: We introduce SurgCheck, a diagnostic benchmark for quantifying linguistic shortcut reliance in surgical VQA. SurgCheck employs a paired-question design in which each surgical frame is associated with an original question containing entity names and a less-biased counterpart that removes these names while preserving identical visual content and ground-truth answers. The resulting performance gap provides a diagnostic signal of shortcut reliance. To ensure that the less-biased question remains well defined even without entity names, four grounding cues are incorporated: bounding box, arrow, spatial position, and periphrasis. We evaluate both general-purpose and surgical-specific VLMs under zero-shot and fine-tuned settings on SurgCheck. To evaluate open-ended zero-shot responses, we introduce an LLM-as-a-judge evaluation protocol. Results: Using SurgCheck, we observe consistent performance degradation on less-biased questions across five VLMs, despite identical visual inputs. Text-only ablation reveals minimal performance drops for action and target prediction, indicating that action and target prediction are largely driven by linguistic shortcuts rather than visual reasoning. Conclusion: SurgCheck provides a controlled diagnostic framework that exposes failure modes masked by linguistic bias in existing surgical VQA benchmarks. Our findings demonstrate that strong benchmark performance does not necessarily imply faithful visual understanding, underscoring the need for bias-aware evaluation in surgical VQA. SurgCheck is publicly available at https://github.com/ailab-kyunghee/SurgCheck.

Although recent trajectory prediction and end-to-end autonomous driving methods improve robustness in urban environments, they still lack meaningful controllability. Existing benchmarks either provide no persona-conditioned annotations or support only a single urgency spectrum (i.e., emergency, normal, relaxed), which cannot distinguish personas that share the same urgency level but require different driving dynamics. To address this, we propose (i) the Persona-Conditioned Trajectory (PCT) dataset, which decomposes driving personas along two axes, Temporal Urgency and Ride Comfort, and combines three levels of each to form a grid of nine personas, each paired with natural-language descriptions and trajectories, and (ii) PersonaDrive, a framework that can learn driving personas from language and can generate persona-specific trajectories. PersonaDrive incorporates Persona-Conditioned Anchor Transform (PCAT), which hierarchically reshapes anchors along both axes, and Persona-Conditioned Multi-Modal Fusion (PCMF) for BEV-level persona fusion. Training is supervised by a Hierarchical Guide Loss enforcing axis-aligned physical orderings and an Axis-Decomposed Diversity Loss preventing diagonal mode collapse. Experimental results show that PersonaDrive consistently improves over the compared baselines across multi-dimensional scenarios. The code and PCT dataset are available at https://github.com/VisualAIKHU/PersonaDrive

Domains shift originated from differences in devices or patients in the medical field, poses a significant challenge when applying pre-trained models to clinical applications. To tackle this challenge, domain adaptation methods have been explored. However, most existing methods are designed for a single target domain adaptation or require sharing all target domain data for adaptation, which is infeasible in the medical field due to privacy issues. In this paper, we propose a novel unsupervised multi-target domain adaptation method without requiring data sharing. To this end, we introduce an additional signal, termed Adaptogen-Perturbation (AP) optimized to bridge the gap between the source and target domains. The optimized AP is injected into the latent feature and facilitates the adaptation of the pre-trained model to the target domain. Moreover, we propose a Spectral/Geometric Consistency learning framework to optimize the AP in an unsupervised manner. This promotes consistent predictions across two types of transformations: geometric and frequency-space spectral transformations, enhancing robustness to both variations. Extensive experiments with multiple medical segmentation datasets demonstrate the effectiveness of APs.

Knowledge graphs are increasingly important in radiology for representing factual clinical information and supporting downstream applications such as decision support, information retrieval, and structured reporting. However, generating radiology-specific knowledge graphs remains challenging due to the specialized vocabulary used in radiology reports, the scarcity of domain-annotated datasets, and the predominance of unimodal approaches that rely solely on text. To develop and evaluate a multimodal Vision-Language-Model (VLM) framework capable of generating radiology knowledge graphs using both radiographic images and the corresponding reports. We designed a VLM-based knowledge graph generation framework that integrates radiology images and free-text reports through instruction tuning and visual instruction tuning. The model is optimized for long-context radiology reports and structured triplet extraction. Its performance was compared with existing unimodal baselines on benchmark datasets. Our multimodal VLM-KG (MIMIC) demonstrated the strongest overall performance across standard NLG metrics, achieving the highest BLEU scores (BLEU-1: 54.98, BLEU-2: 49.65, BLEU-3: 46.12, BLEU-4: 43.29), substantially outperforming all unimodal baselines, including the BERT-based Dygiee++ model. This improvement highlights the effectiveness of multimodal learning, where the integration of visual and linguistic information enhances contextual understanding in text generation. Although Dygiee++ achieved a comparable ROUGE-L score (56.49), VLM-KG (MIMIC) provided markedly higher BLEU scores, indicating stronger n-gram overlap and more accurate triplet generation. VLM-KG (MIMIC) also achieved a competitive ROUGE-L score of 54.69, slightly lower than LLM-KG (MIMIC) (56.53), suggesting that while multimodal features improve precision, they may introduce minor variability in generated outputs. Additionally, LLM-KG (MIMIC) consistently outperformed LLM-KG (IU) across all metrics (e.g., BLEU-3: 35.96 vs. 18.02), underscoring the advantages of training on a large-scale, domain-specific dataset. This study presents the first multimodal VLM-driven approach for radiology knowledge graph generation. By leveraging both images and reports, the framework overcomes limitations of previous text-only systems and provides a more comprehensive foundation for medical knowledge representation and downstream radiology informatics applications.Vision Language Models; Large Language Models; Knowledge Graph; Radiology; Multimodal AI; Medical NLP

Change captioning aims to describe changes between a pair of images. However, existing works rely on visual features alone, which often fail to capture subtle but meaningful changes because they lack the ability to represent explicitly structured information such as object relationships and compositional semantics. To alleviate this, we present CORTEX (COmpositional Reasoning-aware TEXt-guided), a novel framework that integrates complementary textual cues to enhance change understanding. In addition to capturing cues from pixel-level differences, CORTEX utilizes scene-level textual knowledge provided by Vision Language Models (VLMs) to extract richer image text signals that reveal underlying compositional reasoning. CORTEX consists of three key modules: (i) an Image-level Change Detector that identifies low-level visual differences between paired images, (ii) a Reasoning-aware Text Extraction (RTE) module that use VLMs to generate compositional reasoning descriptions implicit in visual features, and (iii) an Image-Text Dual Alignment (ITDA) module that aligns visual and textual features for fine-grained relational reasoning. This enables CORTEX to reason over visual and textual features and capture changes that are otherwise ambiguous in visual features alone.

Registering cone-beam computed tomography (CBCT) to planning computed tomography (CT) in the head-and-neck region is challenging due to the reduced image quality of CBCT and posture-related anatomical deformation, particularly in the cervical spine. We propose a deformation-informed unsupervised reference-augmented synthesis framework that generates an aligned CT (aCT) directly in the CBCT coordinate space from misaligned CBCT–CT image pairs. Unlike conventional deformable registration approaches, the proposed method reformulates image alignment as deformation-informed synthesis. The model integrates a spatial transformer with a Deformation-Informed Efficient Attention (DEA) module that transfers CT features (“keys” and “values”) to spatially corresponding CBCT locations (“queries”). Memory-efficient attention from xFormers enables training within standard GPU memory constraints. Additionally, we introduce a multi-objective loss function and a modular CT style-transfer pre-conditioning stage that is practically essential for narrowing the severe CBCT–CT modality gap, thereby reducing the modality discrepancy before alignment. Experiments on the public SynthRAD2023 head-and-neck dataset demonstrate improved anatomical correspondence (NMI: 0.53 ± 0.12, GC: 0.82 ± 0.09) and dosimetric accuracy comparable to reference CT under surrogate planning conditions (Dose MAE: 0.002 ± 0.001, Gamma pass rate: 90.01 ± 18.2). These findings suggest that reference-guided synthesis can mitigate alignment difficulties in anatomically deformable regions. The implementation is publicly available at https://github.com/A-shazli/Reference_Augmented_Synthesis.

Federated Learning (FL) enables decentralized model training across multiple clients without exposing private data, making it ideal for privacy-sensitive applications. However, in real-world FL scenarios, clients often hold data from distinct domains, leading to severe domain shift and degraded global model performance. To address this, prototype learning has been emerged as a promising solution, which leverages class-wise feature representations. Yet, existing methods face two key limitations: (1) Existing prototype-based FL methods typically construct a single global prototype per class by aggregating local prototypes from all clients without preserving domain information. (2) Current feature-prototype alignment is domain-agnostic, forcing clients to align with global prototypes regardless of domain origin. To address these challenges, we propose Federated Domain-Aware Prototypes (FedDAP) to construct domain-specific global prototypes by aggregating local client prototypes within the same domain using a similarity-weighted fusion mechanism. These global domain-specific prototypes are then used to guide local training by aligning local features with prototypes from the same domain, while encouraging separation from prototypes of different domains. This dual alignment enhances domain-specific learning at the local level and enables the global model to generalize across diverse domains. Finally, we conduct extensive experiments on three different datasets: DomainNet, Office-10, and PACS to demonstrate the effectiveness of our proposed framework to address the domain shift challenges. The code is available at https://github.com/quanghuy6997/FedDAP.

Intracranial hemorrhage (ICH) is among the most lethal forms of stroke, where timely and accurate diagnosis is critical for patient survival. CNN-based computer-aided diagnosis systems have been applied for ICH subtype detection and classification, but their performances remain insufficient for reliable clinical deployment yet. In this study, we propose a novel integration of Real-Time Detection Transformer (RT-DETR) into a reinforcement learning (RL) framework based on Proximal Policy Optimization (PPO). To enhance the diagnostic capability for ICH subtype detection and classification, an adaptive reward mechanism is also proposed that jointly optimizes detection and classification objectives through an actor-critic learning paradigm. Our experimental results on a public database demonstrate that our proposed RL enhanced RT-DETR outperforms the baseline RT-DETR and YOLO variants. In particular, RT-DETR-PPO achieved 6.7% and 6.0% gains in precision under the detection-oriented criterion (IoU>0.7) for the challenging ICH subtypes subarachnoid hemorrhage (SAH) and subdural hemorrhage (SDH), respectively. Under the classification-oriented criterion (IoU>0.1), RT-DETR-PPO achieved $5.0 \%$ and $4.2 \%$ gains in precision for SAH and SDH, respectively.

Haptic texture modeling is essential for immersive environments, yet accurate texture vibration synthesis remains challenging due to high-frequency nature and strong dependence on interaction parameters, such as speed and force. Traditional piecewise approaches handle this variability through algorithmic segmentation of signals into stationary components, followed by labeling across discrete contact conditions. However, the resulting segmentation overhead and postmodeling validation create a procedural bottleneck, limiting their ability to represent continuous user interactions. Recent deep learning methods eliminate explicit segmentation by learning continuous mappings, but their high computational cost hinders real-time deployment. To address these limitations, a lightweight Fourier-enhanced Transformer encoder network is proposed that eliminates segmentation and enables real-time texture rendering while maintaining high reconstruction fidelity. The model integrates a transformer encoder with the Fourier transform to process uniform sliding window inputs, separating spectral magnitude and phase in a Fourier block to capture texture-dependent frequency structure, while temporal features are extracted in a compact encoder for one-step prediction at approximately 1 kHz. The proposed approach was evaluated on a diverse set of textures, achieving strong reconstruction accuracy (time domain: mean absolute error 0.149; spectral domain: goodness-of-fit criterion 92.15%). A psychophysical experiment further confirms its perceptual effectiveness against existing methods when comparing real and synthesized textures.

Surgical Instrument Segmentation (SIS) is crucial for enhancing surgical efficiency and safety but remains challenging due to visual similarities among instruments. Recent advancements, especially those leveraging the Segment Anything Model (SAM) due to its strong capability to generate accurate object masks, have improved segmentation accuracy yet still struggle with subtle semantic distinctions. We propose TeCIS (Text-enhanced Cross-modal Instrument Segmentation), a novel method integrating textual descriptions directly into the segmentation process. TeCIS empolys a cross-attention mechanism in the segmentation mask decoder, leveraging detailed instrument descriptions to convey semantic information that helps the model accurately identify and segment each surgical tool. Evaluations on EndoVis2017 and EndoVis2018 datasets show TeCIS substantially reduces semantic confusion and surpasses existing SAM-based methods, highlighting the effectiveness of integrating text for improved surgical segmentation.

Dual-arm robots hold significant potential for performing medical assistive tasks in healthcare environments. However, executing such diverse and complex tasks requires advanced dual-arm robot intelligence, which faces substantial challenges due to multi-agent interactions in sequential long- horizon (LH) actions. This study introduces a novel multi-agent reinforcement learning approach, termed Counterfactual Multi-Agent Demo Augmented Policy Gradient (COMA-DAPG), to learn and perform LH medical assistive tasks for dual-arm robots. The proposed COMA-DAPG integrates a counterfactual critic network and demonstration-augmented policy gradient (DAPG) with three designed reward functions. Our experimental results demonstrate that COMA-DAPG outperforms each COMA and DAPG with over 25% improvement in average success rate across three LH tasks.Clinical Relevance—COMA-DAPG addresses key challenges in dual-arm robotics, such as credit assignment, gradient variance, and collision avoidance, to enable precise, cooperative execution of complex medical tasks, enhancing reliability and efficiency in clinical care settings.

This paper presents Temporal-Context Planner with Transformer Reinforcement Learning (TCP-TRL), a novel robot intelligence capable of learning and performing complex bimanual lifecare tasks without using long-horizon (LH) human demonstrations and manually designed reward functions. TCPTRL combines a Large Language Model (LLM) enhanced by a built-in Retrieval-Augmented Generation (RAG) as TCP and a Transformer-Based reinforcement learning (TRL) framework. TCP serves as a high-level planner, translating textual task descriptions into temporal action contexts. These contexts then guide TRL to generate precise joint actions needed for bimanual robot arms. TCP use a built-in RAG, which leverages a compact dataset of 10 primitive actions (PAs) and task-specific information to guide LLM for LH task planning. The high-level planner TCP generates structured LH plans comprising task-sequenced PAs and sparse reward functions, replacing human demonstrations and human-designed reward functions. Then, under RL, the generated LH plan and sparse reward functions guide the TRL network through behavior cloning and online fine-tuning training, adding robustness and temporal dependencies. Evaluated on four lifecare LH tasks, the TCPTRL achieved an 81.86% average success rate, matching the performance of a state-of-the-art TRL model trained with human demonstrations and manually designed reward functions. The TCP-TRL provides a scalable human intervention-free solution and overcomes challenges in bimanual LH manipulation tasks, such as LLM hallucinations, dependence on large-scale datasets, and temporal context preservation.Clinical Relevance— TCP-TRL is a novel robotic approach specifically designed for learning and performing long-horizon lifecare tasks with dual-arm robots in hospital and homecare environments.

Implicit Neural Representation (INR) models signals as a function parameterized by a neural network, providing a continuous and concise alternative. In this study, we propose Semantic-Guided INR (SG-INR), a segmentation-based softguided approach designed to enhance image reconstruction by improving structural awareness during the learning process of 2D image-based INR. Existing INR models have introduced various methods to address spectral bias, but they still suffer from a tendency to favor low-frequency components. To address this limitation, we utilize a pre-trained semantic segmentation model to estimate structural regions in the image and generate a weighted guide mask. This mask selectively emphasizes semantically important regions during the optimization process. The proposed loss function mixes the original reconstruction loss with the structural awareness loss to give higher importance to key regions while allowing easy integration with existing INR frameworks. This proposal presents a novel approach to reconstructing high-resolution images and shows potential for advancement in quantitative metrics such as Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM).

Recent advances in video object segmentation (VOS) highlight its potential across various applications. Semi-supervised VOS aims to segment target objects in video frames based on annotations from the initial frame. Collecting a large-scale video segmentation dataset is challenging, which could induce noisy labels. However, it has been overlooked and most of the research efforts have been devoted to training VOS models by assuming the training dataset is clean. In this study, we first explore the effect of VOS models under noisy labels in the training dataset. To investigate the effect of noisy labels, we simulate the noisy annotations on DAVIS 2017 and YouTubeVOS datasets. Experiments show that the traditional training strategy is vulnerable to noisy annotations. To address this issue, we propose a novel noise-robust training method, named SMART (Spatial Mask-based Adaptive Robust Training), which is designed to train models effectively in the presence of noisy annotations. The proposed method employs two key strategies. Firstly, the model focuses on the common spatial areas from clean knowledge-based predictions and annotations. Secondly, the model is trained with adaptive balancing losses based on their reliability. Comparative experiments have demonstrated the effectiveness of our approach by outperforming other noise handling methods over various noise degrees.