Camera-only 3D object detection has emerged as a cost-effective and scalable alternative to LiDAR for autonomous driving, yet existing methods primarily prioritize overall performance while overlooking the severe long-tail imbalance inherent in real-world datasets. In practice, many rare but safety-critical categories such as children, strollers, or emergency vehicles are heavily underrepresented, leading to biased learning and degraded performance. This challenge is further exacerbated by pronounced inter-class ambiguity (e.g., visually similar subclasses) and substantial intra-class diversity (e.g., objects varying widely in appearance, scale, pose, or context), which together hinder reliable long-tail recognition. In this work, we introduce SemLT3D, a Semantic-Guided Expert Distillation framework designed to enrich the representation space for underrepresented classes through semantic priors. SemLT3D consists of: (1) a language-guided mixture-of-experts module that routes 3D queries to specialized experts according to their semantic affinity, enabling the model to better disentangle confusing classes and specialize on tail distributions; and (2) a semantic projection distillation pipeline that aligns 3D queries with CLIP-informed 2D semantics, producing more coherent and discriminative features across diverse visual manifestations. Although motivated by long-tail imbalance, the semantically structured learning in SemLT3D also improves robustness under broader appearance variations and challenging corner cases, offering a principled step toward more reliable camera-only 3D perception.

Whole-slide multiplex brain tissue images for spatial proteomics are massive, information-dense, and challenging to analyze. We present mVISE, an interactive multiplex visual search engine that offers an alternative programming-free query-driven analysis method based on retrieving and profiling communities of similar cells, proximal cell pairs, and multicellular niches. The retrievals can be used for exploratory cell and tissue analysis, delineating brain regions and cortical layers, profiling and comparing brain regions/sub-regions/sub-layers, etc. mVISE is enabled by multiplex encoders that seamlessly integrate visual cues across imaging channels overcoming the limitations of current foundation models. We train separate encoders to learn each facet of tissue including cell morphologies, spatial protein expression (chemoarchitecture), cell arrangements (cytoarchitecture), and wiring patterns (myeloarchitecture) from a set of user-defined molecular marker panels, without the need for human annotations or intervention, and with visual confirmation of successful learning. Multiple encoders can be combined logically to drive specialized searches. We validated mViSE’s ability to retrieve single cells, proximal cell pairs, tissue patches, delineate cortical layers, brain regions, and sub-regions. mVISE is disseminated as an open-source QuPath plug-in tool.

Current multimodal large language models (MLLMs) cannot effectively utilize eye-gaze information for video understanding, even when gaze cues are supplied via visual overlays or text descriptions. We introduce GazeQwen, a parameter efficient approach that equips an open-source MLLM with gaze awareness through hidden-state modulation. At its core is a compact gaze resampler ( 1-5 M trainable parameters) that encodes V-JEPA 2.1 video features together with fixation-derived positional encodings and produces additive residuals injected into selected LLM decoder layers via forward hooks. An optional second training stage adds low-rank adapters (LoRA) to the LLM for tighter integration. Evaluated on all 10 tasks of the StreamGaze benchmark, GazeQwen reaches 63.9

Generative diffusion models are increasingly used for medical imaging data augmentation, but text prompting cannot produce causal training data. Re-prompting rerolls the entire generation trajectory, altering anatomy, texture, and background. Inversion-based editing methods introduce reconstruction error that causes structural drift. We propose PolypSteer, a training-free activation-steering framework for endoscopic synthesis. PolypSteer identifies a pathology vector for each contrastive prompt pair in the cross-attention layers of a diffusion transformer. At inference time, it steers image activations along this vector, generating counterfactual pairs from scratch where the only difference is the steered concept. All other structure is preserved by construction. We evaluate PolypSteer across three experiments on Kvasir v3 and HyperKvasir. On counterfactual generation across three clinical concept pairs, PolypSteer achieves flip rates of 0.800, 0.925, and 0.950, outperforming the best inversion-based baseline in both concept flip rate and structural preservation. On dye disentanglement, PolypSteer achieves 75

Stain variability is a pervasive source of distribution shift and potential shortcut learning in renal pathology AI. We ask whether lupus nephritis glomerular lesion classifiers exploit stain as a shortcut, and how to mitigate such bias without stain or site labels. We curate a multi-center, multi-stain dataset of 9,674 glomerular patches (224×224) from 365 WSIs across three centers and four stains (PAS, H E, Jones, Trichrome), labeled as proliferative vs. non-proliferative. We evaluate Bayesian CNN and ViT backbones with Monte Carlo dropout in three settings: (1) stain-only classification; (2) a dual-head model jointly predicting lesion and stain with supervised stain loss; and (3) a dual-head model with label-free stain regularization via entropy maximization on the stain head. In (1), stain identity is trivially learnable, confirming a strong candidate shortcut. In (2), varying the strength and sign of stain supervision strongly modulates stain performance but leaves lesion metrics essentially unchanged, indicating no measurable stain-driven shortcut learning on this multi-stain, multi-center dataset, while overly adversarial stain penalties inflate predictive uncertainty. In (3), entropy-based regularization holds stain predictions near chance without degrading lesion accuracy or calibration. Overall, a carefully curated multi-stain dataset can be inherently robust to stain shortcuts, and a Bayesian dual-head architecture with label-free entropy regularization offers a simple, deployment-friendly safeguard against potential stain-related drift in glomerular AI.

Adapting vision transformer (ViT) foundation models with parameter-efficient fine-tuning (PEFT) has become increasingly popular in medical imaging, enabling efficient adaptation while updating only a small subset of parameters. However, existing PEFT methods process tokens independently, overlooking cross-token dependencies and limiting their ability to capture global contextual information. To address these challenges, we propose FreqFiT, a novel Frequency-based Fine-Tuning module inserted between ViT blocks to enhance model adaptability. FreqFiT is effective and seamlessly integrates with existing PEFT methods to improve their performance. We evaluate FreqFiT across 2D and 3D medical imaging datasets, such as PAPILA, HAM10000, ADNI-1.5T, and COVID-CT-MD. It improves accuracy 9 ≤ 1.2% of full fine-tuning parameters, FreqFiT achieves state-of-the-art medical imaging adaptation efficiently. The source code is available here .

Medical eye-tracking data is an important information source for understanding how radiologists visually inter-pret medical images. This information not only improves the accuracy of deep learning models for X-ray analysis but also their interpretability, enhancing transparency in decision-making. However, the current eye-tracking data is dispersed, unprocessed, and ambiguous, making it difficult to derive meaningful insights. Therefore, there is a need to create a new dataset with more focus and purposeful eye-tracking data, improving its utility for diagnostic applications. In this work, we propose a refinement method inspired by the target-present visual search challenge: there is a specific finding and fixations are guided to locate it. After re-fining the existing eye-tracking datasets, we transform them into a curated visual search dataset, called Gazesearch. specifically for radiology findings, where each fixation sequence is purposefully aligned to the task of locating a particular finding. Subsequently, we introduce a scan path prediction baseline, called ChestSearch, specifically tailored to Gazesearch. Finally, we employ the newly introduced Gazesearch as a benchmark to evaluate the performance of current state-of-the-art methods, offering a comprehensive assessment for visual search in the medical imaging domain. Code is available at https://github.com/UARK-AICV/GazeSearch.

There are no established models to predict the varied response to intensified immunosuppression in steroid-resistant nephrotic syndrome. Multimodal machine learning, integrating clinical and genetic data with nephropathology imaging, holds great promise to deliver such classifiers with perfect reproducibility. Here, we present such a theranostic classifier based on a multi-centric dataset. The PodoNet cohort with n = 201 biopsies was collected from 14 European centres with large domain shifts in whole slide images. The clinical input data contained 19 parameters including mutation (yes/no) and eGFR at biopsy; the ground truth theranostic endpoint was treatment response as no remission (n = 114), partial (n = 42) and complete (n = 45), defined by serum albumin and proteinuria. We trained our proprietary multimodal MorphSet++ architecture in a weakly supervised fashion. MorphSet++ integrates the clinical data vector from a shallow network at various stages with the transformer-based deep network analysing the nephropathology imaging data. Results are given as mean after 5-fold internal cross-validation. Mean AUC was .80 for complete, .77 for partial and .81 and xx for no remission. Mean positive predictive value was 89.72, 89.83, 89.27, mean negative predictive value was 94.83, 93.43 and 97.59; mean sensitivity was 84.21, 67.09, 98.47; mean specificity was 96.77, 98.40, 83.94; mean F1 Score was 86.88, 76.81, 93.65; mean accuracy was 84.21, 67.09, 98.47; mean balanced accuracy was 90.49, 82.75, 91.21 for no, partial and complete remission, respectively. Performance was best with both clinical data and histology as input. Our MorphSet++ architecture shows promising results as a theranostic tool, predicting response to immunosuppression in SRNS. MorphSet++ allows for rapid up-scaling with additional, larger datasets for even better and more robust performance. This might lead to a re-appraisal of nephropathology in the clinical management of SRNS.

Radiology students often struggle to develop perceptual expertise due to limited time for expert mentorship, leading to errors in visual search patterns and diagnostic interpretation. These perceptual errors—such as missed fixations, brief dwell times, or misinterpretations—are not adequately addressed by existing AI systems, which focus on diagnostic accuracy but fail to explain how and why errors occur. To bridge this gap, we propose MAARTA (Multi-Agentic Adaptive Radiology Teaching Assistant), a multi-agent framework that analyzes gaze patterns and radiology reports to provide personalized feedback. Unlike single-agent models, MAARTA dynamically recruits agents based on error complexity, ensuring adaptive and efficient reasoning. By leveraging thought graphs to compare expert and student gaze behavior, the system identifies missed findings and assigns Perceptual Error Teacher (PET) agents to analyze discrepancies. Using Chain-of-Thought (CoT) prompting, MAARTA generates meaningful insights, helping students understand their errors and refine their diagnostic reasoning, ultimately enhancing AI-driven radiology education.

Using Deep Learning in computer-aided diagnosis systems has been of great interest due to its impressive performance in the general domain and medical domain. However, a notable challenge is the lack of explainability of many advanced models, which poses risks in critical applications such as diagnosing findings in CXR. To address this problem, we propose ItpCtrl-AI, a novel end-to-end interpretable and controllable framework that mirrors the decision-making process of the radiologist. By emulating the eye gaze patterns of radiologists, our framework initially determines the focal areas and assesses the significance of each pixel within those regions. As a result, the model generates an attention heatmap representing radiologists’ attention, which is then used to extract attended visual information to diagnose the findings. By allowing the directional input, our framework is controllable by the user. Furthermore, by displaying the eye gaze heatmap which guides the diagnostic conclusion, the underlying rationale behind the model’s decision is revealed, thereby making it interpretable.In addition to developing an interpretable and controllable framework, our work includes the creation of a dataset, named Diagnosed-Gaze++, which aligns medical findings with eye gaze data. Our extensive experimentation validates the effectiveness of our approach in generating accurate attention heatmaps and diagnoses. The experimental results show that our model not only accurately identifies medical findings but also precisely produces the eye gaze attention of radiologists. The dataset, models, and source code will be made publicly available upon acceptance.

Federated learning has been widely applied in autonomous driving since it enables training a learning model among vehicles without sharing users' data. However, data from autonomous vehicles usually suffer from the non-independent-and-identically-distributed (non-IID) problem, which may cause negative effects on the convergence of the learning process. In this paper, we propose a new contrastive divergence loss to address the non-IID problem in autonomous driving by reducing the impact of divergence factors from transmitted models during the local learning process of each silo. We also analyze the effects of contrastive divergence in various autonomous driving scenarios, under multiple network infrastructures, and with different centralized/distributed learning schemes. Our intensive experiments on three datasets demonstrate that our proposed contrastive divergence loss significantly improves the performance over current state-of-the-art approaches.

Developing an interpretable system for generating reports in chest X-ray (CXR) analysis is becoming increasingly crucial in Computer-aided Diagnosis (CAD) systems, enabling radiologists to comprehend the decisions made by these systems. Despite the growth of diverse datasets and methods focusing on report generation, there remains a notable gap in how closely these models's generated reports align with the interpretations of real radiologists. In this study, we tackle this challenge by initially introducing Fine-Grained CXR (FG-CXR) dataset, which provides fine-grained paired information between the captions generated by radiologists and the corresponding gaze attention heatmaps for each anatomy. Unlike existing datasets that include a raw sequence of gaze alongside a report, with significant misalignment between gaze location and report content, our FG-CXR dataset offers a more grained alignment between gaze attention and diagnosis transcript. Furthermore, our analysis reveals that simply applying black-box image captioning methods to generate reports cannot adequately explain which information in CXR is utilized and how long needs to attend to accurately generate reports. Consequently, we propose a novel explainable radiologist's attention generator network (Gen-XAI) that mimics the diagnosis process of radiologists, explicitly constraining its output to closely align with both radiologist's gaze attention and transcript. Finally, we perform extensive experiments to illustrate the effectiveness of our method. Our datasets and checkpoint is available at https://github.com/UARK-AICV/FG-CXR.

Abstract Background and Aims Diagnostic applications of machine learning in nephropathology are only beginning to emerge. We hypothesized that we could develop a machine learning classifier for 12 different classes of glomerulonephritis with CNN and self-attention-based architectures, following the nephropathology paradigm that globally sclerosed glomeruli are not useful for diagnostic purposes. Method The dataset contains 11,000 PAS-stained glomerular crops from 350 biopsies (four institutions). Each crop retained the diagnosis label from the 12 classes ABMGN, ANCA, C3-GN, CryoGN, DDD, Fibrillary, infection-associated GN (IAGN), IgAGN, MPGN, Membranous, PGNMID, SLEGN-IV; globally sclerotic glomerular crops were stripped from this diagnostic label and were just labeled as the 13th class Sclerotic. This dataset was divided into 75% of samples for training, 15% for validation and 10% testing, avoiding information leak within biopsies. Moreover, a hold-out validation set of another 50 biopsies with 2,000 new crops that were taken from another three centres. A classifier was trained in a fully supervised fashion for 13 classes (12 GN classes and Sclerotic), based on an ensemble of multiple transformer-based classification networks, including Swin-Transformer and ConvNext. This allows classification of each glomerular crop by different network instances. Since each network was trained under different conditions, the whole system acquired a more global knowledge understanding rather than a relying on a single method. For the final decision, the system takes the prediction with the largest confidence threshold, which is also predicted along with the class. Results The Table and the Figure list the metrics for classification performance calculated as Precision, Sensitivity, Specificity, F1 Score and balanced Accuracy. Balanced accuracy was between 0.4797 for CryoGN and 0.5949 for Membranous, it was 0.6892 for Sclerotic. AUCs for ROCs were between 0.40 for PGNMID and 0.82 for Membranous, with 0.81 for Sclerotic. Conclusion This proof-of-concept study establishes a baseline for this challenging classification task, which usually requires immunostains, electron microscopy and even clinical data. Our classification results even on single PAS glomerular crops appear promising. Combined with our automatic glomerular segmentation models, we could rapidly expand the training cohorts sizes and even add more classes of GN.

Data augmentation is a critical component in building modern deep-learning systems. In this paper, we propose MFG Augment , a novel data augmentation method based on the Mean-Field-Game (MFG) theory, that can synthesize a sequence of data between every two images or features. The central idea is to consider every image as a distribution over its pixel or feature space. Using Mean-field Game theory, we can generate a time-continuous “path” from one distribution to another so that the points along the “path” are augmented images or features. Empirically, the experiment results on MNIST, CIFAR-10, and ImageNet demonstrate that the proposed technology has better generalization ability and higher classification accuracy as compared to several benchmark methods. More importantly, our MFG Augment improves the test accuracy significantly when the dataset size is small. MFG Augment consistently shows better affinity and diversity scores, two important empirical metrics for evaluating the generalization of data augmentation techniques.

In the field of chest X-ray (CXR) diagnosis, existing works often focus solely on determining where a radiologist looks, typically through tasks such as detection, segmentation, or classification. However, these approaches are often designed as black-box models, lacking interpretability. In this paper, we introduce Interpretable Artificial Intelligence (I-AI) a novel and unified controllable interpretable pipeline for decoding the intense focus of radiologists in CXR diagnosis. Our I-AI addresses three key questions: where a radiologist looks, how long they focus on specific areas, and what findings they diagnose. By capturing the intensity of the radiologist's gaze, we provide a unified solution that offers insights into the cognitive process underlying radiological interpretation. Unlike current methods that rely on black-box machine learning models, which can be prone to extracting erroneous information from the entire input image during the diagnosis process, we tackle this issue by effectively masking out irrelevant information. Our proposed I-AI leverages a vision-language model, allowing for precise control over the interpretation process while ensuring the exclusion of irrelevant features. To train our I-AI model, we utilize an eye gaze dataset to extract anatomical gaze information and generate ground truth heatmaps. Through extensive experimentation, we demonstrate the efficacy of our method. We showcase that the attention heatmaps, designed to mimic radiologists' focus, encode sufficient and relevant information, enabling accurate classification tasks using only a portion of CXR. The code, checkpoints, and data are at https://github.com/UARK-AICV/IAI

Abstract Background and Aims Thrombotic microangiopathies, comprising atypical hemolytic uremic syndrome (aHUS) and other diseases, can present with a broad clinical and histopathological spectrum. On our way to an evidence-base for the nephropathological work-up of TMAs, we have chosen a machine-learning approach, thus eliminating suboptimal reproducibility of descriptors for individual lesions in the three decisive compartments artery, arteriole and glomerulus with human experts. Here, we present our results for an end-to-end diagnostic system. Method We collected 50 random biopsies with TMAs of various etiologies (including aHUS, hypertension-associated, systemic sclerosis, anti-phospholipid antibody syndrome and others) and 50 biopsies with Mimickers (differential diagnoses of TMA), including severe hypertensive nephropathy, necrotising arteritis/arteriolitis, cryoglobulinemic vasculitis from the three participating centers Cologne, Weill-Cornell Medical Center, and Turin. Whole slide images (WSIs) from all four nephropathology stainings HE, PAS, trichrome and Jones were included in this study. We developed an instance segmentation Mask-RCNN model with a Swin Transformer (t) backbone on tissue crops detected using a lightweight variant of the U-Net segmentation architecture. For the classification model we used our own MorphSet++ set transformer architecture to process batches of EfficientNetv2s-encoded tissue crops entered in three separate compartment channels. Batches were chosen with Monte Carlo sampling or using our own soft Markov Chain Monte Carlo (MCMC) approach. Results of the classification model are reported with 5-fold internal cross-validation. Results Segmentation performance measured as mIOU, mAP, mAR, mF1, mAS for artery reached 0.565, 0.739, 0.679, 0.704, 0.995, for arteriole 0.342, 0.531, 0.488, 0.490, 0.996, for glomeruli 0.818, 0.880, 0.919, 0.896, 0.993. Classification accuracy reached 90% with no false positives for TMA. Missed cases of TMA could be salvaged by an experienced nephropathologist on the display of decisive compartment crops, which were selected using model confidence averaged across each sampling iteration. Conclusion We have designed and trained architectures capable of segmenting decisive compartments and diagnosing TMAs on renal biopsy sections. This will enable automatic analysis of clinicopathological datasets with TMA in large cohorts. Our ultimate goal is to use large cohorts from collaborating institutions for weakly supervised, case-level-annotated training of diagnostic, prognostic and theranostic classifiers.

Abstract Background and Aims Machine learning (ML) holds great promise for improving diagnostics, prognostication and theranostics in nephropathology. So far, applications have not gone much further than segmentation of tissue compartments on whole slide images (WSIs) of paraffin sections. As a proof-of-concept study, we describe the development of a diagnostic classifier for glomerulephritis based on expert-annotated or automatically segmented glomerular transections from periodic-acid Schiff (PAS) paraffin sections only. Method A total of n = 350 biopsies from 5 institutions with 12 classes of glomerulonephritis IgA nephropathy (IgAN), membranous nephropathy (Membranous), anti-glomerular basement membrane antibody GN (ABMGN), infection-associated GN (IAGN), ANCA-associated GN (ANCA-GN), idiopathic membranoproliferative GN (MPGN), SLE GN class IV (SLE-GN-IV), cryglobulinemic GN (CryoGN), C3 GN (C3-GN), dense deposit disease (DDD), fibrillary GN (FibrillaryGN) and proliferative GN with monoclonal immunoglobulin deposits (PGNMID) were included in the study with their respective PAS sections. Glomerular transections were expert-annotated by a nephropathologist and automatically segmented with our own transformer-based segmentation model trained on 100 biopsies with thrombotic microangiopathies and a range of vascular, vasculitic and glomerular diseases closely resembling/mimicking thrombotic microangiopathies. For classification, we divided the cohort into 5 folds for internal cross-validation, performed sample size augmentation with various methods (including shifts in resolution/scale, AutoAugment and others) and trained our proprietary self-attention-based MILx architecture on an EfficientNet backbone with selection of glomerular crop batches by soft Markov chain Monte Carlo sampling in a semi-supervised fashion, with diagnostic class labels for each biopsy. We compared the performance of our proprietary architecture on both expert-annotated and automatically segmented glomerular crops with a recently published benchmark architecture (CLAM) for multiple-instance learning in histopathology. Results Automatic glomerular segmentation performance was excellent with mean AUC and sensitivity (mean average recall) over all classes at 0.904, with near perfect mean average specificity (0.994), as expected best for Membranous, worst for ABMGN. Classification performance of MILx with expert-annotated glomerular crops as inputs had a mean balanced accuracy of 0.84, with AUC metrics in descending order of 0.97 for Membranous, 0.89 for ABMGN, 0.88 for IgAN, 0.86 for Fibrillary, 0.83 for MPGN, 0.80 for ANCA-GN, 0.79 for DDD, 0.78 for PGNMID, 0.75 for IAGN, 0.73 for SLE-GN-IV and CryoGN, 0.67 for C3-GN. Performance with MILx was similar for automatically segmented glomerular crops as input. On this dataset, MILx outperformed CLAM with both entire WSIs as well as expert-annotated glomerular crops as inputs (mean balanced accuracy of 0.72) by a significant margin. Conclusion This proof-of-concept-study indicates that nephropathology-specific architectures like our MILx can be trained for complex tasks on relatively small biopsy cohorts. We should be able to deliver an end-to-end-pipeline for this diagnostic and other tasks based on training sets with case-labels provided by trusted institutions with only minimal expert labeling or annotation required. PAC and HQ contributed equally to this work.

ABSTRACT Reliable and early prediction of cell death (apoptosis) is critically important in various areas of biology, particularly in characterizing the effectiveness of cell-based infusion products utilized for cancer immunotherapy. While deep Convolutional Neural Networks (CNNs) are often used in state-of-the-art approaches for apoptosis classification, they typically focus solely on individual cells and ignore cell-cell interaction. To address this limitation, we propose a novel generative approach based on a video diffusion model, which predicts future cellular behaviors for early detection of apoptosis events, even before molecular markers like Annexin-V or visual indications like membrane blebbing become apparent. Our approach accounts for the interactions of multiple target cells and their spatial and temporal relationships at each time frame. We condition our generative model on two starting frames and utilize an auto-regressive framework to predict the subsequent five frames. Our model achieves a 0.88 F1 score on the cell death event classification and 2.11 mean absolute death-time error, significantly outperforming state-of-the-art methods.