Since the emergence of convolutional neural networks (CNNs) and, later, vision transformers (ViTs), deep learning architectures have predominantly relied on identical block types with varying hyperparameters. We propose a novel block alternation strategy to leverage the complementary strengths of different architectural designs, assembling structurally distinct components similar to Lego blocks. We introduce LegoNet, a deep learning framework that alternates CNN-based and SwinViT-based blocks to enhance feature learning for medical image segmentation. We investigate three variations of LegoNet and apply this concept to a previously unexplored clinical problem: the segmentation of the internal mammary artery (IMA), aorta, and perivascular adipose tissue (PVAT) from computed tomography angiography (CTA) scans. These PVAT regions have been shown to possess prognostic value in assessing cardiovascular risk and primary clinical outcomes. We evaluate LegoNet on large datasets, achieving superior performance to other leading architectures. Furthermore, we assess the model's generalizability on external testing cohorts, where an expert clinician corrects the model's segmentations, achieving DSC > 0.90 across various external, international, and public cohorts. To further validate the model's clinical reliability, we perform intra- and inter-observer variability analysis, demonstrating strong agreement with human annotations. The proposed methodology has significant implications for diagnostic cardiovascular management and early prognosis, offering a robust, automated solution for vascular and perivascular segmentation and risk assessment in clinical practice, paving the way for personalised medicine.
Automatic segmentation of anatomical structures in X-ray images is essential for clinical and research applications, particularly as the increasing volume of medical examinations necessitates workflow automation and efficient data screening. While numerous public X-ray imaging datasets exist, they are predominantly limited to chest X-rays, hindering AI-driven solutions for whole-body segmentation. In this paper, we propose a method for whole-body anatomical segmentation in X-ray images using synthetic data. We generate synthetic X-ray projections from an existing CT dataset using the DiffDRR framework and train five multi-class 2D UNet models, each targeting distinct anatomical groups. To assess generalization, we validate a subset of our models on two real X-ray imaging databases. Our models achieve a perclass median Dice Similarity Coefficient (DSC) above 0.88 for nearly 79 anatomical structures on the synthetic test set and perform on par with models trained on real data for rib segmentation with a similar architecture. We further highlight key challenges in transferring models from simulation to real-world datasets. Our models are made publicly available on GitHub (github.com/risc-mi/totalsegmentator2D) to facilitate further development.
Accurate alignment of histopathological whole slide images (WSIs) across different staining modalities is essential for comprehensive multimodal tissue analysis, yet remains challenging due to complex deformations, sectioning artefacts, and staining variations. This paper presents an efficient registration framework that effectively addresses these challenges through a progressive multi-stage approach. Our approach integrates efficient tissue segmentation using Florence2-SAM2 with a progressive three-stage registration strategy. Initial coarse alignment utilises tissue mask centroids and geometric transformations to establish preliminary correspondence. Subsequently, the accelerated feature extraction module (XFeat) identifies and matches distinctive tissue landmarks from the roughly aligned images, significantly reducing the computational burden of the final stage. The registration concludes with fine alignment using diffusive regulariser based iterative optimisation. Comprehensive evaluation on the ANHIR, ACROBAT and HyReCo datasets demonstrates that our method achieves superior registration accuracy (med-TRE 6.01 m for restained and med-TRE 65.5 m for consecutive sections) and computational efficiency (average runtime of 17 s) compared to state-of-the-art approaches. The proposed framework enables precise spatial correlation of histological features in differently stained modalities, facilitating advanced analysis for both diagnostic and research applications.
In clinical scenarios, radiologists analyse multiple chest Xray (CXR) images from various view positions to identify diseases and abnormalities. To replicate the diagnostic approach of experienced radiologists, we propose an encoder-decoder-based CXR report generation architecture, KCLVA, which leverages the Unified Medical Language System (UMLS) to extract view-specific information from diagnostic reports, focusing on posteroanterior, anteroposterior, and lateral views. This extracted information facilitates view-specific attention (VA) mechanisms and is subsequently used to construct a similarity matrix that enables many-to-many contrastive learning. In the encoder, we employ a knowledge distillation architecture to guide the learning of the student model by freezing the teacher model. Within the student text encoder, the VA mechanism is utilised to automatically assign higher weights to tokens corresponding to a specific view in diagnostic reports based on the view position of the CXR, while assigning lower weights to other tokens. The image and text features are then integrated using contrastive learning. In the decoder, a transformer-based backbone architecture is employed to decode the encoder output and generate a medical diagnosis report. This strategy leverages UMLS to extract view-specific information, employs VA to adjust token weights, and utilises many-to-many contrastive learning through a weighted contrastive loss. Together, these components enable our model to closely simulate the diagnostic process of professional radiologists. Consequently, our method achieves significant improvements of 0.185 on METEOR and 0.078 on ROUGE compared to previous approaches.
Using histological whole slide images (WSIs), how can we stratify patients into meaningful risk groups by leveraging data from both patients with known survival times and event indicators and those without, while requiring no manual post-hoc thresholding of predicted risk scores? Existing survival stratification methods in computational pathology train a supervised model on patients with known survival times and event indicators, then apply it to a test set to generate risk scores. These scores are typically thresholded, often at the median, to assign patients to high- or low-risk groups. Such inductive pipelines overlook the large pool of unlabelled patients even though number of cases with known survival times are typically limited and observed events are even rarer. As a result, existing methods often fail to uncover meaningful risk groups. In this work, we introduce the first Transductive Survival Ranking (TSR) model for WSIs, designed to leverage both labelled and unlabelled data for improved survival prediction. Given a dataset where only a subset of WSIs have associated survival time and event information, our approach (1) ranks patients by predicted survival times, (2) automatically discovers risk groups without requiring manual thresholding, and (3) transduces differential survival patterns from patients with observed events to those without events. We evaluate the proposed approach on multiple pan-cancer datasets, demonstrating that it achieves statistically significant risk sub-grouping in cases where classical approaches fail while producing high concordance indices. Unlike traditional survival models that rely solely on labelled data for training and manual cutoffs, this work leverages transductive learning to make data-driven, threshold-free risk stratification possible, offering a more reliable and data-efficient framework for survival analysis in computational pathology.
Can spatial heterogeneity in whole slide images (WSIs) be automatically learned to predict patient survival outcomes? Spatial heterogeneity within the tissue and tumour microenvironments is increasingly recognised as a critical indicator of cancer prognosis. However, most existing methods do not explicitly model spatial heterogeneity, and those that do typically require segmentation of cellular or tissue structures followed by the use of hand-crafted spatial metrics such as the Morisita-Horn index or colocalisation indices. These approaches are not learnable, and rely on manually defined features and relationships. In this work, we propose a novel neural network-based framework that learns a differentiable variant of Moran’s Index, a classical measure of spatial autocorrelation, to automatically quantify spatial heterogeneity from point cloud representations of whole slide image patches. Guided by survival information in training, the proposed method adaptively learns which image-derived features correlate spatially, and how proximity influences their interaction, enabling the discovery of prognostic spatial patterns directly from patch-level features. Applied to the TCGA breast cancer dataset, the proposed method achieves a test concordance index of 0.662 ± 0.060, outperforming several competitive baselines while providing an interpretable quantification of spatial heterogeneity in WSIs. This approach opens new directions for interpretable, spatially informed prognostic modeling and biomarker discovery in computational pathology.
Digital histopathological image analysis encounters processing level challenges due to the extremely high resolution of whole slide images (WSI), which requires resource-intensive annotations and extensive computational resources. Due to these limitations, histological processing is heavily reliant on self-supervision and patch-level processing. While there are multiple modelling approaches which consider the aspect of limited supervision and patch-level processing, most of the models are not capable of capturing spatial context while focusing into the local cellular-level details. This simultaneous processing of global spatial context along with cellular details at the patch-level again becomes computationally inefficient to handle. In order to circumvent these issues, we devise a spatially aware masked attention based DINO network (SMatt-DINO) that processes spatial neighbourhood in an efficient manner by selectively masking the attention layers within the network. This capacitates our model to generate robust representations that are capable of efficiently classifying brain regions in histological images in a self-supervised manner. We further incorporate positional information into the network to enhance classification in anatomical boundary regions. Through experimentation, we validate that our model has better performance in classification of regions in fetal brain, specifically reducing misprediction in the anatomical boundary regions. We have also validated the generalization capability of our model by testing it on a fetal brain from a completely different acquisition setup. This demonstrates the robustness and effectiveness of our model in histological tasks.
The endometrium undergoes rapid cycles of menstrual breakdown and repair. In each cycle, oestrogen-dependent proliferation followed by progesterone-dependent differentiation of the endometrium culminates in a sterile inflammatory tissue response, termed the decidual reaction, at the start of the embryo implantation window. Analysis of timed endometrial biopsies is widely used to investigate a spectrum of reproductive disorders, including recurrent implantation failure in IVF and recurrent miscarriage. Deep profiling of whole slide images (WSIs), capturing the spatial and functional organization of key histological structures, such as nuclei, glandular and luminal epithelium, subluminal stroma and spiral arterioles, holds significant promise for automated endometrial assessment. To address the lack of such methodologies, we developed ASTER, a multi-task deep learning model for simultaneous segmentation of multiple histological structures in immunostained endometrial WSIs. ASTER has been developed and validated over a large dataset of 2,652 endometrial whole slide images, including 35,135 annotated objects obtained using a pathologist-in-the-loop methodology. The model demonstrates strong performance across all segmentation tasks. Further, analysis of an independent set of 2,082 unseen WSIs showed that ASTER-derived features correlate with cycle-dependent endometrial gene expression. This represents the first systematic study linking segmented morphological characteristics to molecular and temporal markers of endometrial function, demonstrating its effectiveness in enabling comprehensive and automated profiling. This highlights the potential of ASTER to support personalized management of women experiencing reproductive failure. Segmentation results are available for interactive exploration at https://tiademos.dcs.warwick.ac.uk/bokeh_app?demo=ASTER.
Radiology reports are crucial for diagnosing diseases, yet generation them is time-consuming, places a significant workload on medical professionals, and is subject to inter-expert variability, as different radiologists may interpret the same X-ray differently. This paper presents a novel hybrid AI model called CheXSBT, which combines our custom-designed Dual-Attention Swin Transformer (DAST) for vision processing with BERT for natural language understanding to automate the generation of chest X-ray (CXR) reports. Leveraging the MIMIC-CXR dataset, which includes over 370,000 X-ray images and their corresponding reports, CheXSBT learns to interpret chest X-ray images and convert them into structured, meaningful text. Our study focuses on two main objectives: (1) automating report generation to accelerate the diagnostic process and (2) improving model interpretability to foster trust among radiologists. The approach involves preprocessing chest X-ray images and their corresponding text reports using the pre-trained BLIP processor, training the novel hybrid vision-language model on paired data, and fine-tuning it for clinical relevance and coherence. The performance of CheXSBT is rigorously evaluated using established metrics such as BLEU, ROUGE, and METEOR, achieving scores of 0.232 for BLEU-4 and 0.392 for ROUGE-L, outperforming other state-of-the-art models and ensuring high-quality report generation. By reducing radiologists' workload and providing quick, accurate information, CheXSBT aims to transform the intersection between AI and clinical practice, making radiology reporting more efficient, consistent, and accessible.
We present a deep learning approach for nuclear-level prediction of Ki-67 expression directly from H E-stained breast cancer images, potentially eliminating the need for costly and time-consuming immunohistochemistry (IHC). Our two-phase pipeline integrates HoVerNet for precise nuclei segmentation with a specialized ResNet-style classifier (MitoNet) optimized for small patch classification. Our model, trained on 215,825 annotated nuclei, achieves 82.4
Medical vision-language models (Med-VLMs) have shown strong generalization in downstream tasks by leveraging large-scale image-text pretraining. However, their vulnerability to adversarial perturbations poses critical risks in safety-sensitive applications. While randomized smoothing provides certifiable guarantees against such perturbations, it requires the model to remain accurate under Gaussian noise-an assumption that fails in practice without specialized adaptation. Prior work, such as PromptSmooth, introduced text-only prompt tuning to improve robustness, but neglected visual adaptation and struggled under high noise levels. In this paper, we propose PromptSmooth++, a multimodal adaptation framework that enhances the certified robustness of frozen Med-VLMs under randomized smoothing. Our method introduces two complementary variants: (i) a few-shot strategy that jointly optimizes visual and textual prompts for noise-aware cross-modal alignment, and (ii) a zero-shot approach that performs test-time vision-side adaptation using lightweight Low-Rank Adapters (LoRA), optimized with a self-supervised entropy loss on noisy inputs. Extensive experiments across multiple Med-VLMs and multiple datasets spanning diverse medical modalities demonstrate that PromptSmooth++ significantly improves certified accuracy over existing baselines while maintaining high clean performance and computational efficiency. Our results show that modality-aware prompting and vision-side adaptation are both essential for certifiably robust medical imaging systems. Code is available at https://github.com/fahadshamshad/multimodal-promptsmooth.
Although vision-language models (VLMs) have achieved strong results in general computer vision tasks, their effectiveness in medical imaging remains limited—primarily due to their insufficient reasoning capabilities. In this work, we introduce KODER, a novel knowledge-driven reasoning framework aimed at improving diagnostic accuracy for ultrasound-based burn assessment. KODER integrates pre-trained VLMs with first-order logic (FOL) reasoning to generate interpretable diagnostic hypotheses. By combining rich experimental descriptions and clinical insights into a unified prompt, the framework produces multiple diagnostic hypotheses and refines them through iterative consistency checks using an SMT solver. The validated hypotheses are then used to support both surgical decision-making and detailed burn depth classification. We evaluate our approach on a retrospective dataset collected from a U.S. burn center, where it achieves significant performance gains—reaching up to 93
Cardiac ultrasound is widely used for the diagnosis and monitoring of cardiovascular diseases due to its noninvasive nature, real-time imaging capability, and low cost. However, its clinical utility is often limited by noise sensitivity and acquisition variability, which adversely affect automated interpretation and sequence consistency. To overcome these limitations, this paper presents a multimodal deep learning framework that combines a denoising diffusion model with a Temporal Transformer to generate high-quality cardiac ultrasound videos. A unified preprocessing pipeline with intensity normalisation and standardisation is employed to reduce intersample variation and enhance anatomical structures. Spatial features are first extracted from individual frames, followed by temporal modelling across sequences using the Temporal Transformer. These features guide the latent-space denoising process, optionally augmented by ControlNet for structure-aware generation. The experimental results demonstrate that the proposed method achieves robust performance, with an FID of 43.50, an FVD of 274.52, and an inception score of 8.62. Ablation studies further verify the critical contributions of ControlNet and composite loss design, highlighting the effectiveness of the framework in ensuring both spatial fidelity and temporal coherence.
Despite numerous techniques developed for polyp detection, the issue of generalisability to new centres and populations while maintaining fast inference persists. To address this, we compile a multicentre train set consisting of 1941 images, and use it to push the generalisability of the real-time EfficientDet for polyp detection by employing a knowledge distillation teacher-student architecture and instance whitening. We train a large EfficientDet teacher on the combined Kvasir-Seg and PolypGen datasets and subsequently use its softened output probabilities to guide a smaller EfficientDet student, aiming to improve generalisability and speed while preserving performance. To further enhance generalisability across different clinical settings and patient populations, we integrate instance whitening into the backbones of both teacher and student networks to mitigate domain-specific variations, especially in patient populations like IBD. We compare our model with the state-of-the-art (SOTA) models DETR, RetinaNet, and Faster R-CNN under detection and inference metrics on the following benchmark datasets: Kvasir-SEG (seen centre and population), PolypGen-C6 (unseen centre), and our in-house IBD dataset (unseen centre and population). Our approach improves AP50:95 over the best performing benchmark -RetinaNet- by 3.97
Breast cancer (BC) aggressiveness significantly impacts patient prognosis by influencing relapse, metastasis, and mortality rates. Currently, the assessment of tumor aggressiveness is primarily dependent on biopsies and pathological image analysis, which are challenging to obtain during follow-up stages after treatment, particularly if there is a relapse. Our study introduces a novel framework utilizing digital mammography, the standard for early breast cancer detection, to classify tumor aggressiveness non-invasively. This framework employs a Cascaded ResUNet with boundary-aware capabilities for precise tumor segmentation, ensuring accurate delineation of tumor boundaries. Key morphological and texture features are then extracted from the segmented regions to categorize tumor aggressiveness. Validated on an in-house dataset, our method achieved an accuracy of 90.6
Early detection of melanoma, the most lethal form of skin cancer, can greatly enhance patient survival rates. Although AI models have demonstrated strong diagnostic capabilities, their integration into clinical practice remains limited due to concerns over explainability and trust. This work proposes a SHAP-enhanced Convolutional Neural Network (SCNN) for binary classification of skin lesions into melanoma and non-melanoma categories, directly integrating Shapley Additive Explanations (SHAP) as an additional input channel to enhance performance and explainability. We evaluated SCNN on the ISIC 2017 and ISIC 2018 datasets, achieving ROC-AUC scores of 0.80 and 0.91, respectively. These results indicate substantial improvements in classification accuracy and robustness compared to baseline models. An analysis of model explainability on the ISIC 2017 dataset reveals that SCNN more accurately highlights lesion areas identified by experts, achieving a mean Intersection-over-Union score of 0.34, which marginally improved the baseline score of 0.32. 53
Accurate wound size measurement is essential for effective chronic wound management, guiding clinical decisions and predicting treatment efficacy, yet it remains a challenge due to the high time required by the manual measurement process and its subpar reproducibility. This study presents an automated detection, segmentation and measurement pipeline for chronic wounds using computer vision and deep learning. The impact of dataset composition on deep learning-based open wound detection and segmentation is investigated, along with the comparison of three object detection architectures (RetinaNet with MobileNetV2 backbone, and CenterNet with ResNetV1 or MobileNetV2) and two segmentation networks (DeepLabV3+ with ResNet50 backbone, and UPerNet with Swin Transformer). For wound measurement, traditional computer vision methods were employed to estimate the wound’s real-world width, height and area. Separate studies evaluated each task, followed by a complete pipeline assessment that couples the developed wound and reference marker detection model with the segmentation and measurement tasks. For wound and marker detection, the RetinaNet-MobileNetV2 achieved the best performance with mAP@.75IoU of 64.67
The integration of deep learning tools in gastrointestinal vision holds the potential for significant advancements in diagnosis, treatment, and overall patient care. A major challenge, however, is overconfident predictions, even when encountering unseen or newly emerging disease patterns, which undermines the reliability of such tools. We address this critical issue of reliability in gastrointestinal vision through the lens of out-of-distribution (OOD) detection, which handles previously unseen or emerging diseases as OOD samples. To this end, we hypothesize that the features of an in-distribution example will cluster closer to the centroids of their ground truth class, resulting in a shorter distance between the example and the nearest centroid. In contrast, OOD examples maintain more or less an equal distance from all class centroids. Based on this hypothesis, we propose a novel Nearest-Centroid Distance Deficit (NCDD) score in the feature space for gastrointestinal OOD detection. Evaluations across Resnet, ViT, DeiT and MLPmixer and two publicly available benchmarks, Kvasir2 and Gastrovision, demonstrate the effectiveness of our approach compared to several state-of-the-art methods. The code is available at: bhattarailab/NCDD.
Histopathology image analysis plays a vital role in disease diagnosis; however, the limited availability of labeled data and the large size of Whole Slide Images (WSIs) creates a challenge for training deep learning models. Traditional active learning methods, which typically select entire images for labeling, are impractical for WSIs due to their substantial size and computational requirements. This paper presents a novel active learning framework specifically tailored for WSI selection. Our method computes WSI-level uncertainty scores using various techniques, including Margin Sampling, Monte Carlo Sampling, Gradient-based methods, and diversity scores derived from Cosine similarity and K-means++. By focusing on the most informative WSIs, we aim to significantly reduce labeling efforts while maximizing information gain for subsequent model training. We demonstrate that our approach consistently outperforms random data selection strategies, resulting in significant improvements in segmentation Dice scores on the PANDA dataset for Gleason-grade segmentation. Our work offers a new approach to efficiently use limited labeled data, facilitating the development of more accurate and cost-effective Gleason-grade segmentation models.
In digital dentistry, it is increasingly common to consider both teeth and soft tissue when designing treatments or visualising patient models. However, scans of teeth and tissue are usually acquired separately, and few studies have explored the joint geometry of the facial surface and dentition, despite its many potential clinical applications. This work explores how linear and multi-linear shape models of lips and teeth can bridge the gap between disparate 3D scanning modalities. We construct a multi-linear shape model from 284 CT scans. The model disentangles soft-tissue thickness from the underlying dental-arch shape. We use a novel ray-casting approach for determining tissue thickness, and obtain the dental arch shapes from segmented tooth scans in an automated manner. We demonstrate how the model can jointly fit to tooth-arch shapes and face-scans from lip scans, thereby enabling the alignment of separately acquired intra-oral and facial scans. The ray-casting approach outperforms previous work on modelling soft-tissue thickness, as it represents the lip region more accurately. The model is successfully used to estimate the tooth arch shape, and this in turn leads to a good alignment of intra-oral scans within the lip shape. Dento-facial models have many uses in the world of digital dentistry. By modelling the dependence between lip soft-tissue and the shape of the arch, we demonstrate the inference of the tooth arch shape for unseen lips. Further work can expand the model to include appearance and expression, and enable yet more applications.
