In this paper, a new method for automatic MR image inhomogeneity correction is proposed. This method, based on deep learning, uses unsupervised learning to estimate the bias corrected images minimizing a cost function based on the entropy of the corrupted image, the derivative of the estimated bias field and corrected image statistics. The proposed method has been compared with the state-of-the-art method N4 providing improved results.
In the stroke workflow, timely decision-making is crucial. Identifying, localizing, and measuring occlusive arterial thrombi during initial imaging is a critical step that triggers the choice of therapeutic treatment for optimizing vascular re-canalization. We present a recurrent model that segments the thrombus in patients suffering from a hyper-acute stroke. A cross-attention module is defined to merge the diffusion and susceptibility-weighted modalities available in Magnetic Resonance Imaging (MRI), which are fed to a modified version of the Convolutional Long-Short-Term Memory (CLSTM) model. It detects almost all the thrombi with a Dice higher than 0.6. The lesion segmentation prediction reduces the false positives to almost zero and the performance is comparable between distal and proximal occlusions.
The identification of mitotic figures (MFs) is a routine task in the histopathological assessment of tumor malignancy with known limitations for human observers. For a machine learning pipeline to robustly detect MFs, it must overcome a variety of conditions such as different scanners, staining protocols, tissue configurations, and organ types. In order to develop a deep learning-based algorithm that can cope with these challenges, there are two obstacles that need to be overcome: obtaining a large-scale dataset of MF annotations spread across different domains of interest, including whole slide images (WSIs) exhaustively annotated for MFs, and using the annotated MFs in an efficient training process to extract the most relevant features for classification. Our work attempts to address both of these challenges and establishes an MF detection pipeline trained solely on animal data, yet competitive on the mixed human/animal MIDOG22 dataset, and, in particular, on human breast cancer. First, we propose a processing pipeline that allows us to strengthen the true scanner robustness of our dataset by physically rescanning the glass slides of annotated WSIs and registering MF positions. To enable the use of such rescans for training, we propose a novel learning paradigm tailored for labels that match partially, which allows to account for ambiguous MF positions in the rescans caused by spurious, suboptimal fine-focus on potential MFs by the scanner. Second, we demonstrate how a multi-task learning approach for MF subtypes, including the prediction of atypical mitotic figures (AMFs), can significantly enhance a model's ability to distinguish MFs from imposters. Our algorithm, using a standard object detection pipeline, performs very competitively with an average test set F1 value across five runs of 0.80 on the MIDOG22 training set. We also demonstrate its ability to stratify overall survival on the TCGA-BRCA dataset based on mitotic density, though it falls short of reaching significance in stratifying survival based on AMFs.
Low-dose computed tomography (LDCT) enables imaging with minimal radiation exposure but typically results in noisy outputs. Deep learning algorithms have been emerging as popular tools for denoising LDCT images, where they typically rely on large data sets requiring data from multiple centers. However, LDCT images collected from different centers (clients) can present significant data heterogeneity, and the sharing of them between clients is also constrained by privacy regulations. In this work, we propose a personalized federated learning (FL) approach for enhancing model generalization across different organ images from multiple local clients while preserving data privacy. Empirically, we find that earlier FL methods tend to underperform single-set models on non-IID LDCT data due to the presence of data heterogeneity characterized by varying frequency patterns. To address this, we introduce a Federated Learning with Frequency Domain Decomposition (FedFDD) approach, which decomposes images into different frequency components and then updates high-frequency signals in an FL setting while preserving local low-frequency characteristics. Specifically, we leverage an adaptive frequency mask with discrete cosine transformation for the frequency domain decomposition. The proposed algorithm is evaluated on LDCT datasets of different organs and our experimental results show that FedFDD can surpass state-of-the-art FL methods as well as both localized and centralized models, especially on challenging LDCT denoising cases. Our code is available at https://github.com/xuhang2019/FedFDD.
X-ray imaging is a cornerstone in medical diagnosis, constituting a significant portion of the radiation dose encountered by patients. Excessive radiation poses health risks, particularly for pediatric patients, but despite the imperative to reduce radiation doses, conventional image processing methods for X-ray denoising often struggle with heuristic parameter calibration and prolonged execution times. Deep Learning solutions have emerged as promising alternatives, but their effectiveness varies, and challenges persist in preserving image quality. This paper presents an exploration of diffusion models for planar X-ray image denoising, a novel approach that to our knowledge has not been yet investigated in this domain. We perform real time denoising of Poisson noise while preserving image resolution and structural similarity. The results indicate that diffusion models show promise for planar X-ray image denoising, offering a potential improvement in the optimization of diagnostic utility amid dose reduction efforts.
Continual medical image segmentation primarily explores the utilization of U-Net and its derivatives within the realm of medical imaging, posing significant challenges in meeting the demands of shifting domains over time. Foundation models serve as robust knowledge repositories, offering unique advantages such as general applicability, knowledge transferability, and continuous improvements. By leveraging pre-existing domain insights, adaptability, generalization, and performance across diverse tasks can be enhanced. In this work, we show how to deploy Segment Anything Model's (SAM) natural image pretraining for the continual medical image segmentation, where data is sparse. We introduce UnCLe SAM, a novel approach that uses the knowledge of the pre-trained SAM foundation model to make it suitable for continual segmentation in dynamic environments. We demonstrate that UnCLe SAM is a robust alternative to U-Net-based approaches and showcase its state-of-the-art (SOTA) continual medical segmentation capabilities. The primary objective of UnCLe SAM is to strike a delicate balance between model rigidity and plasticity, effectively addressing prevalent pitfalls within CL methodologies. We assess UnCLe SAM through a series of prostate segmentation tasks, applying a set of different CL methods. Comparative evaluations against the SOTA Lifelong nnU-Net framework reveal the potential application of UnCLe SAM in dynamically changing environments like healthcare. Our code base will be made public upon acceptance.
Large dataset requirements for deep learning methods can pose a challenge in the medical field, where datasets tend to be relatively small. Synthetic data can provide a suitable solution to this problem, when complemented with real data. However current generative methods normally require all data to be of the same resolution and, ideally, aligned to an atlas. This not only creates more stringent restrictions on the training data but also limits what data can be used for inference. To overcome this our work proposes a latent diffusion model that is able to control sample geometries by varying their resolution, field of view, and orientation. We demonstrate this work on whole body CT data, using a spatial conditioning mechanism. We showcase how our model provides samples as good as an ordinary latent diffusion model trained fully on whole body single resolution data. This is in addition to the benefit of further control over resolution, field of view, orientation, and even the emergent behaviour of super-resolution. We found that our model could create realistic images across the varying tasks showcasing the potential of this application.
Deep learning models offer unprecedented opportunities for diagnosis, prognosis, and treatment planning. However, conventional deep learning pipelines often encounter challenges in learning unbiased classifiers within imbalanced data settings, frequently exhibiting bias towards minority classes. In this study, we aim to improve medical image classification by effectively addressing class imbalance. To this end, we employ differentiable loss functions derived from classification metrics commonly used in imbalanced data settings: Matthews correlation coefficient (MCC) and the F1 score. We explore the efficacy of these loss functions both independently and in combination with cross-entropy loss and various batch sampling strategies on diverse medical datasets of 2D fundoscopy and 3D magnetic resonance images. Our findings demonstrate that, compared to conventional loss functions, we achieve notable improvements in overall classification performance, with increases of up to +12% in balanced accuracy and up to +51% in class-wise F1 score for minority classes when utilizing cross-entropy coupled with metrics-derived loss. Additionally, we conduct feature visualization to gain insights into the behavior of these features during training with imbalance-aware loss functions. Our visualization reveals a more pronounced clustering of minority classes in the feature space, consistent with our classification results. Our results underscore the effectiveness of combining cross-entropy loss with class-imbalance-aware loss functions in training more accurate classifiers, particularly for minority classes.
Obesity is associated with an increased risk of morbidity and mortality. Achieving a healthy body composition, which involves maintaining a balance between fat and muscle mass, is important for metabolic health and preventing chronic diseases. Computed tomography (CT) imaging offers detailed insights into the body’s internal structure, aiding in understanding body composition and its related factors. In this feasibility study, we utilized CT image data from 2,724 subjects from the large metabolic health cohort studies SCAPIS and IGT. We train and evaluate an uncertainty-aware deep regression based ResNet-50 network, which outputs its prediction as mean and variance, for quantification of cross-sectional areas of liver, visceral adipose tissue (VAT), and thigh muscle. This was done using collages of three single-slice CT images from the liver, abdomen, and thigh regions. The model demonstrated promising results with the evaluation metrics – including R-squared ($R^2$) and mean absolute error (MAE) for predictions. Additionally, for interpretability, the model was evaluated with saliency analysis based on Grad-CAM (Gradient-weighted Class Activation Mapping) at stages 2, 3, and 4 of the network. Deformable image registration to a template subject further enabled cohort saliency analysis that provide group-wise visualization of image regions of importance for associations to biomarkers of interest. We found that the networks focus on relevant regions for each target, according to prior knowledge. The source code is available at: \url{https://github.com/noumannahmad/dr_3slice_ct}.
The Segment Anything Model (SAM) is an interactive foundation segmentation model, showing impressive results for 2D natural images using prompts such as points and boxes. Transferring these results to medical image segmentation is challenging due to the 3D nature of medical images and the high demand of manual interaction. As a 2D architecture, SAM is applied slice-per-slice to a 3D medical scan. This hinders the application of SAM for volumetric medical scans since at least one prompt per class for each single slice is needed. In our work, the applicability is improve by reducing the number of necessary user-generated prompts. We introduce and evaluate multiple training-free strategies to automatically place box prompts in bone CT volumes, given only one initial box prompt per class. The average performance of our methods ranges from 54.22% Dice to 88.26% Dice. At the same time, the number of annotated pixels is reduced significantly from a few millions to two pixels per class. These promising results underline the potential of foundation models in medical image segmentation, paving the way for annotation-efficient, general approaches.
Histopathological analysis, vital for medical diagnostics, is often challenged by artifacts in sample preparation and imaging, such as staining inconsistencies and physical obstructions. Addressing this, our work introduces a novel, fully unsupervised histopathological artifact restoration pipeline (HARP). HARP integrates artifact detection, localization, and restoration into one pipeline. The first step to make artifact restoration applicable is an analysis of anomaly detection algorithms. Then, HARP leverages the power of unsupervised segmentation techniques to propose localizations for potential artifacts, for which we select the best localization based on our novel inpainting denoising diffusion model. Finally, HARP employs an inpainting model for artifact restoration while conditioning it on the artifact localizations. We evaluate the artifact detection quality along with the image reconstruction quality, surpassing the state-of-the-art artifact restoration. Furthermore, we demonstrate that HARP improves the robustness and reliability of downstream models and show that pathologists can not tell the difference between clean images and images restored through HARP. This demonstrates that HARP significantly improves image quality and diagnostic reliability, enhancing histopathological examination accuracy for AI systems.
Conditional deep generative models have emerged as powerful tools for generating realistic images enabling fine-grained control over latent factors. In the medical domain, data scarcity and the need to integrate information from diverse sources present challenges for existing generative models, often resulting in low-quality image generation and poor controllability. To address these two issues, we propose Multi-Source StyleGAN (MSSG). MSSG learns jointly from multiple heterogeneous data sources with different available covariates and can generate new images controlling all covariates together, thereby overcoming both data scarcity and heterogeneity. We validate our method on semi-synthetic data of hand-written digit images with varying morphological features and in controlled multi-source simulations on retinal fundus images and brain magnetic resonance images. Finally, we apply MSSG in a real-world setting of brain MRI from different sources. Our proposed algorithm offers a promising direction for unbiased data generation from disparate sources. For the reproducibility of our experimental results, we provide [detailed code implementation](https://github.com/weslai/msstylegans).
The integration of Large Language Models (LLMs), such as ChatGPT, in radiology could offer insight and interpretation to the increasing number of radiological findings generated by Artificial Intelligence (AI). However, the complexity of medical text presents many chal- lenges for LLMs, particularly in uncommon languages such as Dutch. This study therefore aims to evaluate ChatGPT’s ability to generate accurate ‘Impression’ sections of radiol- ogy reports, and its effectiveness in evaluating these sections compared against human radiologist judgments. We utilized a dataset of CT-thorax radiology reports to fine-tune ChatGPT and then conducted a reader study with two radiologists and GPT-4 out-of-the- box to evaluate the AI-generated ‘Impression’ sections in comparison to the originals. The results revealed that human experts rated original impressions higher than AI-generated ones across correctness, completeness, and conciseness, highlighting a gap in the AI’s abil- ity to generate clinically reliable medical text. Additionally, GPT-4’s evaluations were more favorable towards AI-generated content, indicating limitations in its out-of-the-box use as an evaluator in specialized domains. The study emphasizes the need for cautious integration of LLMs into medical domains and the importance of expert validation, yet also acknowledges the inherent subjectivity in interpreting and evaluating medical reports.
The advancements in computational modeling and simulations have facilitated the emergence of in-silico clinical trials (ISCTs). ISCTs are valuable in developing and evaluating novel treatments targeting acute ischemic stroke (AIS), a prominent contributor to both mortality and disability rates. However, obtaining large populations of accurate anatomical structures that are required as input to ISCTs is labor-intensive and time-consuming. In this work, we propose and evaluate diffusion-based generative modeling and set transformers to generate a population of synthetic intracranial vessel tree centerlines with associated radii and vessel types. We condition our model on the presence of an occlusion in the middle cerebral artery, a frequently occurring occlusion location in AIS patients. Our analysis of generated synthetic populations shows that our model accurately produces diverse and realistic cerebral vessel trees that represent the geometric characteristics of the real population.
Pre-training on a large dataset such as ImageNet followed by supervised fine-tuning has brought success in various deep learning-based tasks. However, the modalities of natural images and ultrasound images have considerable differences, making pre-training on natural images ineffective for ultrasound-related tasks. In this paper, we introduce a unified masking-based model for both ultrasound images and videos that learns better visual representation than the network with single-modality representations. This is the first large-scale generalized ultrasound pre-training network that simultaneously utilizes 100,000+ videos and images of different parts of the human anatomy such as the liver, bones, heart, thyroids, nerves, etc, making the network an effective benchmark pretrained model for any ultrasound-specific downstream tasks. We propose a novel method for ultrasound image analysis that utilizes an ultrasound-specific confidence map to guide low-level representation learning through masked feature acquisition. Our pre-trained network has demonstrated remarkable efficacy and versatility in tackling both classification and segmentation tasks across a range of ultrasound pathologies, highlighting its potential for widespread adoption and impact in the ultrasound field. In addition, we show that our pre-training model can be leveraged to learn efficiently with a small number of labeled ultrasound images.
Eye diseases such as the chronic central serous chorioretinopathy are characterized by fluid deposits that alter the retina and impair vision. These fluids occur at irregular intervals and may dissolve spontaneously or thanks to treatment. Accurately capturing this behavior within an image registration framework is challenging due to the resulting prominent tissue deformations and missing image correspondences between visits. This paper presents FluidRegNet, a convolutional neural network for the registration of successive optical coherence tomography images of the retina. The correspondence between time points is established by predicting the position of the origin of the fluids by creating a fluid seed in the form of sparse intensity offsets in the moving image and registering the fluid seed to the affected area in the follow-up image. We show that this leads to deformation fields that more accurately reflect the actual dynamics of retinal fluid growth compared to other image registration methods. In addition, the network outputs are used for unsupervised fluid segmentation.
This study examines the realism of medical images created with deep generative models, specifically their replication of aging and Alzheimer's disease (AD) related anatomical changes. Previous research focused on developing generative methods with limited attention to image fidelity. We aim to assess the resemblance of brain MRI generated by a StyleGAN3 model with causal controls to neurodegenerative changes. For a benchmark, we conducted a visual Turing test (VTT) to see if radiologists could distinguish between synthetic and real images. Then, we employed a U-Net-based model to segment hallmarks relevant to normal aging and (AD). Finally, we conducted statistical tests for our hypothesis that no significant differences existed between real and synthetic images. (VTT) results showed radiologists struggled to differentiate between image types, highlighting (VTT)'s limitations due to subjectivity and time constraints. We found slight hippocampus distribution differences ($\textit{P}$ = 5.7e-2) and significant lateral ventricle discrepancies ($\textit{P}$s $<$ 5.0e-2), indicating higher hippocampus realism and ventricle size inconsistencies. The model more effectively simulated changes in the hippocampus than in the lateral ventricles, where difficulties were encountered with certain subgroups. We conclude that the (VTT) alone is inadequate for a comprehensive quality evaluation, promoting a more objective approach. Future research could adapt our approach to evaluate other generated medical images intended for different downstream tasks. For reproducibility, we provide detailed code implementation$^1$.
We introduce a novel method leveraging conditional generative adversarial networks (cGANs) to generate diverse, high-resolution microscopy images for neuron tracing model training. This approach addresses the challenge of limited annotated data availability, a significant obstacle in automating neuron dendrite tracing. Our technique utilizes a multi-scale cascade process to generate synthetic images from single neuron tractograms, accurately replicating the complex characteristics of real microscopy images, encompassing imaging artifacts and background structures. In experiments, our method generates diverse images that mimic the characteristics of two distinct neuron microscopy datasets, which were successfully used as training data in the segmentation task of real neuron images.
The wide variability in the progression rates of Age-Related Macular Degeneration (AMD) and the absence of well-established clinical biomarkers make it difficult to predict an individual's risk of AMD progression from intermediate stage (iAMD) to late dry stage (dAMD) using Optical Coherence Tomography (OCT) scans. To address this challenge, we propose to jointly train an AMD stage classifier to discriminate between iAMD and dAMD with a Neural-ODE that models the future trajectory of the disease progression in the learned embedding space. A temporal ordering is imposed such that the distance of a scan from the decision hyperplane of the AMD stage classifier is inversely related to its time-to-conversion. In addition, an intra-subject temporal consistency in the predicted conversion risk scores is ensured by incorporating a pair of longitudinal scans from the same eye during training. We evaluated our proposed method on a longitudinal dataset comprising 235 eyes (3,534 OCT scans) with 40 converters. The results demonstrate the effectiveness of our approach, achieving an average area under the ROC of 0.84 for predicting conversion within the next 6, 12, 18 and 24 months. Additionally, the Concordance Index of 0.78 surpasses the performance of several popular methods for survival analysis.
