Dual-energy computed tomography (DECT) enables to generate a series of virtual monoenergetic images (VMIs). Using VMIs of a desired energy level (5 – 45 keV) can enhance the lesion-to-background and voxel-to-voxel within lesion contrast, because that the lesion material composition may vary from voxel to voxel. However, there are also strong correlation of the voxel values among different energy channels. This correlation may result in redundant information for the VMIs based lesion pathology differentiation. Therefore, we transformed the VMIs in the Karhunen–Loève domain to reduce the correlation. In the new domain, the leading three principal components accounts for more than 99% information and then were used to form a new descriptor for the differentiation task. Two pathological proven datasets were used for the evaluation. Experimental results showed that the VMIs can improved the AUC (area under the receiver operating characteristic curve) value from 0.862 and 0.647 to 0.912 and 0.830 comparing to using the conventional CT.
Osteoporosis is a complex multifactorial skeletal disease and has become a major socioeconomic issue, causing tremendous hospitalization and rehabilitation costs. In addition to age, premature menopause or the use of oral corticosteroids, vertebral fractures are regarded as main risk factors for developing osteoporosis and associated fractures. In this work, we adapt an existing automated pipeline to classify fracture grades of individual vertebrae and the spine as a whole: First, vertebral body centers were identified on CT images by a hierarchical neural network. Next, the fracture grades of individual vertebrae were processed by a multi-head, feed-forward convolutional neural network. The sum of the classified grades was then evaluated according to the German radiology guidelines. A hyperparameter search on validation data showed the most promising results for an output configuration based on three sequentially applied binary classification outputs trained using binary cross-entropy: Grade 0–1 vs 2–3, and 0 vs 1–3, and 0–2 vs 3. In a cross-validation setting on 159 low-dose CT images, our pipeline accurately classified patients to have sum-scores ≥ 2 with sensitivities and specificities of 90 % ± 5.0 % and 87 % ± 2.7 %, respectively. As our method was based on classifying individual vertebrae, we were able to provide both the fracture position and severity to enhance transparency, interpretability and usability.
The coronavirus disease 2019 (COVID-19) pandemic had a major impact on global health and was associated with millions of deaths worldwide. During the pandemic, imaging characteristics of chest X-ray (CXR) and chest computed tomography (CT) played an important role in the screening, diagnosis and monitoring the disease progression. Various studies suggested that quantitative image analysis methods including artificial intelligence and radiomics can greatly boost the value of imaging in the management of COVID-19. However, few studies have explored the use of longitudinal multi-modal medical images with varying visit intervals for outcome prediction in COVID-19 patients. This study aims to explore the potential of longitudinal multimodal radiomics in predicting the outcome of COVID-19 patients by integrating both CXR and CT images with variable visit intervals through deep learning. 2274 patients who underwent CXR and/or CT scans during disease progression were selected for this study. Of these, 946 patients were treated at the University of Pennsylvania Health System (UPHS) and the remaining 1328 patients were acquired at Stony Brook University (SBU) and curated by the Medical Imaging and Data Resource Center (MIDRC). 532 radiomic features were extracted with the Cancer Imaging Phenomics Toolkit (CaPTk) from the lung regions in CXR and CT images at all visits. We employed two commonly used deep learning algorithms to analyze the longitudinal multimodal features, and evaluated the prediction results based on the area under the receiver operating characteristic curve (AUC). Our models achieved testing AUC scores of 0.816 and 0.836, respectively, for the prediction of mortality.
Pulmonary emphysema is a form of Chronic Obstructive Pulmonary Disease (COPD) and a chronic lung condition that results in a breakdown of alveoli walls. Quantitative Computed Tomography (CT) is increasingly used to assess the presence or progression of emphysema. CT quantifications are affected by the acquisition protocols and scanner makes and models. This variability is a major concern for cross-sectional and longitudinal disease characterizations with largescale, multi-institutional datasets. Therefore, CT images need to be harmonized to reflect the patient condition and not the attributes of the imaging systems. The purpose of this study was to develop a physics-based harmonization framework that transforms CT images to a reference quality index (iso resolution and noise conditions) enabling robust emphysema quantifications across varied CT conditions. The harmonizer was developed using a virtual imaging trial (VIT) platform taking advantage of ground truth knowledge and control over CT parameters in VIT. The developed harmonizer was applied to clinical data from the COPDGene dataset to demonstrate its clinical utility. The established imaging biomarkers of “LAA-950” and “Perc15” were selected for emphysema quantifications. Results demonstrated that the harmonizer improved the quantification performance by reducing the bias in “LAA-950" from 7.03 (𝐶𝐼: [6.38, 7.68]) to 0.14 (𝐶𝐼: [0.08, 0.20]) after matching for kernel and from 2.48 (𝐶𝐼: [2.21, 2.76]) to −0.34 (𝐶𝐼: [−0.48, −0.20]) after matching for noise settings on the COPDGene dataset. This developed harmonization framework provides robust emphysema quantifications, enabling objective disease characterizations in large-scale, multi-center, and longitudinal studies.
Deep learning methods are the state-of-the-art for medical imaging segmentation tasks. Still, numerous segmentation algorithms based on heuristic-based methods have been proposed with exceptional results. To validate segmentation algorithms, manual annotations are typically considered as ground truth. However, manual annotations often suffer from inter/intra-operator variability and can also be occasionally inaccurate, especially when considering time-consuming and precise tasks. A sample case is the manual delineation of the lumen-intima (LI) and media-adventitia (MA) borders for intima-media thickness (IMT) measurement in B-mode ultrasound images. In this work, a novel hybrid learning paradigm which combines manual segmentations with the automatic segmentation of a dynamic programming technique for ground truth determination is presented. A profile consensus strategy is proposed to construct the hybrid ground truth. Two open-source datasets (n=2576) were employed for training four deep learning networks using the hybrid learning paradigm and three single source training targets as a comparison. The pipeline was fixed across the four tests and included a Faster R-CNN detection network to locate the carotid artery and then subsequent division into patches which were segmented using a UNet. The validation of the results was performed on an external test set comparing the predictions of the four different models to the annotations of three independent manual operators. The hybrid learning paradigm showed the best overall segmentation results (Dice=0.907±0.037, p<0.001) and demonstrated an exceptional correlation between the mean of three operators and the automatic measure (ICC(2,1)=0.958), demonstrating how the incorporation of heuristic-based segmentation methods within the learning paradigm of a deep neural network can enhance and improve final segmentation performance results.
Parkinson’s disease (PD) is the second most common neurodegenerative disease affecting 2-3% of the population over 65 years of age. Considerable research has investigated the benefit of using neuroimaging to improve PD diagnosis. However, it is challenging for medical experts to manually identify the subtle differences associated with PD in such complex data. It has been shown that machine learning models can achieve human-like accuracies for many computer-aided diagnosis applications. However, model performance usually depends on the amount and diversity of training data available, whereas most Parkinson’s disease classification models were trained on rather small datasets. Training data size and diversity can be increased by curating multi-site datasets. However, this may also increase biological and non-biological variances due to differences in participant cohorts, scanners, and data acquisition protocols. Thus, data harmonization is important to reduce those variances and enable the models to focus primarily on the patterns associated with PD. This work compares intensity harmonization techniques on 1796 MRI scans from twelve studies. Our results show that a histogram matching approach does not improve classification accuracy (78%) compared to the model trained on unharmonized data (baseline). However, it reduces the disparity between sensitivity and specificity from 81% and 73% to 77% and 79%, respectively. Moreover, combining histogram matching and least squares mean tissue intensity harmonization methods outperform the baseline model (accuracy of 74% compared to 67%) for an independent test set. Finally, our analysis considering sex (male, female) and groups (PD, healthy) shows that models trained on harmonized data exhibited reduced performance disparities between groups, which may be interpreted as a form of bias mitigation.
Myocardial Infarction (MI), commonly known as heart attack, is the irreversible death of the Myocardium’s tissue due to oxygen deprivation for an extended period of time. LGE-MRI scans are considered the defacto in the diagnosis and prognosis of MI. Still, they require manually segmenting the Myocardium and the infarcted tissue, which is a complex and time-consuming task. Hence, an automatic segmentation method of the Myocardium tissue is highly desirable. CNNs (Convolutional Neural Networks) are used extensively in cardiac tissue segmentation in general and for solving this problem particularly. EMIDEC (automatic Evaluation of Myocardial Infarction from Delayed Enhancement Cardiac MRI) challenge in MICCAI 2020 provides a good overview of the different CNN-based architectures used for MI segmentation. Still, they required complex pipelines to achieve state-of-the-art results and the research followed the challenge tends to go in the same direction of building more complex pipelines revolving around the famous CNN-UNet either 2D or 3D. In this paper, we present a different direction by presenting a simple 2D novel architecture based on Self-Attention Transformers offering a possible alternative to the CNN-UNet as a building block for bigger systems. We introduce NesT-UNet a novel segmentation architecture based on the NesT architecture as an encoder and we also introduce a novel decoder inspired by the same architecture. NesT achieved state-of-the-art results on ImageNet and CIFAR classification tasks with minimal training compared to other transformer networks. 2D NesT-UNet produced results comparable to the state-of-the-art on the EMIDEC dataset using a simple training process and an extra self-supervised pre-training step to improve the network’s performance. we also present a novel loss function for false positives reduction.
Due to the superior soft tissue contrast in magnetic resonance imaging (MRI), MRI may be well suited for renal mass characterization (e.g., benign vs. malignant). Though renal mass detection and characterization using deeplearning (DL) methods have been extensively studied for CT images, those same tasks are yet to be investigated on MR images. Existing algorithms for renal mass characterization require manual segmentation, therefore development of algorithms to localize and detect renal masses is important fully automatically. In this study, we developed a DL-based fully automated renal mass detection model on T2- weighted (T2W) images. In a cascaded approach, we initially segmented kidneys as a region-of-interest (ROI) using 2D U-Net model, then renal masses were detected on segmented kidneys using 2D U-Net convolutional neural network (CNN) model. We trained our model on randomly selected 80% of dataset using 5-fold cross-validation technique and evaluated on remaining 20% test cases for renal mass detection. Our T2W MRI dataset contained 108 patients with malignant (renal cell carcinoma- clear cell, papillary and chromophobe) and benign (fat poor angiomyolipoma-fpAML, oncocytomas) renal masses. The U-Net model for renal mass detection generated Dice similarity coefficient (DSC) of 90.00 ± 6.00 % (mean ± standard deviation). When localized kidneys evaluated on U-Net renal mass detection model yielded a sensitivity/recall, and specificity of 76.49% and 86.55%, respectively. Thus, our proposed fully automated cascaded approach has potential to be used as the first step in renal mass characterization study on T2W MRI images.
Accurate polyp segmentation from colonoscopy is essential for the early detection of colorectal cancer. However, the variety of polyps manifested in images and the blurry boundary between a polyp and its surrounding mucosa make segmentation challenging. Hence, it is crucial to accurately identify regional boundaries of polyps in colonoscopy. In this paper, we propose Gated Semantic Boundary Network (Gate-SBNet), which is a novel twostream CNN architecture based on an encoder-decoder framework to segment polyps in colonoscopy images. One stream of Gate-SBNet uses a pre-trained ConvNeXt-B model from the image classification task as the semantic encoder to obtain multi-level semantic features from colonoscopy images. Another branch uses the Semantic Boundary Learning Module (SBLM) to learn boundary features based on multi-level semantic features, which process information in parallel with the semantic encoder. By introducing the Gate Convolution Layer (GCLs) into the SBLM module, the semantic information is converted more accurately to boundary information. Therefore, only boundary-related information will be processed by the SBLM. Then, we merge semantic and boundary features as input to an Unet model to obtain the final segmentation result. Our proposed approach was evaluated on five benchmark datasets: Kvasir, CVC-ClinicDB, CVC-ColonDB, CVC-300, and ETIS-LaribPolypDB. Experiments have demonstrated that it is an efficient architecture and capable of making accurate predictions about object boundaries and significantly improving the performance of finding thin and small objects.
Non-small cell lung cancer (NSCLC) accounts for approximately 85% of lung cancer patients. Recurrence rate for NSCLC patients is 30%-50% with a significant risk of mortality. Predicting recurrence can lead to personalized target therapy. Prediction models using radiomic features extracted from CT images have been developed but have not shown optimal performance. Gabor filters are linear filters that improve the performance of texture classification. This work shows how Gabor features improve performance in models used to predict recurrence in NSCLC patients.
Architectural distortion (AD) is one of the important breast abnormal signs in digital breast tomosynthesis (DBT). It is hard to be detected due to its subtle appearance and similar intensity with surrounding tissue. To assist radiologists to detect ADs, a single-view based computer-aided detection model in DBT was developed by us previously. In this study, considering the fact that radiologists always use information from craniocaudal (CC) and mediolateral oblique (MLO) views of DBT simultaneously for better diagnosis of each breast in clinic, we further develop a multi-view based AD detection model in DBT that combines the information from the two views. In this model, AD candidates in each view are detected by our previous AD detection model. Anatomical position priors of AD candidates in the two views are considered through establishing a 3D anatomical coordinate system. A multi-view based classifier is trained to fuse information from the two views and distinguish the true AD candidates. A dataset of 196 CC-MLO DBT pairs were collected with IRB approval, 101 of them contained ADs and the remaining were negative pairs. Ten-fold cross-validation showed that after involving our proposed multi-view method, the sensitivities of AD detection at 1, 2, 3 and 4 false positive predictions per DBT pairs improved from 0.66, 0.73, 0.77 and 0.79 to 0.69, 0.77, 0.78, and 0.83, respectively. The results showed that the multi-view based model achieved better detection performance than single-view based model. This model has potential to assist radiologists in detection of ADs in DBT.
Deep learning models are widely studied for radiotherapy toxicity prediction; however, one of the major challenges is that they are complex models and difficult to understand. To aid in the creation of optimal dose treatment plans, it is critical to understand the mechanism and reasoning behind the network’s prediction, as well as the specific anatomical regions involved in toxicity. In this work, we propose a convolutional neural network to predict the toxicity after pelvic radiotherapy that is able to explain the network’s prediction. The proposed model analyses the dose treatment plan using multiple instance learning and convolutional encores. A dataset of 315 patients was included in the study, and experiments with both quantitative and qualitative approaches were conducted to assess the network’s performance.
Anterior cruciate ligament (ACL) is one of the most common injuries associated with sports. Knee osseous morphology can play a role in increased knee instability. Our hypothesis is that the morphological features of the knee, as seen in knee osseous morphology, can contribute to increased knee instability and, thus, increase the likelihood of ACL tear. To test this relationship, it is necessary to segment the femur and tibia bones and extract relevant imaging features. However, manual annotation of 3D medical images, such as on magnetic resonance imaging (MRI) scans, can be a time-consuming and challenging task. In this work, we propose an automated pipeline for creating pseudo-masks of the femur and tibia bones in knee MRI. Our approach involves unsupervised segmentation and deep learning models to classify ACL integrity (intact or torn). Our results demonstrate a high agreement between the automated pseudo-masks and a radiologist’s manual segmentation, which also leads to comparable AUC values for the ACL integrity classification.
COVID-19 still affects a large population worldwide with possible post-traumatic sequelae requiring long-term patient follow-up for the most severe cases. The lung is the primary target of severe acute respiratory syndrome coronavirus 2 (SARS- CoV-2) infection. In particular, the virus affects the entire pulmonary vascular tree from large vessels to capillaries probably leading to an abnormal vascular remodeling. In this study we investigated two modalities for assessing this remodeling, SPECT perfusion scintigraphy and computed tomography, the latter enabling the computation of vascular remodeling patterns. We analyzed on a cohort of 30 patients the relationship between vascular remodeling and perfusion defects in the peripheral lung area, which is a predominant focus of the COVID-19 infectious patterns. We found that such relationship exists, demonstrated by moderate significant correlations between SPECT and CT measures. In addition, a vascular remodeling index derived from the z-score normalized peripheral CT images showed a moderate significant correlation with the diffusing capacity of the lung for carbon monoxide (DLCO) measures. Altogether these results point CT scan as a good tool for a standardized, quantitative, and easy-to-use routine characterization and follow-up of COVID19-induced vascular remodeling. An extensive validation of these results will be carried out in the near future on a larger cohort.
The Corpus Callosum is the major interhemispheric commisure and, because of its highly organized fibers, it is often studied using diffusion tensor images (DTI). A firstnecessary step for CC studies is its segmentation, preferably automated. Since most available softwares are not able to perform CC volumetric segmentation, and the only ones that do it, are based on T1-weighted images and not DTI, this work presents the extension of an open-source software, called inCCsight, incorporating a DTI-based CC volumetric segmentation method into it. The software is open-source and offers the possibility of incorporating customized plots and integrating other segmentation and/or parcellation methods by the user.
Segmentation of medical images with known ground truth is useful for investigating properties of performance metrics and comparing different approaches of combining multiple manual segmentations to establish a reference standard, thereby informing selection of performance metrics and truthing methods. For medical images, however, segmentation ground truth is typically not available. One way of synthesizing segmentation errors is to use regular geometric objects as ground truth, but they lack the complexity and variability of real anatomical objects. To address this problem, we developed a medical image segmentation synthesis (MISS)-tool. The MISS-tool emulates segmentations by adjusting truth masks of anatomical objects extracted from real medical images. We categorized six types of segmentation errors and developed contour transformation tools with a set of user-adjustable parameters to modify the defined truth contours to emulate different types of segmentation errors, thereby generating synthetic segmentations. In a simulation study, we synthesized multiple segmentations to emulate algorithms or observers with pre-defined sets of segmentation errors (e.g., under/over-segmentation) using 220 lung nodule cases from the LIDC lung computed tomography dataset. We verified that the synthetic segmentation results manifest the type of errors that are consistent with our pre-configured setting. Our tool is useful for synthesizing a range of segmentation errors within a clinical segmentation task.
High breast density (BD) is recognized as an independent risk factor for breast cancer development, in addition to negatively impacting the sensitivity of mammography. Although BD is normally assessed with the BI-RADS reporting system, this evaluation is qualitative and has been shown to vary considerably across readers. In this pilot study, we present a deep learning (DL) method to quantify BD from a standard two-view (cranio-caudal, and medio-lateral-oblique) mammography exam. With the aim of developing a method based on an objective ground truth, the DL model was trained and validated using 88 simulated mammograms from an equal number of distinct 3D digital breast phantoms for which BD is known. The phantoms had been previously generated through segmentation and simulated mechanical compression of patient dedicated breast CT images, allowing for the exact calculation of BD in each case. Different data augmentations were applied prior to simulation, to increase the dataset size, yielding a total of 528 cases. These were divided, randomly and on a patient level, into training (N=360), validation (N=60), and test sets (N=108). The DL model performance was tested by stratifying the breasts into four different density ranges: 1-15%, 15-25%, 25-60%, and <60%. The median absolute errors and interquartile ranges (IQR), in percentage points, were 3.3 (IQR: 3.5), 3.4 (IQR: 2.5), 3.5 (IQR: 3.9), and 14.8 (IQR: 8.4), respectively. Although preliminary, these results show the potential of the proposed approach for accurate BD quantification, which is based, as opposed to most previously proposed approaches, on an objective ground truth.
Pulmonary vessel segmentation from CT images is essential to diagnosis and treatment of lung diseases, particularly in treatment planning and clinical outcome evaluation. The main challenge for pulmonary vessel segmentation is complicated structures of the vascular trees and their similar intensity values with other tissues like the tracheal wall and lung nodules. This paper presents a novel relation extractor U-shaped network combining convolution and self-attention mechanism in an encoder-decoder mode. Particularly, we employ convolution in the shallow layers to extract local information of vessels in a short range and apply self-attention in the deep layers to capture long-range contextual relationship between ancestors and descendants of the vascular tree. We evaluate our proposed method on 50 computer tomography volumes, with the experimental results showing that our method can improve the average coefficient dice and recall to 85.60 and 86.04 respectively.
When a CAD-AI network is created and employed, both safety and effectiveness need to be guaranteed for all subgroups of the target population. We present a novel toolbox for automatic slicing and performance assessment in a generic and modular approach, helping to find the subpopulations where cautiousness is warranted, and the model may need improvement. In a first step slices are generated and saved for further analysis inspired by the existing ‘Slice Finder’ algorithm. Depending on the type of AI task (classification, object detection, segmentation, instance segmentation...) multiple metrics are evaluated. Both labeled (specificity, sensitivity...), unlabeled (outlier score, confidence score...) and user-defined metrics can be included. Optionally, the confidence interval (CI) is calculated. In a last step, the metric values and CI are used to rank the slices to quickly find the slices of interest. Custom ranking methods can be added, keeping the full process from slice generation up to and including visualization modular and customizable. We illustrate the toolbox with a dermatology classification and object detection use-case. First a single model is evaluated down to crosses of three slices where slices of interest are detected on degree three which would be difficult to find if not automated. Additionally, the usage of unlabeled metrics such as outlier score is illustrated to automatically find slices of interest.
We are developing a decision support system for treatment response assessment of bladder cancer in CT urography (CTU). Accurate segmentation of bladder cancer is a critical and challenging task. We previously developed a bladder cancer segmentation method using a deep learning convolutional neural network and level set (DL-CNN+LS) approach. In this study we investigated the application of a U-Net based deep learning (U-Net) model for bladder cancer segmentation. Our new U-Net method did not require the second-stage level set refinement, greatly simplifying the overall segmentation pipeline. The proposed U-Net model utilized a user-defined box to direct the attention of the U-Net to the lesion region by masking out the structured background outside the box. We trained and evaluated the performance of the U-Net segmentations by using hand-drawn 3D contours from a radiologist as reference standard. The segmentation accuracy was evaluated by the average volume intersection ratio (AVI), average percent volume error (AVE), average absolute volume error (AAVE), average minimum distance (AMD), and the Jaccard index (JI). On the validation set, the cropped U-Net achieved values of AVI = 65.5±20.2%, AVE = 3.7±44.4%, AAVE = 30.9±30.8%, AMD = 2.9±1.5 mm, and JI = 51.4±16.8%. The previous DL-CNN+LS approach achieved values of AVI = 36.2±28.4%, AVE = 51.1±39.3%, AAVE = 52.0±38.0%, AMD = 4.1±1.7 mm, and JI = 32.9±27.2%. On the independent test set, the cropped U-Net achieved values of AVI = 66.0±22.8%, AVE = -3.7±39.4%, AAVE = 28.1±27.3%, AMD = 4.5±3.3 mm, and JI = 49.2±20.0%. The DL-CNN+LS achieved values of AVI = 31.2±24.3%, AVE = 59.9±30.9%, AAVE = 60.8±29.0%, AMD = 5.5±2.2 mm, and JI = 27.9±20.8%, respectively. The results demonstrated that the U-Net model could achieve a higher accuracy than the previous DL-CNN+LS model while reducing the complexity of the segmentation pipeline.
