The detection of prostate cancer within H&E-stained tissue slides helps identify the presence, location, and extent of the disease. Machine learning approaches have been developed to accomplish this task for biopsies and radical prostatectomies. Deep learning approaches, primarily using convolutional neural networks (CNN), have demonstrated substantial ability to identify cancer in each of these sample types. Some models trained on biopsies have even been approved for clinical application. Biopsies are small tissue samples acquired through a syringe to establish cancer diagnosis while radical prostatectomies are large planar sections of prostate tissue acquired post-surgery. This leads to differing morphology and heterogeneity between sample types, making it unclear whether algorithms trained on biopsies are robust enough to perform well on radical prostatectomies and vice versa. Our goal was to investigate whether morphological differences between sample types affected the performance of radical prostatectomy trained CNN models when applied to biopsies and vice versa. Radical prostatectomies (N=100) and biopsies (N=50) were acquired from The University of Pennsylvania to train (80%) and validate (20%) a CNN using the Densenet architecture for biopsies (MB), radical prostatectomies (MR), and a combined dataset (MB+R). Model performance was compared using sensitivity, specificity, and F1 score. To isolate the effect of morphological differences on performance, we acquired all data from the same institution, ensured no batch effects were present via UMAP, applied data augmentation during training, and applied stain normalization. Additionally, the radical prostatecomy training set was optimized for transfer to biopsies by reducing heterogeneity (MR*). This reduction was based on distance from the centroid of a texture feature distribution. Models performed well when applied to their own sample type (F1 > 0.88) but performed poorly when applied across sample types (F1 < 0.65). Reducing heterogeneity to optimize the radical prostatectomy training set resulted in MR* (F1=0.72) noticeably outperforming MR (F1=0.53) on biopsies. This indicates that differences in morphology and heterogeneity drive performance differences between cancer detection models trained on different sample types. Our results suggest that these morphological differences can be overcome with sample type specific models or training set optimization methods.
Detecting the background is often one of the first tasks when analyzing whole slide images. However, these methods greatly vary and are generally not reported in sufficient detail for subsequent studies to be able to reproduce. Therefore, we sought to determine the effect of varying background detection methods on downstream performance metrics. Specifically, we applied attention-based multiple instance learning, CLAM, and Attention2majority to classify whole slide images from Camelyon16 using four different background detection methods. Our results show that performance metrics (i.e., accuracy and area under the curve - AUC) differ when different background detection methods are utilized. Furthermore, all classification methods perform worse when using hand-drawn tissue annotations for background detection. Finally, we show that depending on the background detection method, the best performing method changes. We conclude that background detection methods must be fully reported so that results can be accurately reproduced and techniques can be fairly compared.
Accurate quantification of renal fibrosis has profound importance in the assessment of chronic kidney disease (CKD). Visual analysis of a biopsy stained with trichrome under the microscope by a pathologist is the gold standard for evaluation of fibrosis. Trichrome helps to highlight collagen and ultimately interstitial fibrosis. However, trichrome stains are not always reproducible, can underestimate collagen content and are not sensitive to subtle fibrotic patterns. Using the Dual-mode emission and transmission (DUET) microscopy approach, it is possible to capture both brightfield and fluorescence images from the same area of a tissue stained with hematoxylin and eosin (H&E) enabling reproducible extraction of collagen with high sensitivity and specificity. Manual extraction of spectrally overlapping collagen signals from tubular epithelial cells and red blood cells is still an intensive task. We employed a UNet++ architecture for pixel-level segmentation and quantification of collagen using 760 whole slide image (WSI) patches from six cases of varying stages of fibrosis. Our trained model (Deep-DUET) used the supervised extracted collagen mask as ground truth and was able to predict the extent of collagen signal with a MSE of 0.05 in a holdout testing set while achieving an average AUC of 0.94 for predicting regions of collagen deposits. Expanding this work to the level of the WSI can greatly improve the ability of pathologists and machine learning (ML) tools to quantify the extent of renal fibrosis reproducibly and reliably.
Reference histomorphometric data of healthy human kidneys are lacking due to laborious quantitation requirements. We leveraged deep learning to investigate the relationship of histomorphometry with patient age, sex, and serum creatinine in a multinational set of reference kidney tissue sections. A panoptic segmentation neural network was developed and used to segment viable and sclerotic glomeruli, cortical and medullary interstitia, tubules, and arteries/arterioles in digitized images of 79 periodic acid-Schiff (PAS)-stained human nephrectomy sections showing minimal pathologic changes. Simple morphometrics (e.g., area, radius, density) were measured from the segmented classes. Regression analysis was used to determine the relationship of histomorphometric parameters with age, sex, and serum creatinine. The model achieved high segmentation performance for all test compartments. We found that the size and density of nephrons, arteries/arterioles, and the baseline level of interstitium vary significantly among healthy humans, with potentially large differences between subjects from different geographic locations. Nephron size in any region of the kidney was significantly dependent on patient creatinine. Slight differences in renal vasculature and interstitium were observed between sexes. Finally, glomerulosclerosis percentage increased and cortical density of arteries/arterioles decreased as a function of age. We show that precise measurements of kidney histomorphometric parameters can be automated. Even in reference kidney tissue sections with minimal pathologic changes, several histomorphometric parameters demonstrated significant correlation to patient demographics and serum creatinine. These robust tools support the feasibility of deep learning to increase efficiency and rigor in histomorphometric analysis and pave the way for future large-scale studies.
Circulating Tumor Cells, or CTCs, are cancerous cells that shed from a primary tumor and intravasate into the bloodstream. This type of screening, otherwise known as biopsy screening, is effective at determining the early stages of cancer and discussing treatment. Expert cytopathologists have been required to look at these images to screen for cancer. Anything between the numbers of hundreds to thousands of images must be gone through. At a heavy time cost and a basis of work effort, we thus proposed an idea using a U-Net to screen for these CTCs and enumerate them in a more efficient and time-costing method. The Ball-scale transform technique, a filter that allows us to determine the maximum sphericality in a thresholded homogeneity, was introduced into this field of digital pathology alongside our proposed novel deep learning-based (UNet) CTC detection and enumeration approach. We collected 466 images for CTC detection and another 198 images with 323 CTCs for testing CTC enumeration. We investigated two ways to use the Ball-scale image: using B-scale images in the input channel of deep learning and using B-scale images in the output layer by providing high-level information (size and shape) encoded in the B-scale image itself to do the enumeration. We also tested deep learning-based CTC detection by using different labels. Results show that our method is much better than those which utilize thresholding with a missing rate comparison of 0.04 to 0.30. Meanwhile, our method is certainly comparable and competitive with the results in recent publications and may facilitate other types of research.
Whole slide imaging (WSI), also called digital virtual microscopy, is a new imaging modality. It allows for the application of AI and machine learning methods to cancer pathology to help establish a means for the automatic diagnosis of cancer cases. However, designing machine-learning models for WSI is computationally challenging due to its required ultra-high resolution. The current state-of-the-art models use multiple instance learning (MIL). MIL is a weakly-supervised learning method in which the model uses an array of inferences from many smaller instances to make a final classification about the entire set. In the context of WSI, researchers divide the ultra-high-resolution image into many patches. The model then classifies the slide based on an array of inferences from the patches. Among several ways of making the final classification, attention-based mechanisms have resulted in superb accuracy scores. The Transformer, one attention-based algorithm, has reported substantial improvements for WSI comprehension tasks. In this project, we studied and compared several WSI comprehension algorithms. We used the following three datasets: CAMELYON16+17, TCGALung, and TCGA-Kidney. We found that attention-based MIL algorithms performed better than standard MIL algorithms for classifying WSI images, achieving a higher mean accuracy and AUC. However, none of the attention-based algorithms performed significantly better than the others, reporting accuracy scores that varied widely. Presumably, it is due to the limited availability of training samples in the data corpus. Since it is not easy to increase the samples from human subjects, some machine learning techniques like transfer learning could help mitigate this issue.
The effectiveness of untrained convolutional layers for feature extraction in a computational pathology task using real-world data from a necrosis detection dataset is investigated. The study aims to determine whether ImageNet pretrained layers from deep CNNs combined with frozen untrained weights are sufficient for effective necrosis detection in canine Perivascular Wall Tumour (cPWT) whole slide images. Additionally, the authors investigate the impact of pruning CNNs, and whether it can be effective for necrosis detection as this technique can contribute towards reducing memory requirements and improve inference speed in diagnostic settings. The study found that fine-tuning the last (deepest) layers of a pretrained ImageNet model for necrosis detection in cPWT produces the highest test F1-score (0.715) when compared to alternative set ups. This score is further improved to 0.754 when the results are optimised using an optimal threshold predetermined on maximising the validation set F1-score. Resetting weights (untrained) and freezing the last few convolutional layers in the last dense block also demonstrated some capability in necrosis detection with an optimised F1-score of 0.747, still outperforming models trained from scratch as well as an ImageNet pretrained feature extraction model. Pruning the fine-tuned model using lower thresholds also showed the potential to improve performance, however thresholds higher than 0.40 negatively impacted performance.
In situ RNA capturing represents an excellent opportunity for bridging transcriptomic data to a spatial domain, making it possible to map the gene expression to the corresponding anatomical structure. As a result, scientists can better understand the transcriptional heterogeneity with spatially resolved, anatomical, and pathophysiological contexts. However, high throughput sequencing technologies paired with histological images suffer from lower resolution mapping between transcriptome and imaging data. Here, we present Spatial Transcriptome Auto-encoder & Deconvolution (ST-AnD), a scalable deep generative model for predicting gene expression at cellular or nuclei level based on H&E imaging and in situ RNA capturing, thus allowing a better understanding of the tissue microenvironment.
Deep-learning techniques have been used widely to alleviate the labour-intensive and time-consuming manual annotation required for pixel-level tissue characterization. Our previous study introduced an efficient single dynamic network - Omni-Seg - that achieved multi-class multi-scale pathological segmentation with less computational complexity. However, the patch-wise segmentation paradigm still applies to Omni-Seg, and the pipeline is time-consuming when providing segmentation for Whole Slide Images (WSIs). In this paper, we propose an enhanced version of the Omni-Seg pipeline in order to reduce the repetitive computing processes and utilize a GPU to accelerate the model's prediction for both better model performance and faster speed. Our proposed method's innovative contribution is two-fold: (1) a Docker is released for an end-to-end slide-wise multi-tissue segmentation for WSIs; and (2) the pipeline is deployed on a GPU to accelerate the prediction, achieving better segmentation quality in less time. The proposed accelerated implementation reduced the average processing time (at the testing stage) on a standard needle biopsy WSI from 2.3 hours to 22 minutes, using 35 WSIs from the Kidney Tissue Atlas (KPMP) Datasets. The source code and the Docker have been made publicly available at https://github.com/ddrrnn123/Omni-Seg.
Tissue examination and quantification in a 3D context on serial section whole slide images (WSIs) were labor-intensive and time-consuming tasks. Our previous study proposed a novel registration-based method (Map3D) to automatically align WSIs to the same physical space, reducing the human efforts of screening serial sections from WSIs. However, the registration performance of our Map3D method was only evaluated on single-stain WSIs with large-scale kidney tissue samples. In this paper, we provide a Docker for an end-to-end 3D slide-wise registration pipeline on needle biopsy serial sections in a multi-stain paradigm. The contribution of this study is three-fold: (1) We release a containerized Docker for an end-to-end multi-stain WSI registration. (2) We prove that the Map3D pipeline is capable of sectional registration from multi-stain WSI. (3) We verify that the Map3D pipeline can also be applied to needle biopsy tissue samples. The source code and the Docker have been made publicly available at https://github.com/hrlblab/Map3D.
The Tangram algorithm is a benchmarking method of aligning single-cell (sc/snRNA-seq) data to various forms of spatial data collected from the same region. With this data alignment, the annotation of the single-cell data can be projected to spatial data. However, the cell composition (cell-type ratio) of the single-cell data and spatial data might be different because of heterogeneous cell distribution. Whether the Tangram algorithm can be adapted when the two data have different cell-type ratios has not been discussed in previous works. In our practical application that maps the cell-type classification results of single-cell data to the Multiplex immunofluorescence (MxIF) spatial data, cell-type ratios were different, though they were sampled from adjacent areas. In this work, both simulation and empirical validation were conducted to quantitatively explore the impact of the mismatched cell-type ratio on the Tangram mapping in different situations. Results show that the cell-type difference has a negative influence on classification accuracy.
Hyperspectral imaging is a label-free and non-invasive imaging modality that seeks to capture images in different wavelengths. In this study, we used a vision transformer that was pre-trained from video data to detect thyroid cancer on hyperspectral images. We built a dataset of 49 whole slide hyperspectral images (WS-HSI) of thyroid cancer. To improve training, we introduced 5 new data augmentation methods that transform spectra. We achieved an F-1 score of 88.1% and an accuracy of 89.64% on our test dataset. The transformer network and the whole slide hyperspectral imaging technique can have many applications in digital pathology.
Hyperspectral imaging (HSI) is an emerging modality for digital pathology. The purpose of this study is to develop an extended depth of field (EDOF) method for mosaic hyperspectral images acquired with a snapshot camera. EDOF is a technique for ensuring that an image is in focus at all points. A stack of mosaicked hyperspectral images of hematoxylin and eosin (H&E)-stained histologic slides were acquired at different positions along the z-axis and used to output a hyperspectral histologic image that was in-focus at every point. Three different methods were compared to achieve a fully focused image. We compared conventional patch-based methods to our proposed growth-based and band-based methods. The Brenner function was used to quantitatively measure the focus quality of each image measured. The results show that both of our proposed methods performed better qualitatively and quantitatively than the patch-based method, with the band-based method performing the best, as it leveraged dividing pixels into their proper wavelengths in addition to spatially, giving the algorithm better contrast to measure. In terms of speed, the band-based method was the fastest, followed by the patch-based method, with the growth-based method being the slowest. Our proposed extended depth of field hyperspectral imaging methods can have immediate applications in digital pathology, especially whole slide imaging, and other microscopic imaging.
The aim of this study is to develop a method to transform hyperspectral images of H&E-stained slides to natural-color RGB histology images for easy visualization. Hyperspectral images were obtained at 40× objective magnification and downsampled by various factors to generate data equivalent to different magnifications. High-resolution digital histologic RGB images were cropped and registered to the corresponding hyperspectral images as the ground truth. A conditional generative adversarial network (cGAN) was trained to output natural color RGB images of the histological tissue samples. The generated synthetic RGBs have similar color and sharpness to real RGBs. Image classification was implemented using the real and synthetic RGBs, respectively, with a pretrained network. The classification of tumor and normal tissue using the HSI-synthesized RGBs yielded a comparable but slightly higher accuracy and AUC than the real RGBs.
ER, PR (estrogen, progesterone receptor), and HER2 (human epidermal growth factor receptor 2) status are assessed using immunohistochemistry and reported in standard clinical workflows as they provide valuable information to help treatment planning. The protein Ki67 has also been suggested as a prognostic biomarker but is not routinely evaluated clinically due to insufficient quality assurance. The routine pathological practice usually relies on small biopsies, such that the reduction in consumption is necessary to save materials for special assays. For this purpose, we developed and validated an automatic system for segmenting and identifying the (ER, PR, HER2, Ki67) positive cells from hæmatoxylin and eosin (H&E) stained tissue sections using multiplexed immunofluorescence (MxIF) images at cellular level as a reference standard. In this study, we used 100 tissue-microarray cores sampled from 56 cases of invasive breast cancer. For ER, we extracted cell nucleus images (HoverNet) from the H&E images and assigned each cell nucleus as ER positive vs. negative based on the corresponding MxIF signals (whole cell segmentation with DeepCSeg) upon H&E to MxIF image registration. We trained a Res-Net 18 and validated the model on a separate test-set for classifying the cells as positive vs. negative for ER, and performed the same experiment for the other three markers. We obtained area-under-the- receiver-operating-characteristic-curves (AUCs) of 0.82 (ER), 0.85 (PR), 0.75 (HER2), 0.82 (Ki67) respectively. Our study demonstrates the feasibility of using machine learning to identify molecular status at cellular level directly from the H&E slides.
Nowadays, the availability of different types of biomedical digital data offers many opportunities to investigate the relationships between the different modalities and thus develop a more comprehensive understanding of complex diseases such as cancer. In this paper, we propose a multi-modal model, called deep modality association learning (DMAL), that maps immune cell sequencing patterns to morphological tissue features of whole slide imageds (WSIs) in an embedding space. Useful information is extracted from T-cell receptor (TCR) sequences to guide the training process. DMAL maps the TCR features to the morphology features in histopathology images, which in turn enables the model to learn the association features between the two modalities. The discrimination power of the WSI-TCR association features has been assessed by classifying samples with different cancer subtypes. The conducted experiments have shown that DMAL generates more discriminative features compared to features obtained from single-modal data. In addition, DMAL has been utilized to predict TCR information from histopathology image representations without the need to have the actual TCR sequencing data.
CMYC positivity is an important prognostic factor for diffuse large B-cell lymphoma. However, manual quantification of CMYC can be subjective and may show intra- and inter-observer variability. Therefore, we sought to develop an automated method to quantify CMYC. Our method applies attention-based multiple instance learning to regress the proportion of CMYC positive tumor cells from pathologist-scored tissue microarray cores. The results of our experiments indicate a high Pearson correlation of 0.8421+/-0.1268. Additionally, we show that regardless of cross-validation methodology, this correlation remains relatively stable. When utilizing a standard clinical threshold of 40% for positivity, our method results in a sensitivity and specificity of 0.7600 and 0.9595. Finally, using clinical outcomes, we found that regressions provided more significant and robust stratification when compared to pathologist scoring. We conclude that proportion of positive stain can be regressed using attention-based multiple instance learning.
The tumor microenvironment (TME) plays an important role in driving cancer progression and affecting treatment efficacy. Cellular components of the TME include various immune subsets (tumor infiltrating lymphocytes (TILs) and macrophages), cancer-associated fibroblasts (CAFs) and vascular cells. While immune lineage has been a main focus of intensive research on the TME, CAFs have also been shown to be highly heterogeneous in their molecular phenotype and function. Using a protein marker immunofluorescence multiplexing system (Cell DIVE, Leica Microsystems) and quantitative imaging tools, we investigated the identity of various CAF clusters based on the expression of α-Smooth Muscle Actin (αSMA) and Fibroblast Activation Protein (FAP), and compared their distributions across breast cancer subtypes. We determined the cell counts of various CAF subsets using binary counting and identified the heterogeneous presentations of clusters using K-means clustering and Uniform Manifold Approximation and Projection (UMAP). We found that the abundance of CAF clusters varied among breast cancer subtypes. An integrated analysis of CAF cluster composition in each cancer and the transcriptomic data of CAF-associated genes such as CD29, IL6 and PDGFRβ was performed. We observed increased densities of proliferative, αSMA-positive CAFs in basal-like breast cancers that exhibited a co-expression signature of CAF-associated genes. Finally, an association analysis of CAF cluster composition and gene expression with previously identified radiomic phenotype was performed, but significant correlation was not detected.
Diabetic nephropathy (DN) in the context of type 2 diabetes is the leading cause of end-stage renal disease (ESRD) in the United States. DN is graded based on glomerular morphology and has a spatially heterogeneous presentation in kidney biopsies that complicates pathologists' predictions of disease progression. Artificial intelligence and deep learning methods for pathology have shown promise for quantitative pathological evaluation and clinical trajectory estimation; but, they often fail to capture large-scale spatial anatomy and relationships found in whole slide images (WSIs). In this study, we present a transformer-based, multi-stage ESRD prediction framework built upon nonlinear dimensionality reduction, relative Euclidean pixel distance embeddings between every pair of observable glomeruli, and a corresponding spatial self-attention mechanism for a robust contextual representation. We developed a deep transformer network for encoding WSI and predicting future ESRD using a dataset of 56 kidney biopsy WSIs from DN patients at Seoul National University Hospital. Using a leave-one-out cross-validation scheme, our modified transformer framework outperformed RNNs, XGBoost, and logistic regression baseline models, and resulted in an area under the receiver operating characteristic curve (AUC) of 0.97 (95% CI: 0.90-1.00) for predicting two-year ESRD, compared with an AUC of 0.86 (95% CI: 0.66-0.99) without our relative distance embedding, and an AUC of 0.76 (95% CI: 0.59-0.92) without a denoising autoencoder module. While the variability and generalizability induced by smaller sample sizes are challenging, our distance-based embedding approach and overfitting mitigation techniques yielded results that suggest opportunities for future spatially aware WSI research using limited pathology datasets.
