Deep learning has advanced digital pathology and multi-omics data analysis. However, there is still a critical need to integrate these modalities to better understand complex diseases. Deep learning models, such as Prov-Gigapath, a state-of-the-art pathology foundation model, have been developed for analyzing whole-slide images (WSI). Our previously published deep-learning framework, OmicsFootPrint (OFP), transforms high-dimensional multi-omics data into intuitive genomically-arranged circular images and utilizes transfer learning to overcome the challenges of high-dimensional, low-sample-size data. In this study, we aimed to combine WSI and multi-omics data to develop a framework capable of distinguishing between invasive ductal carcinoma (IDC) and invasive lobular carcinoma (ILC), which is clinically significant as it may help explain differential responses to therapies, such as tamoxifen versus letrozole. We enhanced the performance of our OFP model by dividing each multi-omics circular image (1024x1024x3 pixels) into 16 equal-sized patches (256x256x3 pixels). Each patch was trained using a separate EfficientNetV2 architecture with pre-trained ImageNet weights. The probability outcomes from these 16 models were subsequently combined through a meta-learner model using the AutoGluon framework. Concurrently, WSI images corresponding to the same patients were independently trained, maintaining consistent train/validation/test splits. The Prov-Gigapath model, which was trained using a dataset from the Providence Health System, was employed to extract patch-level embeddings from the WSI data. These embeddings were fine-tuned using a LongNet-based masked autoencoder to predict outcomes. Finally, predictions from the OFP model and the WSI-based Prov-Gigapath model were integrated through a meta-learner, preserving the original train/validation/test split. Model performances (AUC) were evaluated using the held-out test set. The gold standard used for the analysis was histological identification of IDC versus ILC, against which the performance of omics data alone, WSI data alone, and the multi-modal integration approach were compared. We evaluated our integrated model as a proof-of-concept using The Cancer Genome Atlas (TCGA) breast cancer cohort, comprising 92 ILC and 391 IDC samples. Multi-omics data, including gene expression, microRNA, copy number variation, and RPPA, were analyzed using the OFP framework, while WSI data were processed using the Prov-Gigapath method. The results demonstrated that the multi-modal integration approach significantly outperformed individual modalities, achieving an AUC of 0.98 compared to 0.87 for OFP (original), 0.95 for OFP (16-patches), and 0.92 for WSI. Integrating WSI and multi-omics data improves predictive accuracy, significantly improving the classification of breast cancer subtypes over histology alone. Naresh Prodduturi, Xiaojia Tang, Hamid R. Tizhoosh, Kevin J. Thompson, Richard Weinshilboum, Karthik V. Giridhar, Judy C. Boughey, Eric W. Klee, Liewei Wang, Matthew P. Goetz, Vera Suman, Krishna R. Kalari. A multi-modal learning framework to integrate digital pathology image and multi-omics data in breast cancer [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2025; Part 1 (Regular Abstracts); 2025 Apr 25-30; Chicago, IL. Philadelphia (PA): AACR; Cancer Res 2025;85(8_Suppl_1):Abstract nr 2429.

Background Patients with peripheral artery disease are at increased risk for major adverse cardiac events, major adverse limb events, and all‐cause death. Developing tools capable of identifying those patients with peripheral artery disease at greatest risk for major adverse events is the first step for outcome prevention. This study aimed to determine whether computer‐assisted analysis of a resting Doppler waveform using deep neural networks can accurately identify patients with peripheral artery disease at greatest risk for adverse outcome events. Methods and Results Consecutive patients (April 1, 2015, to December 31, 2020) undergoing ankle–brachial index testing were included. Patients were randomly allocated to training, validation, and testing subsets (60%/20%/20%). Deep neural networks were trained on resting posterior tibial arterial Doppler waveforms to predict major adverse cardiac events, major adverse limb events, and all‐cause death at 5 years. Patients were then analyzed in groups based on the quartiles of each prediction score in the training set. Among 11 384 total patients, 10 437 patients met study inclusion criteria (mean age, 65.8±14.8 years; 40.6% women). The test subset included 2084 patients. During 5 years of follow‐up, there were 447 deaths, 585 major adverse cardiac events, and 161 MALE events. After adjusting for age, sex, and Charlson comorbidity index, deep neural network analysis of the posterior tibial artery waveform provided independent prediction of death (hazard ratio [HR], 2.44 [95% CI, 1.78–3.34]), major adverse cardiac events (HR, 1.97 [95% CI, 1.49–2.61]), and major adverse limb events (HR, 11.03 [95% CI, 5.43–22.39]) at 5 years. Conclusions An artificial intelligence–enabled analysis of Doppler arterial waveforms enables identification of major adverse outcomes among patients with peripheral artery disease, which may promote early adoption and adherence of risk factor modification.

The OmicsFootPrint framework addresses the need for advanced multi-omics data analysis methodologies by transforming data into intuitive two-dimensional circular images and facilitating the interpretation of complex diseases. Utilizing Deep Neural Networks and incorporating the SHapley Additive exPlanations (SHAP) algorithm, the framework enhances model interpretability. Tested with The Cancer Genome Atlas (TCGA) data, OmicsFootPrint effectively classified lung and breast cancer subtypes, achieving high Area Under Curve (AUC) scores - 0.98±0.02 for lung cancer subtype differentiation, 0.83±0.07 for breast cancer PAM50 subtypes, and successfully distinguishe between invasive lobular and ductal carcinomas in breast cancer, showcasing its robustness. It also demonstrated notable performance in predicting drug responses in cancer cell lines, with a median AUC of 0.74, surpassing existing algorithms. Furthermore, its effectiveness persists even with reduced training sample sizes. OmicsFootPrint marks an enhancement in multi-omics research, offering a novel, efficient, and interpretable approach that contributes to a deeper understanding of disease mechanisms.

AbstractContextBreast cancer is one of the most common cancers in women. With early diagnosis, some breast cancers are highly curable. However, the concordance rate of breast cancer diagnosis from histology slides by pathologists is unacceptably low. Classifying normal versus tumor breast tissues from microscopy images of breast histology is an ideal case to use for deep learning and could help to more reproducibly diagnose breast cancer. Since data preprocessing and hyperparameter configurations have impacts on breast cancer classification accuracies of deep learning models, training a deep learning classifier with appropriate data preprocessing approaches and optimized hyperparameter configurations could improve breast cancer classification accuracy.Methods and MaterialUsing 12 combinations of deep learning model architectures (i.e., including 5 non-specialized and 7 digital pathology-specialized model architectures), image data preprocessing, and hyperparameter configurations, the validation accuracy of tumor versus normal classification were calculated using theBreAstCancerHistology (BACH) dataset.ResultsThe DenseNet201, a non-specialized model architecture, with transfer learning approach achieved 98.61% validation accuracy compared to only 64.00% for the digital pathology-specialized model architecture.ConclusionsThe combination of image data preprocessing approaches and hyperparameter configurations have a profound impact on the performance of deep neural networks for image classification. To identify a well-performing deep neural network to classify tumor versus normal breast histology, researchers should not only focus on developing new models specifically for digital pathology, since hyperparameter tuning for existing deep neural networks in the computer vision field could also achieve a high (often better) prediction accuracy.

<p>Supplmentary File 1. genes from the integrated colon cancer co-expression network</p>

Supplementary File 4. Promoter regions from HUMAN NFAT family target genes shown in Figure 1E, showing location of NFAT consensus binding sequences

Introduction: Predictive algorithms using multiple clinical variables can identify patients at higher risk for abdominal aortic aneurysms (AAA), however they are cumbersome and cannot used directly by patients. A simple, cheap, noninvasive, and readily available (virtual) screening tool for identification of AAA could aid in screening algorithms. Aims: We sought to determine if a machine learning algorithm could predict the presence of an AAA by using a image of the face Methods: Diagnostic imaging studies were extracted from the electronic health record (EHR) from 5/7/2018 through 1/1/2023 and analyzed by a natural language processing algorithm previously validated to identify abdominal aortic aneurysms. Patient EHR profile pictures were extracted and matched to imaging studies. Various deep neural network (DNN) architectures were explored, all trained as classifiers on the face images along with clinical variables to predict clinical outcomes. All models used were CNNs with an EfficientNet architecture. Model performance was evaluated by standard metrics such as the area under the curve (AUC), specificities (Sp), and sensitivities (Sn). All performance metrics reported results from the test subset of the primary dataset and did not contain any observations considered during training. Results: A total of 5522 patients with facial images and AAA diagnostic studies were analyzed, among whom 1314 (23.8%) had a AAA. The optimal model had an AUC of 0.72 (Sn 0.44, Sp 0.83) for predicting AAA and remained similar in women (AUC 0.7), men (AUC 0.65), age >65 (AUC 0.63), and age <=65 (AUC 0.72) (Table). Conclusions: Using only an image of a patients face, the presence of an AAA could be identified with moderate performance in men, women, and in those less than 65. The model’s performance in demographics not typically screened for AAA may allow for targeted screening programs in these groups.

Supplementary File 3. Sixty-three core genes from the mouse metastatic model that are enriched in the developmental co-expression module

<p>Supplementary information. Supplementary methods and additional results as referred to in the manuscript</p>

<p>Supplmentary File 2. Genes in 441 co-expression modules identified within the integrated colon cancer co-expression network.</p>

<p>Supplementary File 5. Promoter regions from MURINE NFATc1 target genes showing the locations of NFAT consensus binding sequences and PCR primers.</p>

Data Supplement from Nuclear Factor of Activated T-cell Activity Is Associated with Metastatic Capacity in Colon Cancer

ABSTRACTWhole slide imaging (WSI) is transforming the practice of pathology, converting a qualitative discipline into a quantitative one. However, one must exercise caution in interpreting algorithm assertions, particularly in pathology where an incorrect classification could have profound impacts on a patient, and rare classes exist that may not have been seen by the algorithm during training. A more robust approach would be to identify areas of an image for which the pathologist should concentrate their effort to make a final diagnosis. This anomaly detection strategy would be ideal for WSI, but given the extremely high resolution and large file sizes, such an approach is difficult. Here, we combine progressive generative adversarial networks with a flexible adversarial autoencoder architecture capable of learning the “normal distribution” of WSIs of normal skin tissue at extremely high resolution and demonstrate its anomaly detection performance. Our approach yielded pixel-level accuracy of 89% for identifying melanoma, suggesting that our label-free anomaly detection pipeline is a viable strategy for generating high quality annotations - without tedious manual segmentation by pathologists. The code is publicly available at https://github.com/Steven-N-Hart/P-CEAD.

Background: Patients with peripheral artery disease (PAD) are at increased risk for major adverse limb and cardiac events including mortality. Developing screening tools capable of accurate PAD identification is a necessary first step for strategies of adverse outcome prevention. This study aimed to determine whether machine analysis of a resting Doppler waveform using deep neural networks can accurately identify patients with PAD. Methods: Consecutive patients (4/8/2015 – 12/31/2020) undergoing rest and postexercise ankle–brachial index (ABI) testing were included. Patients were randomly allocated to training, validation, and testing subsets (70%/15%/15%). Deep neural networks were trained on resting posterior tibial arterial Doppler waveforms to predict normal (> 0.9) or PAD (⩽ 0.9) using rest and postexercise ABI. A separate dataset of 151 patients who underwent testing during a period after the model had been created and validated (1/1/2021 – 3/31/2021) was used for secondary validation. Area under the receiver operating characteristic curves (AUC) were constructed to evaluate test performance. Results: Among 11,748 total patients, 3432 patients met study criteria: 1941 with PAD (mean age 69 ± 12 years) and 1491 without PAD (64 ± 14 years). The predictive model with highest performance identified PAD with an AUC 0.94 (CI = 0.92–0.96), sensitivity 0.83, specificity 0.88, accuracy 0.85, and positive predictive value (PPV) 0.90. Results were similar for the validation dataset: AUC 0.94 (CI = 0.91–0.98), sensitivity 0.91, specificity 0.85, accuracy 0.89, and PPV 0.89 (postexercise ABI comparison). Conclusion: An artificial intelligence-enabled analysis of a resting Doppler arterial waveform permits identification of PAD at a clinically relevant performance level.

Background: Adoption of the Digital Imaging and Communications in Medicine (DICOM) standard for whole slide images (WSIs) has been slow, despite significant time and effort by standards curators. One reason for the lack of adoption is that there are few tools which exist that can meet the requirements of WSIs, given an evolving ecosystem of best practices for implementation. Eventually, vendors will conform to the specification to ensure enterprise interoperability, but what about archived slides? Millions of slides have been scanned in various proprietary formats, many with examples of rare histologies. Our hypothesis is that if users and developers had access to easy to use tools for migrating proprietary formats to the open DICOM standard, then more tools would be developed as DICOM first implementations. Methods: The technology we present here is dicom_wsi, a Python based toolkit for converting any slide capable of being read by the OpenSlide library into DICOM conformant and validated implementations. Moreover, additional postprocessing such as background removal, digital transformations (e.g., ink removal), and annotation storage are also described. dicom_wsi is a free and open source implementation that anyone can use or modify to meet their specific purposes. Results: We compare the output of dicom_wsi to two other existing implementations of WSI to DICOM converters and also validate the images using DICOM capable image viewers. Conclusion: dicom_wsi represents the first step in a long process of DICOM adoption for WSI. It is the first open source implementation released in the developer friendly Python programming language and can be freely downloaded at https://github.com/Steven N Hart/dicom_wsi.

Gestational trophoblastic disease (GTD) is a heterogeneous group of lesions arising from placental tissue. Epithelioid trophoblastic tumor (ETT), derived from chorionic-type trophoblast, is the rarest form of GTD with only approximately 130 cases described in the literature. Due to its morphologic mimicry of epithelioid smooth muscle tumors and carcinoma, ETT can be misdiagnosed. To date, molecular characterization of ETTs is lacking. Furthermore, ETT is difficult to treat when disease spreads beyond the uterus. Here using RNA-Seq analysis in a cohort of ETTs and other gestational trophoblastic lesions we describe the discovery of LPCAT1-TERT fusion transcripts that occur in ETTs and coincide with underlying genomic deletions. Through cell-growth assays we demonstrate that LPCAT1-TERT fusion proteins can positively modulate cell proliferation and therefore may represent future treatment targets. Furthermore, we demonstrate that TERT upregulation appears to be a characteristic of ETTs, even in the absence of LPCAT1-TERT fusions, and that it appears linked to copy number gains of chromosome 5. No evidence of TERT upregulation was identified in other trophoblastic lesions tested, including placental site trophoblastic tumors and placental site nodules, which are thought to be the benign chorionic-type trophoblast counterpart to ETT. These findings indicate that LPCAT1-TERT fusions and copy-number driven TERT activation may represent novel markers for ETT, with the potential to improve the diagnosis, treatment, and outcome for women with this rare form of GTD.

Purpose: Deep learning models are showing promise in digital pathology to aid diagnoses. Training complex models requires a significant amount and diversity of well-annotated data, typically housed in institutional archives. These slides often contain clinically meaningful markings to indicate regions of interest. If slides are scanned with the ink present, then the downstream model may end up looking for regions with ink before making a classification. If scanned without the markings, the information regarding where the relevant regions are located is lost. A compromise solution is to scan the slide with the annotations present but digitally remove them. Approach: We proposed a straightforward framework to digitally remove ink markings from whole slide images using a conditional generative adversarial network based on Pix2Pix. Results: The peak signal-to-noise ratio increased 30%, structural similarity index increased 20%, and visual information fidelity increased 200% relative to previous methods. Conclusions: When comparing our digital removal of marked images with rescans of clean slides, our method qualitatively and quantitatively exceeds current benchmarks, opening the possibility of using archived clinical samples as resources to fuel the next generation of deep learning models for digital pathology.

For many disease conditions, tissue samples are colored with multiple dyes and stains to add contrast and location information for specific proteins to accurately identify and diagnose disease. This presents a computational challenge for digital pathology, as whole-slide images (WSIs) need to be properly overlaid (i.e. registered) to identify co-localized features. Traditional image registration methods sometimes fail due to the high variation of cell density and insufficient texture information in WSIs-particularly at high magnifications. In this paper, we proposed a robust image registration strategy to align re-stained WSIs precisely and efficiently. This method is applied to 30 pairs of immunohistochemical (IHC) stains and their hematoxylin and eosin (H&E) counterparts. Our approach advances the existing methods in three key ways. First, we introduce refinements to existing image registration methods. Second, we present an effective weighting strategy using kernel density estimation to mitigate registration errors. Third, we account for the linear relationship across WSI levels to improve accuracy. Our experiments show significant decreases in registration errors when matching IHC and H&E pairs, enabling subcellular-level analysis on stained and re-stained histological images. We also provide a tool to allow users to develop their own registration benchmarking experiments.