Integrating molecular, morphological, and clinical data is essential for basic and translational biomedical research, yet systematic frameworks for jointly modeling these modalities remain limited. Here we present Haiku, a tri-modal contrastive learning model trained on multiplexed immunofluorescence (mIF). It comprises 26.7 million spatial proteomics patches from 3,218 tissue sections across 1,606 patients spanning 11 organ types, with matched hematoxylin and eosin (H E) histology and clinical metadata aligned in a shared embedding space. Haiku enables three-way cross-modal retrieval, improves downstream classification and clinical prediction tasks over unimodal baselines, and supports zero-shot biomarker inference through fusion retrieval conditioned on clinical metadata-only text descriptions. Across tasks, Haiku outperforms competing approaches, achieving cross-modal retrieval (Recall@50 up to 0.611 versus near-zero baseline), survival prediction (C-index 0.737, +7.91

ABSTRACT:The bone marrow microenvironment (BMME) is essential for hematopoiesis and immunity, yet spatiotemporal single-cell analysis during leukemogenesis remains challenging. We characterized the BMME in femurs from wild-type and chronic myeloid leukemia (CML) mice at 7, 14, and 21 days after induction by highly multiplexed and 3-dimensional (3D) microscopy. Using a 54-marker codetection by indexing panel, we profiled 2 033 725 cells in 55 regions of interest and identified 41 cell types. During CML progression, we observed myeloid and progenitor cell expansion, increased programmed death ligand 1-positive leukemic cells, programmed death 1 (PD-1) upregulation on CD4+ and CD8+ T cells, and a profound loss of B cells, plasma cells, and bone cells. Advanced CML exhibited a striking expansion of immature, pericyte-deficient vasculature that disrupted vascular niches and impaired hematopoietic stem/progenitor cell positioning. Spatial mapping revealed leukemia-specific cellular neighborhoods enriched in PD-1+CD8+ T cells, suggesting localized immune exhaustion. Early CML showed increased contacts between plasmacytoid dendritic cells and megakaryocytes (MKs), whereas advanced CML featured heightened MK emperipolesis of nonleukemic granulocytes. MKs were morphologically irregular in CML mice and patient bone marrow biopsies. In contrast, in mice with acute myeloid leukemia, vasculature and MKs were reduced, whereas the remaining MKs retained normal morphology. Laser-capture microdissected MKs from patients with newly diagnosed CML had reduced cytoskeleton gene expression, which was reversed in advanced cases treated with tyrosine kinase inhibitors. 3D imaging revealed vascular disorganization and depleted MKs in the diaphysis, underscoring region-specific pathology. Together, this study provides a spatiotemporal single-cell atlas of the BMME during leukemic progression, showing how leukemic cells reprogram it to support their expansion and immune evasion.

Immune checkpoint blockers (ICB) improve outcomes in metastatic melanoma (MM), but resistance limits benefit. This phase I/II (NCT02706353) study evaluated intratumoral sotigalimab (anti-CD40 agonist) with pembrolizumab in 32 patients with ICB-naïve MM. Primary endpoints were safety and objective response rate (ORR). Sotigalimab was well tolerated. At the recommended phase II dose, the ORR was 50%, and the disease control rate was 92%, with ORRs of 67% in injected and 50% in non-injected tumors. Multiomic analyses of tumor and blood showed that sotigalimab effectively engaged the CD40 pathway, boosting infiltration and activation of myeloid cells, including CD11c+DC-LAMP+ dendritic cells and macrophages. The combination therapy activated innate and adaptive immunity in injected tumors and cytotoxic responses in non-injected tumors. T-cell receptor sequencing showed increased T-cell clonality with expanded new clones shared across tumors. Clinical responses correlated with these immunologic changes but not with baseline features associated with response to anti-PD-1 monotherapy. SIGNIFICANCE:In this study, we provide compelling data that intratumoral sotigalimab combined with pembrolizumab is safe, activates antigen-presenting cells, and elicits broad innate and adaptive immune responses in both injected and non-injected tumors, supporting further randomized phase II trials to evaluate sotigalimab's potential to enhance anti-PD-1 therapy through "in situ" immunization.

Abstract Hematoxylin and eosin (H&E) stained images are fundamental clinical tools for disease assessment. However, even with advanced computational models, their prognostic capabilities remain limited. Spatial omics characterizes tumor microenvironments (TME) in detail yet remains clinically inaccessible due to cost and complexity. In this study, we present HOPE, a lightweight framework that learns TME signatures from paired H&E and spatial omics data during training, then applies these to H&E alone at inference. Leveraging H&E foundation models, HOPE consistently outperforms identical architectures trained without spatial omics guidance across cancer types and cohorts. It further generates interpretable annotations of TME signature on H&E regions, stratifying patients into biologically coherent groups with different prognostic outcomes. HOPE establishes a practical route to translate high-content spatial omics discoveries into scalable, clinically deployable tools.

Spatial omics provides unprecedented high-resolution insights into molecular tissue compositions but poses significant analytical challenges due to massive data volumes, complex hierarchical spatial structures, and domain-specific interpretive demands. To address these limitations, we introduce OmicsNavigator, an LLM-driven multi-agent system that autonomously distills expert-level biological insights from raw spatial omics data without domain-specific fine-tuning. OmicsNavigator encodes spatial data into concise natural language summaries, enabling zero-shot annotation of structural components, quantitative analysis of pathological relevance, and semantic search of regions of interest using free-form text queries. We evaluated OmicsNavigator on multiple spatial omics studies of kidney cohorts with different phenotypes and biomarker panels, where OmicsNavigator achieved outstanding performances in structural annotations, pathology assessments, and semantic search across studies. OmicsNavigator offers a scalable, interpretable, and modality-agnostic solution for spatial omics analysis. ### Competing Interest Statement Some authors are affiliated with Enable Medicine as employees or former employees (N.V., A.T.M., A.E.T., Z.W.).

Hematoxylin and eosin (H&E) is a common and inexpensive histopathology assay. Though widely used and information-rich, it cannot directly inform about specific molecular markers, which require additional experiments to assess. To address this gap, we present ROSIE, a deep-learning framework that computationally imputes the expression and localization of dozens of proteins from H&E images. Our model is trained on a dataset of over 1300 paired and aligned H&E and multiplex immunofluorescence (mIF) samples from over a dozen tissues and disease conditions, spanning over 16 million cells. Validation of our in silico mIF staining method on held-out H&E samples demonstrates that the predicted biomarkers are effective in identifying cell phenotypes, particularly distinguishing lymphocytes such as B cells and T cells, which are not readily discernible with H&E staining alone. Additionally, ROSIE facilitates the robust identification of stromal and epithelial microenvironments and immune cell subtypes like tumor-infiltrating lymphocytes (TILs), which are important for understanding tumor-immune interactions and can help inform treatment strategies in cancer research.

List of tumor samples subjected to DNA and RNA sequencing along with sample quality metrics

List of OncoKB Cancer Genes with Overexpression and Downregulation

Comparison of Overexpressed and Downregulated Genes between RNA-Sequencing Cluster 1 and Cluster 3

BACKGROUND:While coronavirus disease 2019 (COVID-19) is primarily a respiratory infection, few studies have characterised the immune response to COVID-19 in lung tissue. We sought to understand the pathogenic role of microenvironmental interactions and the extracellular matrix in post-mortem COVID-19 lung using an integrative multi-omic approach. METHODS:Post-mortem formalin-fixed paraffin-embedded lung tissue from fatal COVID-19 and nonrespiratory death control lung underwent multi-omic evaluation by Quantseq Bulk RNA sequencing, Nanostring GeoMx spatial transcriptomics, RNAscope, multiplex immunofluorescence and immunohistochemistry, to evaluate virus distribution, immune composition and the extracellular matrix. Markers of extracellular synthesis and breakdown were measured in the serum of 215 patients with COVID-19 and 54 healthy volunteer controls using ELISA. RESULTS:We found that severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection was restricted to the pneumocytes and macrophages of early-stage disease. Spatial analyses revealed an immunosuppressive virus microenvironment, enriched for PDL1+IDO1+ macrophages and depleted of T-cells. Oligoclonal T-cells in COVID-19 lung showed no enrichment of SARS-CoV-2 specific T-cell receptors. Collagen VI was upregulated and contributed to alveolar wall thickening and impaired gas exchange in COVID-19 lung. Serum from COVID-19 patients showed increased levels of PRO-C6, a marker of collagen VI synthesis, predicted mortality in hospitalised patients. CONCLUSIONS:Our data refine the current model of respiratory COVID-19 with regard to virus distribution, immune niches and the role of the noncellular microenvironment in pathogenesis and risk stratification in COVID-19. We show that collagen deposition is an early event in the course of the disease.

The bone marrow microenvironment (BMME) is essential for hematopoiesis and immunity, yet spatiotemporal single-cell analysis during leukemogenesis remains challenging. We characterized the BMME in femurs from wild-type and chronic myeloid leukemia (CML) mice at 7, 14 and 21 days post-induction by highly multiplexed and 3D microscopy. Using a 54-marker CODEX panel, we profiled 2,033,725 cells in 55 tissue regions of interest and identified 41 cell types through unsupervised clustering and supervised annotation. During leukemic progression, we observed an expansion of myeloid and progenitor cell populations, increased PD-L1+ leukemic cells, the upregulation of PD-1 on CD4+ and CD8+ T cells, and a profound loss of B cells, plasma cells and bone cells. Advanced CML exhibited a striking expansion of immature, pericyte-deficient vasculature that disrupted vascular niches and impaired hematopoietic stem/progenitor cell positioning. Spatial mapping revealed leukemia-specific cellular neighborhoods enriched in PD-1+CD8+ T cells, suggesting localized immune cell exhaustion. Early-stage CML showed increaseds between plasmacytoid dendritic cells and megakaryocytes, whereas advanced CML featured heightened megakaryocyte emperipolesis of non-leukemic granulocytes. Megakaryocytes were morphologically irregular in CML mice and BM trephine biopsies from CML patients. Laser-capture microdissected megakaryocytes from newly diagnosed CML patients had reduced expression of cytoskeleton genes, which was reversed in advanced cases treated with tyrosine kinase inhibitors. 3D imaging revealed vascular disorganization and depleted megakaryocytes in the diaphysis, underscoring region-specific pathology. Together, this study provides a spatiotemporal single-cell atlas of the BMME during leukemic progression, showing how leukemic cells reprogram the niche to support their expansion and immune evasion.

Despite advances in immunotherapy treatment, nonresponse rates remain high, and mechanisms of resistance to checkpoint inhibition remain unclear. To address this gap, we performed spatial transcriptomic and proteomic profiling on human hepatocellular carcinoma tissues collected before and after immunotherapy. We developed an interpretable, multimodal deep learning framework to extract key cellular and molecular signatures from these data. Our graph neural network approach based on spatial proteomic inputs achieved outstanding performance (ROC-AUC > 0.9) in predicting patient treatment response. Key predictive features and associated spatial transcriptomic profiles revealed the multi-omic landscape of immunotherapy response and resistance. One such feature was an interface niche expressing restrictive extracellular matrix factors that physically separates tumor tissue and lymphoid aggregates in nonresponders. We integrate this and other spatially-resolved signatures into SPARC, a multi-omic "fingerprint" comprising scores for immunotherapy response and resistance mechanisms. This study lays groundwork for future patient stratification and treatment strategies in cancer immunotherapy.

PURPOSE:Germline pathogenic variants (PV) in ATM increase the risk of pancreatic ductal adenocarcinoma (PDAC), but the underlying tumor biology of PDAC associated with germline PV in ATM has not been adequately explored. EXPERIMENTAL DESIGN:Whole-genome, whole-exome, and RNA sequencing were performed on PDAC tumors from 25 germline ATM PV carriers diagnosed at Mayo Clinic between 2007 and 2017. Somatic and copy-number alterations, mutational signatures, transcriptomic subtypes, and the immune landscape were evaluated. RESULTS:High-quality whole-exome and whole-genome sequencing were obtained from 21 and 15 tumors, respectively. Biallelic inactivation of ATM was observed in 87%, KRAS PV in 90%, CDKN2A homozygous loss in 60%, and TP53 alterations in <10% of these tumors. A predominant clock-like mutational signature was present in all samples. Whole-transcriptome analysis identified that the aberrantly differentiated endocrine exocrine subtype accounted for 18% of PDAC and was consistently associated with >5-year overall survival. In addition, a 28-gene expression-based signature associated with overall survival was identified and further validated in The Cancer Genome Atlas cohort. Immune landscape analysis through CODEX identified enriched CD4 T-helper cell/tumor interactions and reduced B7H3-high cell/tumor interactions in ATM PV carriers compared with noncarriers. CONCLUSIONS:The observed absence of TP53 PV and enrichment for CDKN2A alterations in ATM tumors, along with differences in the mutational signatures, transcriptomic subtypes and immune landscape, improve our understanding of the mechanistic pathways involved in PDAC development in germline ATM PV carriers and help identify potential targeted therapeutic strategies.

Hematoxylin and eosin (H&E) staining has been a standard in clinical histopathology for many decades but lacks molecular detail. Advances in multiplexed spatial proteomics imaging allow cell types and tissues to be annotated by their expression patterns as well as their morphological features. However, these technologies are at present unavailable in most clinical settings. In this work, we present a machine learning framework that leverages histopathology foundation models and paired H&E and spatial proteomic imaging data to enable enhanced cell type annotation on H&E-only datasets. We trained and evaluated our method on kidney datasets with paired H&E and spatial proteomic imaging data and found that models trained using our methods outperform models trained directly on the imaging data. We also show how our framework can be used to study biological differences between two major kidney diseases.

Surgical removal of primary tumors reverses tumor-mediated immune suppression in pre-clinical models with metastatic disease. However, how cytoreductive surgery in the metastatic setting modulates the immune responses in patients, especially in the context of immune checkpoint therapy (ICT), is not understood. We report the first prospective, pilot, non-comparative clinical trial (NCT02210117) to evaluate the feasibility, clinical benefits, and immunologic changes of combining three different ICT-containing strategies with cytoreductive surgery or biopsy for patients with metastatic clear cell renal cell carcinoma. Primary safety endpoint of this trial has been met, with 43 patients completing cytoreductive surgery, 36 patients undergoing post-ICT biopsy, and 25 patients without either procedure due to progressive disease or toxicities or withdrawal of consent (total N = 104). Patients receiving ICT with cytoreductive surgery or biopsy, did not experience additional ICT- or procedure-related toxicities. The median overall survival was 54.7 months for patients who received ICT plus cytoreductive surgery. Immune-monitoring studies demonstrated that cytoreductive surgery increased antigen-presenting dendritic cell population and decreased KDM6B-expressing immune-suppressive myeloid cells in the peripheral blood. This study highlighted the feasibility of combining ICT with cytoreductive surgery in a metastatic setting and demonstrated the potential enhancement of immune responses following ICT plus cytoreductive surgery. Clinical evidence suggests that debulking surgery in patients with metastatic disease could enhance response to immune checkpoint therapy. Here the authors report the results of a clinical trial of immune checkpoint inhibitors plus debulking surgery in patients with metastatic renal cell carcinoma.

Background. Kidney allograft rejections are orchestrated by a variety of immune cells. Because of the complex histopathologic features, accurate pathological diagnosis poses challenges even for expert pathologists. The objective of this study was to unveil novel spatial indices associated with transplant rejection by using a spatial bioinformatic approach using 36-plex immunofluorescence image data. Methods. The image obtained from 11 T cell-mediated rejection (TCMR) and 12 antibody-mediated rejection (AMR) samples were segmented into 753 737 single cells using DeepCell’s Mesmer algorithm. These cells were categorized into 13 distinct cell types through unsupervised clustering based on their biomarker expression profiles. Cell neighborhood analysis allowed us to stratify kidney tissue into 8 distinct neighborhood components consisting of unique cell type enrichment profiles. Results. In contrast to TCMR samples, AMR samples exhibited a higher frequency of neighborhood components that were characterized by an enrichment of CD31 + endothelial cells. Although the overall frequency of CD68 + macrophages in AMR samples was not significantly high, CD68 + macrophages within endothelial cell-rich lesions exhibited a significantly higher frequency in AMR samples than TCMR samples. Furthermore, the frequency of interactions between CD31 + cells and CD68 + cells was significantly increased in AMR samples, implying the pivotal role of macrophages in AMR pathogenesis. Importantly, patients demonstrating a high frequency of CD31:CD68 interactions experienced significantly poorer outcomes in terms of chronic AMR progression. Conclusions. Collectively, these data indicate the potential of spatial bioinformatic as a valuable tool for aiding in pathological diagnosis and for uncovering new insights into the mechanisms underlying transplant rejection.

Heterogeneous resistance to immunotherapy remains a major challenge in cancer treatment, often leading to disease progression and death. Using CITE-seq and matched 40-plex PhenoCycler tissue imaging, we performed longitudinal multimodal single-cell analysis of tumors from metastatic melanoma patients with innate resistance, acquired resistance, or response to immunotherapy. We established the multimodal integration toolkit to align transcriptomic features, cellular epitopes, and spatial information to provide deeper insights into the tumors. With longitudinal analysis, we identified an "immune-striving" tumor microenvironment marked by peri-tumor lymphoid aggregates and low infiltration of T cells in the tumor and the emergence of MITF+SPARCL1+ and CENPF+ melanoma subclones after therapy. The enrichment of B cell-associated signatures in the molecular composition of lymphoid aggregates was associated with better survival. These findings provide further insights into the establishment of microenvironmental cell interactions and molecular composition of spatial structures that could inform therapeutic intervention.