PRISM: A Multi-Modal Generative Foundation Model for Slide-Level Histopathology
CoRR(2024)
Abstract
Foundation models in computational pathology promise to unlock the
development of new clinical decision support systems and models for precision
medicine. However, there is a mismatch between most clinical analysis, which is
defined at the level of one or more whole slide images, and foundation models
to date, which process the thousands of image tiles contained in a whole slide
image separately. The requirement to train a network to aggregate information
across a large number of tiles in multiple whole slide images limits these
models' impact. In this work, we present a slide-level foundation model for
H E-stained histopathology, PRISM, that builds on Virchow tile embeddings and
leverages clinical report text for pre-training. Using the tile embeddings,
PRISM produces slide-level embeddings with the ability to generate clinical
reports, resulting in several modes of use. Using text prompts, PRISM achieves
zero-shot cancer detection and sub-typing performance approaching and
surpassing that of a supervised aggregator model. Using the slide embeddings
with linear classifiers, PRISM surpasses supervised aggregator models.
Furthermore, we demonstrate that fine-tuning of the PRISM slide encoder yields
label-efficient training for biomarker prediction, a task that typically
suffers from low availability of training data; an aggregator initialized with
PRISM and trained on as little as 10
supervised baseline that uses all of the data.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined