Transformer-Based Multimodal Fusion for Survival Prediction by Integrating Whole Slide Images, Clinical, and Genomic Data

Survival prediction using whole slide images (WSIs) is a complex and difficult task, as handling gigapixel WSI directly is computationally impossible. In the past few years, people have worked out multiple instance learning (MIL) strategies to deal with WSIs, i.e., splitting WSI into many patches (instances) and aggregating features across patches. Moreover, to better predict the survival outcome of patients, different modalities have been explored, among which gene features are used the most frequently. In this paper, we explore a graph-based strategy to handle WSIs and investigate a transformer-based strategy to combine different modalities for survival prediction. Moreover, clinical data was also adopted and different encoding manners of clinical information were explored. Experiments on two public datasets from The Cancer Genome Atlas (TCGA) demonstrate the effectiveness of the proposed graph-transformer framework for survival prediction.