Comparison of deep-learning data fusion strategies in mandibular osteoradionecrosis prediction modelling using clinical variables and radiation dose distribution volumes

Purpose. NTCP modelling is rapidly embracing DL methods as the need to include spatial dose information is acknowledged. Finding the most appropriate way of combining radiation dose distribution images and clinical data involves technical challenges and requires domain knowledge. We propose different data fusion strategies that we hope will serve as a starting point for future DL NTCP studies. Methods. Early, joint and late DL multi-modality fusion strategies were compared using clinical variables and mandibular radiation dose distribution volumes. The discriminative performance of the multi-modality models was compared to that of single-modality models. All the experiments were conducted on a control-case matched cohort of 92 ORN cases and 92 controls from a single institution. Results. The highest ROC AUC score was obtained with the late fusion model (0.70), but no statistically significant differences in discrimination performance were observed between strategies. While late fusion was the least technically complex strategy, its design did not model the inter-modality interactions that are required for NTCP modelling. Joint fusion involved the most complex design but resulted in a single network training process which included intra- and inter-modality interactions in its model parameter optimisation. Conclusions. This is the first study that compares different strategies for including image data into DL NTCP models in combination with lower dimensional data such as clinical variables. The discrimination performance of such multi-modality NTCP models and the choice of fusion strategy will depend on the distribution and quality of both types of data. We encourage future DL NTCP studies to report on different fusion strategies to better justify their choice of DL pipeline.