MRI pulse sequence integration for deep-learning-based brain metastases segmentation

Purpose Magnetic resonance (MR) imaging is an essential diagnostic tool in clinical medicine. Recently, a variety of deep-learning methods have been applied to segmentation tasks in medical images, with promising results for computer-aided diagnosis. For MR images, effectively integrating different pulse sequences is important to optimize performance. However, the best way to integrate different pulse sequences remains unclear. In addition, networks trained with a certain subset of pulse sequences as input are unable to perform when given a subset of those pulse sequences. In this study, we evaluate multiple architectural features and characterize their effects in the task of metastasis segmentation while creating a method to robustly train a network to be able to work given any strict subset of the pulse sequences available during training. Methods We use a 2.5D DeepLabv3 segmentation network to segment metastases lesions on brain MR's with four pulse sequence inputs. To study how we can best integrate MR pulse sequences for this task, we consider (1) different pulse sequence integration schemas, combining our features at early, middle, and late points within a deep network, (2) different modes of weight sharing for parallel network branches, and (3) a novel integration level dropout layer, which will allow the networks to be robust to performing inference on input with only a subset of pulse sequences available at the training. Results We find that levels of integration and modes of weight sharing that favor low variance work best in our regime of small amounts of training data (n = 100). By adding an input-level dropout layer, we could preserve the overall performance of these networks while allowing for inference on inputs with missing pulse sequences. We illustrate not only the generalizability of the network but also the utility of this robustness when applying the trained model to data from a different center, which does not use the same pulse sequences. Finally, we apply network visualization methods to better understand which input features are most important for network performance. Conclusions Together, these results provide a framework for building networks with enhanced robustness to missing data while maintaining comparable performance in medical imaging applications.