AutoEncoder-Driven Multimodal Collaborative Learning for Medical Image Synthesis

Multimodal medical images have been widely applied in various clinical diagnoses and treatments. Due to the practical restrictions, certain modalities may be hard to acquire, resulting in incomplete data. Existing methods attempt to generate the missing data with multiple available modalities. However, the modality differences in tissue contrast and lesion appearance become an obstacle to making a precise estimation. To address this issue, we propose an autoencoder-driven multimodal collaborative learning framework for medical image synthesis. The proposed approach takes an autoencoder to comprehensively supervise the synthesis network using the self-representation of target modality, which provides target-modality-specific prior to guide multimodal image fusion. Furthermore, we endow the autoencoder with adversarial learning capabilities by converting its encoder into a pixel-sensitive discriminator capable of both reconstruction and discrimination. To this end, the generative model is completely supervised by the autoencoder. Considering the efficiency of multimodal generation, we also introduce a modality mask vector as the target modality label to guide the synthesis direction, empowering our method to estimate any missing modality with a single model. Extensive experiments on multiple medical image datasets demonstrate the significant generalization capability as well as the superior synthetic quality of the proposed method, compared with other competing methods. The source code will be available: https://github.com/bcaosudo/AE-GAN .