Coupled Matrix Factorization Constrained Deep Hyperspectral and Multispectral Image Fusion

The application of convolutional neural networks (CNNs) has yielded remarkable outcomes in the fusion of hyperspectral and multispectral images (HSI+MSI). However, most existing researches design black-box models for the direct reconstruction of high-resolution images from low-resolution images, which cannot theoretically guarantee the fusion mechanism throughout the network flow, thus limiting the restoration accuracy. This paper proposes a novel coupled matrix factorization (CMF) constrained deep interpretable network for HSI+MSI fusion, termed as CMF-FUSnet. Specifically, the iterative process of coupled matrix factorization is unfolded into a two-branch network interwoven with multiple denoiser modules and matrix factorization modules. In each iteration, the CMF-encoded two-branch sub-network alternately decomposes HSI and MSI to estimate their abundance and endmember matrices respectively. High-resolution HSI can be obtained by multiplying the endmembers extracted from the HSI and the abundances extracted from the MSI. The benefit is that the proposed CMF-FUSnet breaks through the black-box operation mode while adopting an end-to-end data-driven model, realizes the embedding of physical meaning, and improves the generalization of the model. Numerical experiments show that our proposed CMF-FUSnet compares favorably with both state-of-the-art model-driven and data-driven fusion methods in terms of visual analysis and quality assessment.