Diverse Hyperspectral Remote Sensing Image Synthesis With Diffusion Models

Hyperspectral image (HSI) synthesis overcomes the limitations of imaging sensors and enables low-cost acquisition of HSIs with high spatial resolution. Using RGB as a conditional input for hyperspectral generation is promising and valuable, as it can leverage abundant existing multispectral/RGB images without the intervention of hyperspectral sensors. However, most existing generation methods follow one-to-one mapping frameworks and ignore generation diversity. In addition, the current evaluation metrics of hyperspectral generation are based on the similarity with the reference image, which cannot reflect the diversity of the generated spectra. In this article, we propose a novel method for diverse hyperspectral remote sensing image generation based on the diffusion model. The diffusion model uses a denoising model to gradually remove noise from the normal distribution and generates the hyperspectral data step-by-step with the conditional RGB image as input. To address the high-dimensional noise prediction problem caused by a large number of bands in the HSI, we introduce a conditional vector quantized generative adversarial network (VQGAN) that maps the high-dimensional hyperspectral data into a low-dimensional latent space and conduct the diffusion process in the latent space. The latent-diffusion process makes the diffusion process faster and more stable. The conditional VQGAN decodes HSIs from the latent code generated by diffusion, with the conditional RGB image as the input, which restricts the diversity to a specific object distribution. We also designed two new metrics to evaluate the generation spectral diversity (SD). Experiments on the IEEE grss_dfc_2018 dataset demonstrate that our method can synthesize highly diverse hyperspectral data. In addition, the rationality of the proposed metrics is also verified.