Spectral reconstruction of fundus images using retinex-based semantic spectral separation transformer, applied for retinal oximetry

Multispectral imaging, a non-invasive technique to measure retinal oxygen saturation levels, is limited due to its time-consuming and expensive nature. In this paper, we present R3ST (Retinex-based Semantic Spectral Separation Transformer), an innovative end-to-end method for RGB fundus image spectral reconstruction. Our proposed model leverages RGB images to reconstruct multispectral images, making retinal oximetry more accessible for diagnostics.Existing spectral reconstruction methods face challenges in achieving high accuracy and generalizing across camera sources, affecting diagnostic results. R3ST addresses these issues through an unsupervised semantic spectral separation module that strives to encourage the separate reconstruction of pixels with different semantics as much as possible. Additionally, we introduce an oxygen saturation loss to improve reconstruction accuracy within the relevant regions. To overcome the limitations of generalization properties, our model incorporates a self-supervised Retinex reflectance extraction module. This simulates various imaging systems and extracts reflectance information from fundus images for reconstruction.Extensive experiments demonstrate R3ST’s superior performance in spectral reconstruction accuracy, making retinal oximetry a more viable, cost-effective diagnostic option that provides essential eye health and systemic health information. The code of R3ST is available at https://github.com/telesc0pe/R3ST.