A Multimodal Transfer Learning Method for Classifying Images of Celestial Point Sources

A large fraction of celestial objects exhibit point shapes in CCD images, such as stars and QSOs, which contain less information due to their few pixels. Point source classification based solely on image data may lead to low accuracy. To address this challenge, this paper proposes a Multi-modal Transfer Learning-based classification method for celestial objects with point shape images. Considering that spectral data possess rich features and that there is a correlation between spectral data and image data, the proposed approach fully utilizes the knowledge gained from celestial spectral data and transfers it to the original image-based classification, enhancing the accuracy of classifying stars and QSOs. Initially, a one-dimensional residual network is employed to extract a 128-dimensional spectral feature vector from the original 3700-dimensional spectral data. This spectral feature vector captures important features of the celestial object. The Generative Adversarial Network is then utilized to generate a simulated spectral vector of 128 dimensions, which corresponds to the celestial object image. By generating simulated spectral vectors, data from two modals (spectral and image) for the same celestial object are available, enriching the input features of the model. In the upcoming multimodal classification model, we only require the images of celestial objects along with their corresponding simulated spectral data, and we no longer need real spectral data. With the assistance of spectral data, the proposed method alleviates the above disadvantages of the original image-based classification method. Remarkably, our method has improved the F1-score from 0.93 to 0.9777, while reducing the error rate in classification by 40%. These enhancements significantly increase the classification accuracy of stars and QSOs, providing strong support for the classification of celestial point sources.