Self-supervised learning has achieved remarkable performance in computer vision, utilizing two key paradigms: contrastive learning and masked image modeling. Contrastive learning focuses on global representations by learning similarities and dissimilarities from different views of the inputs. On the other hand, masked image modeling learns from a pixel-level reconstruction objective and has shown improved performance compared to contrastive learning. However, masked image modeling lacks global semantics due to its pixel-level objective. To this end, we propose MOMA, a novel self-supervised distillation framework that employs a contrastive learning teacher to enhance the global representation of the masked image modeling student. Specifically, the teacher provides masks for the student, encouraging reconstructions that favor better global semantics. The feature alignment between the teacher and the student further enhances the global features in masked image modeling. Experimental results demonstrate that the proposed MOMA outperforms other masked image modeling methods and achieves competitive performance compared to other self-supervised baselines.
原文链接
分享
引用
