Morphing Tokens Draw Strong Masked Image Models
arxiv(2023)
摘要
Masked image modeling (MIM) is a promising option for training Vision
Transformers among various self-supervised learning (SSL) methods. The essence
of MIM lies in token-wise masked token predictions, with targets tokenized from
images or generated by pre-trained models such as vision-language models. While
tokenizers or pre-trained models are plausible MIM targets, they often offer
spatially inconsistent targets even for neighboring tokens, complicating models
to learn unified discriminative representations. Our pilot study confirms that
addressing spatial inconsistencies has the potential to enhance representation
quality. Motivated by the findings, we introduce a novel self-supervision
signal called Dynamic Token Morphing (DTM), which dynamically aggregates
contextually related tokens to yield contextualized targets. DTM is compatible
with various SSL frameworks; we showcase an improved MIM by employing DTM,
barely introducing extra training costs. Our experiments on ImageNet-1K and
ADE20K demonstrate the superiority of our methods compared with
state-of-the-art, complex MIM methods. Furthermore, the comparative evaluation
of the iNaturalists and fine-grained visual classification datasets further
validates the transferability of our method on various downstream tasks. Code
is available at https://github.com/naver-ai/dtm
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要