Mix-ViT: Mixing attentive vision transformer for ultra-fine-grained visual categorization

•We introduce Mix-ViT, a novel mixing attentive vision transformer, bridging the gap between vision transformers and ultra-fine-grained visual categorization (ultra-FGVC) tasks.•A self-supervised module is developed to mix the high-level sample tokens and predict attentively substituted tokens, which enables understanding of contextual details towards discriminative ultra-FGVC.•Mix-ViT achieves competitive performance on five ultra-fine-grained datasets and two fine-grained datasets, demonstrating its effectiveness and efficiency for the challenging ultra-FGVC tasks.