When Self-supervised Transformers Meet Knowledge Distillation: Efficient Chest X-Ray Classification.

Chest X-ray classification is limited by scarce annotations and the heavy cost of transformer models. We propose a three-stage framework: (1) self-supervised pretraining of a Vision Transformer (ViT) with DINOv2 on 880k unlabeled radiographs, (2) fine-tuning on ChestX-ray14, and (3) knowledge transfer into MobileViT using a combination of Binary Cross-Entropy (BCE) and Multi-Label Distillation (MLD) loss. The distilled MobileViT achieves a mean AUROC of 0.8404, surpassing its supervised counterpart by 1.9