Effective Visual Token Aggregation for ViT Classification

Vision Transformer (ViT) has garnered significant research attention and demonstrated remarkable success in visual classification tasks. However, most existing improvements primarily focus on the Self-Attention and Feed-Forward Network modules, while comparatively little emphasis has been placed on the classification head, with limited exploration of the potential of visual tokens for ViT classification. In this paper, we propose Multi-headed Global Cross-Covariance Pooling (MGCrP), an effective second-order visual token aggregation method, to construct a powerful classification head for ViTs. Specifically, motivated by idea of Self-Attention in ViT, our MGCrP computes a group of cross-covariances of visual tokens in a multi-headed manner, which efficiently captures channel correlation among visual tokens and offers a powerful global representation. Furthermore, we present Singular Value Power Normalization (svPN) and its fast variant, which enable us to construct an effective linear classifier based on our MGCrP by considering the geometric structure of cross-covariance representations. Particularly, our MGCrP with svPN can be flexibly integrated into existing Vision Transformer architectures, and we construct a family of ViT models, called Second-order Vision Transformer (SoViT). To evaluate our methods, we conduct experiments under two paradigms: training From scratch on ImageNet-1K and pre-training on large-scale datasets followed by tuning on downstream tasks. Moreover, the proposed MGCrP is modality-agnostic and can be applied both vision tasks and NLP tasks. Extensive experimental results on various classification tasks demonstrate that the proposed SoViT consistently outperforms its counterparts, achieving very competitive performance.