Multi-criteria Token Fusion with One-step-ahead Attention for Efficient Vision Transformers
CVPR 2024(2024)
摘要
Vision Transformer (ViT) has emerged as a prominent backbone for computer
vision. For more efficient ViTs, recent works lessen the quadratic cost of the
self-attention layer by pruning or fusing the redundant tokens. However, these
works faced the speed-accuracy trade-off caused by the loss of information.
Here, we argue that token fusion needs to consider diverse relations between
tokens to minimize information loss. In this paper, we propose a Multi-criteria
Token Fusion (MCTF), that gradually fuses the tokens based on multi-criteria
(e.g., similarity, informativeness, and size of fused tokens). Further, we
utilize the one-step-ahead attention, which is the improved approach to capture
the informativeness of the tokens. By training the model equipped with MCTF
using a token reduction consistency, we achieve the best speed-accuracy
trade-off in the image classification (ImageNet1K). Experimental results prove
that MCTF consistently surpasses the previous reduction methods with and
without training. Specifically, DeiT-T and DeiT-S with MCTF reduce FLOPs by
about 44
model, respectively. We also demonstrate the applicability of MCTF in various
Vision Transformers (e.g., T2T-ViT, LV-ViT), achieving at least 31
without performance degradation. Code is available at
https://github.com/mlvlab/MCTF.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要