ACC-ViT : Atrous Convolution's Comeback in Vision Transformers
arxiv(2024)
摘要
Transformers have elevated to the state-of-the-art vision architectures
through innovations in attention mechanism inspired from visual perception. At
present two classes of attentions prevail in vision transformers, regional and
sparse attention. The former bounds the pixel interactions within a region; the
latter spreads them across sparse grids. The opposing natures of them have
resulted in a dilemma between either preserving hierarchical relation or
attaining a global context. In this work, taking inspiration from atrous
convolution, we introduce Atrous Attention, a fusion of regional and sparse
attention, which can adaptively consolidate both local and global information,
while maintaining hierarchical relations. As a further tribute to atrous
convolution, we redesign the ubiquitous inverted residual convolution blocks
with atrous convolution. Finally, we propose a generalized, hybrid vision
transformer backbone, named ACC-ViT, following conventional practices for
standard vision tasks. Our tiny version model achieves ∼ 84 % accuracy on
ImageNet-1K, with less than 28.5 million parameters, which is 0.42%
improvement over state-of-the-art MaxViT while having 8.4% less parameters.
In addition, we have investigated the efficacy of ACC-ViT backbone under
different evaluation settings, such as finetuning, linear probing, and
zero-shot learning on tasks involving medical image analysis, object detection,
and language-image contrastive learning. ACC-ViT is therefore a strong vision
backbone, which is also competitive in mobile-scale versions, ideal for niche
applications with small datasets.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要