Multimodal Higher-Order Statistical Adapter for Video Action Recognition.

Recently, Contrastive Language-Image Pre-Training(CLIP) and Parameter-Efficient Fine-Tuning(PEFT) have attracted significant attention in the field of video action recognition. Nevertheless, effectively capturing both temporal dynamics and spatial local variations remains essential. Previous methods either concentrated on modeling long-term dependencies in video-level features while ignoring intra-frame spatial details, which frequently contain crucial discriminative information, or captured lower-order features in spatial details. In this paper, a multimodal higher-order statistical adapter (MHoS-Adapter) approach for CLIP is proposed, introducing a visual-branch Higher-Order Statistical Adapter (HS-Adapter) and a textual-branch Parallel Text Adapter (Para-Adapter). To drastically cut down on computational expense and time overhead, both employ adapter tuning, which involves freezing the initial two-branch structure and adjusting just a few parameters. Higher-order statistics of feature maps, including long-term and higher-order data, are modeled by HS-Adapter to improve local detail discrimination and capture long-term dependencies with minimal temporal complexity. Para-Adapter improves text-visual feature alignment by using Video Caption to produce video-level text descriptions for labels. Comprehensive tests on standard benchmarks like Kinetics-400, Mini Kinetics-200, UCF101, and HMDB51 validate the method’s effectiveness and show that MHoS-Adapter outperforms most full fine-tuning approaches, achieving 83.2