Unleash and Integrate the Power of Pre-Trained ViTs Via Feature Fusion for Open-Vocabulary Object Detection

Alleviating over-fitting is one of the major concerns in open-vocabulary object detection (OVOD). Most OVOD methods rely on base data training and inherit the model structure from the close-set detectors. Final predictions are derived from the extracted features from backbones. Thus, backbone design plays a key role in improving the generalization capacity. However, existing works either choose fully optimizable network or a single frozen visual encoder as backbone, which limits the representation capacity of backbone features for OVOD and leads to sub-optimal performances. Therefore, we propose a novel multi-branch backbone network, named ViT-Feature-Modulated Multi-Scale Convolutional Network (VMCNet), which can effectively integrate and unleash the power of multiple pre-trained ViTs via the proposed feature fusion strategy. Drawing an analogy to the modulation mechanism in communication, we use an additional light-weight CNN branch to produce multi-scale carrier features, then modulates the representations from pre-trained ViTs to attain the final detection features. Our method not only leverages the information from base data but also utilizes the knowledge from multiple ViTs from CLIP and SAM, which ensembles the knowledge and generalization ability for OVOD setting. Equipped with the proposed backbone network, the detector could achieve better performance on novel categories. Evaluated on two popular benchmarks, our method boosts the detection performance on novel category and outperforms state-of-the-art methods. On OV-COCO, the proposed method achieves 47.5 APnovel 50 with ViT-B/16 and 52.8 APnovel50 with ViT-L/14. On OV-LVIS, VMCNet with ViT-B/16 reaches 27.7 mAPr.