Siamese infrared and visible light fusion network for RGB-T tracking

Due to the different photosensitive properties of infrared and visible light, infrared and visible light images have individual features. However, since the registered RGB-T image pairs shot in the same scene, they also contain common features. This paper proposes a Siamese infrared and visible light fusion Network (SiamIVFN) for RBG-T image-based tracking. SiamIVFN contains two main subnetworks: a complementary-feature-fusion network (CFFN) and a contribution-aggregation network (CAN). CFFN utilizes a two-stream multilayer convolutional structure that separately extracts individual features, and filters in each layer are partially coupled to extract common features. CFFN is a feature-level fusion network, which can cope with the misalignment of the RGB-T image pairs. Through adaptively calculating the contributions of infrared and visible light features obtained from CFFN, CAN makes the tracker robust under various light conditions. Experiments show that compared to state-of-the-art techniques, SiamIVFN improves the PR/SR score with 1.5%/8.8% on RGBT234 and 2.1%/6.9% on GTOT. The tracking speed of SiamIVFN is 147.6FPS, the current fastest RGB-T fusion tracker. The source codes are available at https://github.com/PengJingchao/SiamIVFN .