BF3D: Bi-directional fusion 3D detector with semantic sampling and geometric mapping

3D object detection is a key task in environmental awareness, which plays a vital role in autonomous driving safety. Lidars and cameras are widely used sensors that provide complementary information for accurate 3D detection. However, due to the domain difference between the two modalities, it is challenging to leverage their respective strengths and fuse them perfectly. In this paper, an end-to-end detector termed BF3D is proposed, which integrates with the semantic sampling module, geometric point-image mapping module, and bi-directional attention fusion module. Specifically, the semantic sampling module incorporates a novel downsampling strategy to preserve more foreground points and pixels. Additionally, the geometric point-image mapping module is developed to find geometric correlation pixels of the point and take advantage of the high density of image features. We also introduce a bi-directional attention fusion module to combine useful information from the two modalities by attention mechanism. Extensive experiments demonstrate that BF3D outperforms both single- and multi-modal 3D detectors. Codes are available at: https://github.com/hustzyj/BF3D.