Automatic registration of point cloud and panoramic images in urban scenes based on pole matching

Given the initial calibration of multiple sensors, the fine registration between Mobile Laser Scanning (MLS) point clouds and panoramic images is still challenging due to the unforeseen movement and temporal misalignment while collecting. To tackle this issue, we proposed a novel automatic method to register the panoramic images and MLS point clouds based on the matching of pole objects. Firstly, 2D pole instances in the panoramic images are extracted by a semantic segmentation network and then optimized. Secondly, every corresponding frustum point cloud of each pole instance is obtained by a shape-adaptive buffer region in the panoramic image, and the 3D pole object is extracted via a combination of slicing, clustering, and connected domain analysis, then all 3D pole objects are fused. Finally, 2D and 3D pole objects are re-projected onto virtual images respectively, and then fine 2D-3D correspondences are collected through maximizing pole overlapping area by Particle Swarm Optimization (PSO). The accurate extrinsic orientation parameters are acquired by the Efficient Perspective-N-Point (EPnP). The experiments indicate that the proposed method performs effectively on two challenging urban scenes with an average registration error of 2.01 pixels (with RMSE 0.88) and 2.35 pixels (with RMSE 1.03), respectively.