Aligning the real and the virtual world: Mixed reality localisation using learning-based 3D-3D model registration

Existing camera localisation methods for indoor augmented and mixed reality (AR/MR) are almost exclusively image based. The main issue with image-based methods is that they do not scale well and, as a consequence, AR/MR applications are mostly limited to small-scale room experiences. To tackle the challenge of large-scale indoor AR/MR localisation, we propose a novel framework for AR/MR localisation based solely on 3D-3D model registration. The localisation is performed by an automated registration of a low-density model of the surroundings created by the device to the existing point cloud of the environment based on learning -based keypoint detection and description. Our solution takes advantage of recent significant improvements in automated coarse-to-fine 3D-3D model registration methods. Unlike the existing image-based AR/MR localisation methods, which are restricted to small room-sized environments, the proposed 3D registration -based approach is applicable to large environments and is robust to changes in colour and illumination of the scene. We perform extensive testing and analysis of the approach with real-world experiments and datasets using a prototype developed for the Microsoft HoloLens. Experimental results show high localisation reliability and accuracy, with a mean translation error of 2.8 cm and a mean rotation error of 0.30 degrees. The method performs well in a large-scale environment (300 m2) and shows good robustness to changes in scene geometry.