Photo-SLAM: Real-time Simultaneous Localization and Photorealistic Mapping for Monocular, Stereo, and RGB-D Cameras

The integration of neural rendering and the SLAM system recently showedpromising results in joint localization and photorealistic view reconstruction.However, existing methods, fully relying on implicit representations, are soresource-hungry that they cannot run on portable devices, which deviates fromthe original intention of SLAM. In this paper, we present Photo-SLAM, a novelSLAM framework with a hyper primitives map. Specifically, we simultaneouslyexploit explicit geometric features for localization and learn implicitphotometric features to represent the texture information of the observedenvironment. In addition to actively densifying hyper primitives based ongeometric features, we further introduce a Gaussian-Pyramid-based trainingmethod to progressively learn multi-level features, enhancing photorealisticmapping performance. The extensive experiments with monocular, stereo, andRGB-D datasets prove that our proposed system Photo-SLAM significantlyoutperforms current state-of-the-art SLAM systems for online photorealisticmapping, e.g., PSNR is 30faster in the Replica dataset. Moreover, the Photo-SLAM can run at real-timespeed using an embedded platform such as Jetson AGX Orin, showing the potentialof robotics applications.