A real-time indoor scene analysis method based on RGBD stream

Indoor scene analysis is important for some applications, such as augmented reality. Specifically, we confine the indoor scene analysis problem to two aspects: reconstructing the geometry and understanding the observed element. Traditional analysis methods focusing on 2D RGB images are limited because of the lack of stereo measurements, and thus RGBD-based indoor scene analysis has received wide attention. However, geometric analysis and semantic analysis have been treated as two separate problems in most recent works, leaving scene analysis incomplete. In our work, we combine a deep network architecture with a 3D online reconstruction algorithm and propose a complete pipeline to simultaneously analyse the indoor scene from the geometric level and the semantic level. We take a live depth camera as input and consider the scene analysis as two steps. The first step estimates the camera pose and labels scene objects for a single view. The second step fuses the scene objects into an integrated map for a global view. Specifically, we first transfer the input frame to geometric maps and propose a structural constraint iterative closest point (SC-ICP) algorithm for camera tracking. Then, we propose a structural constraint recurrent neural network (SC-RNN) to generate a semantic map for each frame. Finally, the geometric maps and semantic maps from multiple viewpoints can be fused into a complete model according to the camera pose. Our method can improve the accuracy of camera pose estimation and significantly analyse 3D indoor scenes consisting of high-quality geometric details and rich semantic information. It is noteworthy that our method can meet real-time requirements with frame rates of approximate to 25 Hz.