Depth-assisted RefineNet for Indoor Semantic Segmentation

This paper focuses on indoor semantic segmentation using RGB-D data. It has been shown that incorporating depth information into RGB information is helpful to improve segmentation accuracy. However, previous studies have revealed two problems. One is about the model size. Recent state-of-the-art methods generally build a network branch for depth images, inherently increasing the model size. The other is about boundary segmentation. The complex and various object configurations with severe occlusions influence the segmentation precision of object boundaries. To address these two problems, we propose a depth-assisted RefineNet (D-RefineNet) for refining the boundary segmentation. The proposed network only uses RGB images to predict segmentation results. Depth images are just used in the proposed loss function without increasing the model size. When the depth values of adjacent pixels change drastically but the adjacent pixels have the same predicted semantic labels, the proposed loss function penalizes the predicted result. Experimental evaluations demonstrate that the proposed method is effective on two challenging RGB-D indoor datasets, NYUDv2 and SUN RGB-D.