Three-Dimensional Offset Vector for Panoptic Segmentation

Panoptic segmentation has become a research hotspot in the field of computer vision. In this paper, we propose a novel and efficient method for panoptic segmentation. Based on semantic segmentation architecture DeepLabv3+, we predict a three-dimensional offset vector for each pixel, and produce a unified panoptic segmentation mask by combining semantic prediction. Previous bottom-up panoptic segmentation methods only consider offset vector in a horizontal plane, i.e. a two-dimensional space. In contrast, we consider offset vector in both horizontal plane and vertical direction, and predict an offset vector in three-dimensional space for each pixel. Offset vector in vertical direction is used to determine whether a pixel belongs to a foreground object, while offset vector in horizontal plane is used to determine which foreground object the foreground pixel belongs to. In this way, we can predict offset vectors for both background and foreground object pixels at the same time, and effectively model foreground pixels clustering process. During inference, we can produce a unified panoptic segmentation mask by combining the foreground pixels clustering result and semantic prediction. We demonstrate our method’s effectiveness on Cityscapes dataset, and obtain competitive results.