A General Framework for Interactive Semantic Segmentation Refinement of Point Clouds

Although deep learning-based point cloud semantic segmentation has been extensively studied in the past decade, it is still challenging to produce high quality masks that meet high-precision downstream applications. This challenge stems from the distribution mismatch between training and testing data. The pre-trained segmentation networks optimized on the training dataset may perform sub-optimally on individual unseen testing data, resulting in performance drop. To this end, we propose a general interactive framework to enhance off-the-shelf networks. This framework integrates with off-the-shelf semantic segmentation networks in a fully test-time manner, allowing users to refine mis-segmented regions with a few corrective clicks. Specifically, we formulate a correction energy that treats user clicks as sparse training examples for test-time optimization. To mitigate catastrophic overfitting caused by sparse supervision, we formulate a stabilization energy that selectively minimizes the entropy of global points. Both the correction and stabilization energies constitute the test-time loss, promoting effective refinement of mis-segmented regions while maintaining the stability of others. Furthermore, a warm-up pre-process and an interaction simulation scheme are proposed for performance improvement and reproducible evaluation, respectively. We evaluate our framework on indoor and outdoor datasets with off-the-shelf networks, showing promising results in semantic segmentation refinement. The source code is available at https://github.com/Pengz98/ISSR.