Test Input Prioritization for Image Segmentation: an Empirical Study

Image segmentation is critical in domains such as medical image analysis, autonomous driving, and remote sensing. While deep learning-based image segmentation systems have achieved remarkable performance, ensuring the reliability of such systems remains challenging due to the high cost of manual labeling effort, where labeling refers to annotating test inputs with pixel-level ground truth masks to determine whether model predictions are correct, which is prohibitively expensive and time-consuming. Test input prioritization offers a promising solution by identifying potentially misclassified inputs for early inspection. However, unlike in traditional deep learning, test prioritization in image segmentation presents unique challenges: 1) Image segmentation models output images rather than a single probability vector, making confidence-based uncertainty estimation unsuitable. 2) Evaluating prioritization effectiveness is more complex, as it relies on IoU, and whether a test is considered misclassified depends on an adjustable threshold, increasing assessment uncertainty. To address these challenges, we adapt complexity-based and learning-based prioritization approaches to image segmentation and conduct a comprehensive empirical study. We evaluate 26 methods across nine models and three datasets, analyzing factors such as IoU thresholds, embedding models, feature fusion techniques, and key parameters. Our study finds that: 1) Complexity-based methods outperform the baseline in 72.8 ∼ 52.68