VRP-SAM: SAM with Visual Reference Prompt

In this paper, we propose a novel Visual Reference Prompt (VRP) encoder thatempowers the Segment Anything Model (SAM) to utilize annotated reference imagesas prompts for segmentation, creating the VRP-SAM model. In essence, VRP-SAMcan utilize annotated reference images to comprehend specific objects andperform segmentation of specific objects in target image. It is note that theVRP encoder can support a variety of annotation formats for reference images,including point, box, scribble, and mask.VRP-SAM achieves a breakthrough within the SAM framework by extending itsversatility and applicability while preserving SAM's inherent strengths, thusenhancing user-friendliness. To enhance the generalization ability of VRP-SAM,the VRP encoder adopts a meta-learning strategy. To validate the effectivenessof VRP-SAM, we conducted extensive empirical studies on the Pascal and COCOdatasets. Remarkably, VRP-SAM achieved state-of-the-art performance in visualreference segmentation with minimal learnable parameters. Furthermore, VRP-SAMdemonstrates strong generalization capabilities, allowing it to performsegmentation of unseen objects and enabling cross-domain segmentation.