Test-Time Adaptation with SaLIP: A Cascade of SAM and CLIP for Zero shot Medical Image Segmentation
arxiv(2024)
摘要
The Segment Anything Model (SAM) and CLIP are remarkable vision foundation
models (VFMs). SAM, a prompt driven segmentation model, excels in segmentation
tasks across diverse domains, while CLIP is renowned for its zero shot
recognition capabilities. However, their unified potential has not yet been
explored in medical image segmentation. To adapt SAM to medical imaging,
existing methods primarily rely on tuning strategies that require extensive
data or prior prompts tailored to the specific task, making it particularly
challenging when only a limited number of data samples are available. This work
presents an in depth exploration of integrating SAM and CLIP into a unified
framework for medical image segmentation. Specifically, we propose a simple
unified framework, SaLIP, for organ segmentation. Initially, SAM is used for
part based segmentation within the image, followed by CLIP to retrieve the mask
corresponding to the region of interest (ROI) from the pool of SAM generated
masks. Finally, SAM is prompted by the retrieved ROI to segment a specific
organ. Thus, SaLIP is training and fine tuning free and does not rely on domain
expertise or labeled data for prompt engineering. Our method shows substantial
enhancements in zero shot segmentation, showcasing notable improvements in DICE
scores across diverse segmentation tasks like brain (63.46
and fetal head (30.82
prompts are available at: https://github.com/aleemsidra/SaLIP.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要