Zero-Shot Medical Phrase Grounding with Off-the-shelf Diffusion Models
arxiv(2024)
摘要
Localizing the exact pathological regions in a given medical scan is an
important imaging problem that requires a large amount of bounding box ground
truth annotations to be accurately solved. However, there exist alternative,
potentially weaker, forms of supervision, such as accompanying free-text
reports, which are readily available. The task of performing localization with
textual guidance is commonly referred to as phrase grounding. In this work, we
use a publicly available Foundation Model, namely the Latent Diffusion Model,
to solve this challenging task. This choice is supported by the fact that the
Latent Diffusion Model, despite being generative in nature, contains mechanisms
(cross-attention) that implicitly align visual and textual features, thus
leading to intermediate representations that are suitable for the task at hand.
In addition, we aim to perform this task in a zero-shot manner, i.e., without
any further training on target data, meaning that the model's weights remain
frozen. To this end, we devise strategies to select features and also refine
them via post-processing without extra learnable parameters. We compare our
proposed method with state-of-the-art approaches which explicitly enforce
image-text alignment in a joint embedding space via contrastive learning.
Results on a popular chest X-ray benchmark indicate that our method is
competitive wih SOTA on different types of pathology, and even outperforms them
on average in terms of two metrics (mean IoU and AUC-ROC). Source code will be
released upon acceptance.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要