Tackling Structural Hallucination in Image Translation with Local Diffusion
arxiv(2024)
摘要
Recent developments in diffusion models have advanced conditioned image
generation, yet they struggle with reconstructing out-of-distribution (OOD)
images, such as unseen tumors in medical images, causing “image
hallucination” and risking misdiagnosis. We hypothesize such hallucinations
result from local OOD regions in the conditional images. We verify that
partitioning the OOD region and conducting separate image generations
alleviates hallucinations in several applications. From this, we propose a
training-free diffusion framework that reduces hallucination with multiple
Local Diffusion processes. Our approach involves OOD estimation followed by two
modules: a “branching” module generates locally both within and outside OOD
regions, and a “fusion” module integrates these predictions into one. Our
evaluation shows our method mitigates hallucination over baseline models
quantitatively and qualitatively, reducing misdiagnosis by 40
real-world medical and natural image datasets, respectively. It also
demonstrates compatibility with various pre-trained diffusion models.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要