PuzzleTuning: Explicitly Bridge Pathological and Natural Image with Puzzles
arXiv (Cornell University)(2023)
摘要
Pathological image analysis is a crucial field in computer vision. Due to the
annotation scarcity in the pathological field, recently, most of the works have
leveraged self-supervised learning (SSL) trained on unlabeled pathological
images, hoping to mine the representation effectively. However, there are two
core defects in current SSL-based pathological pre-training: (1) they do not
explicitly explore the essential focuses of the pathological field, and (2)
they do not effectively bridge with and thus take advantage of the knowledge
from natural images. To explicitly address them, we propose our large-scale
PuzzleTuning framework, containing the following innovations. Firstly, we
identify three task focuses that can effectively bridge knowledge of
pathological and natural domain: appearance consistency, spatial consistency,
and restoration understanding. Secondly, we devise a novel multiple puzzle
restoring task, which explicitly pre-trains the model regarding these focuses.
Thirdly, we introduce an explicit prompt-tuning process to incrementally
integrate the domain-specific knowledge, aligning the large domain gap between
natural and pathological images. Additionally, a curriculum-learning training
strategy is designed to regulate task difficulty, making the model adaptive to
the puzzle restoring complexity. Experimental results show that our
PuzzleTuning framework outperforms the previous state-of-the-art methods in
various downstream tasks on multiple datasets. The code, demo, and pre-trained
weights are available at https://github.com/sagizty/PuzzleTuning.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要