Low-Rank Knowledge Decomposition for Medical Foundation Models
CVPR 2024(2024)
摘要
The popularity of large-scale pre-training has promoted the development of
medical foundation models. However, some studies have shown that although
foundation models exhibit strong general feature extraction capabilities, their
performance on specific tasks is still inferior to task-specific methods. In
this paper, we explore a new perspective called “Knowledge Decomposition” to
improve the performance on specific medical tasks, which deconstruct the
foundation model into multiple lightweight expert models, each dedicated to a
particular task, with the goal of improving specialization while concurrently
mitigating resource expenditure. To accomplish the above objective, we design a
novel framework named Low-Rank Knowledge Decomposition (LoRKD), which
explicitly separates graidents by incorporating low-rank expert modules and the
efficient knowledge separation convolution. Extensive experimental results
demonstrate that the decomposed models perform well in terms of performance and
transferability, even surpassing the original foundation models.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要