Data Upcycling Knowledge Distillation for Image Super-Resolution
arxiv(2023)
摘要
Knowledge distillation (KD) compresses deep neural networks by transferring
task-related knowledge from cumbersome pre-trained teacher models to compact
student models. However, current KD methods for super-resolution (SR) networks
overlook the nature of SR task that the outputs of the teacher model are noisy
approximations to the ground-truth distribution of high-quality images (GT),
which shades the teacher model's knowledge to result in limited KD effects. To
utilize the teacher model beyond the GT upper-bound, we present the Data
Upcycling Knowledge Distillation (DUKD), to transfer the teacher model's
knowledge to the student model through the upcycled in-domain data derived from
training data. Besides, we impose label consistency regularization to KD for SR
by the paired invertible augmentations to improve the student model's
performance and robustness. Comprehensive experiments demonstrate that the DUKD
method significantly outperforms previous arts on several SR tasks.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要