Knowledge Distillation $\approx$ Label Smoothing: Fact or Fallacy?

arxiv(2023)

引用 0|浏览11
暂无评分
摘要
Contrary to its original interpretation as a facilitator of knowledge transfer from one model to another, some recent studies have suggested that knowledge distillation (KD) is instead a form of regularization. Perhaps the strongest support of all for this claim is found in its apparent similarities with label smoothing (LS). This paper investigates the stated equivalence of these two methods by examining the predictive uncertainties of the models they train. Experiments on four text classification tasks involving teachers and students of different capacities show that: (a) In most settings, KD and LS drive model uncertainty (entropy) in completely opposite directions, and (b) In KD, the student's predictive uncertainty is a direct function of that of its teacher, reinforcing the knowledge transfer view.
更多
查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络