Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Study
ICLR(2021)
摘要
This work aims to empirically clarify a recently discovered perspective that label smoothing is incompatible with knowledge distillation. We begin by introducing the behind motivation on how this incompatibility is raised, i.e., label smoothing erases relative information between teacher logits. We provide a novel connection on how label smoothing affects distributions of semantically similar and dissimilar classes. Then we propose a metric to quantitatively measure the degree of erased information in sample's representation. After that, we study its one-sidedness and imperfection of the incompatibility view through massive analyses, visualizations and comprehensive experiments on Image Classification, Binary Networks, and Neural Machine Translation. Finally, we broadly discuss several circumstances wherein label smoothing will indeed lose its effectiveness.
更多查看译文
关键词
knowledge distillation,label smoothing truly incompatible,empirical study
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络