Weighted Sample Correlation Knowledge Distillation for Visual Recognition

Deploying deep neural networks with massive parameter counts on resource-constrained edge devices remains a significant challenge due to limited computational power, storage capacity, and energy efficiency. Knowledge distillation has emerged as a promising model compression technique, enabling the transfer of knowledge from a large teacher model to a compact student model. Traditional relation-based knowledge distillation methods focus on distilling inter-instance relational knowledge—typically, pairwise similarities between samples—but often overlook the asymmetry in each sample’s contribution to these relations. In reality, the relational information derived from a pair of samples may hold different importance for each sample within the feature space. To address this issue, we propose Weighted Sample Correlation Knowledge Distillation (WSCKD), a novel approach that explicitly models the asymmetric contributions of samples to relational knowledge. WSCKD leverages pairwise similarities in the source (teacher) embedding space as transferable knowledge and introduces two asymmetric loss functions: the unidirectional weighted distillation loss and the bidirectional weighted distillation loss. These losses enable the student model to prioritize more informative relationships during training, without imposing constraints on the manifold structure of the student’s embedding space. Extensive experiments and ablation studies on multiple visual image recognition benchmarks demonstrate that WSCKD outperforms sixteen distillation methods.