RightFeatKD: Selective Feature-Based Knowledge Distillation

Knowledge distillation (KD) has emerged as a widely adopted approach for cross-modal knowledge transfer, where transferring knowledge from a semantically rich teacher model to a lightweight student model enhances the performance of the student model. However, existing KD approaches rely on logit-based distillation, which suffers from limited logit diversity inherent in binary classification tasks such as fall detection. Moreover, existing KD methods transfer knowledge indiscriminately from both correct and incorrect predictions of the teacher model, which can undermine the reliability of the student model. We propose RightFeatKD, a novel feature-based distillation framework that captures rich semantic knowledge from the intermediate representations of the teacher model using a specialized loss function. Importantly, RightFeatKD focuses more on distilling knowledge from the teacher’s correct predictions, ensuring reliable and meaningful supervision. Experimental evaluations of RightFeatKD with two public fall detection datasets SmartFallMM and Up-Fall respectively demonstrate 4.04