Taking Lessons from History: Memory Augmented Self Distillation for Graph Neural Networks

In the realm of knowledge distillation (KD), a method widely acknowledged for compressing and regularizing models, there has been growing interest in its application to Graph Neural Networks (GNNs), which often suffer from the overfitting issue, significantly limiting their generalizability. Recent KD techniques aim to alleviate this overfitting by enhancing GNN model capabilities. However, conventional KD approaches face challenges in effectively leveraging well-trained GNN teachers for student training, notably due to the oversmoothing phenomenon in GNNs, and conventional self-distillation techniques suffer from the information bottleneck problem. To address the overfitting issue in GNNs, this study introduces a novel memory-augmented self-learning framework that extracts and provides diverse learning sources for adaptive knowledge distillation from the student model itself. A competency-based knowledge source selection mechanism dynamically determines the most pertinent source. Unlike existing self-distillation approaches, our method demonstrates improved learning with enhanced regularization, resulting in a 2.5-6