GRLT: Learning More from Teachers by Rethinking Knowledge Distillation from GNNs to MLPs

GNN-to-MLP (G2M) distillation has been proven effective in alleviating the inference latency of Graph Neural Networks (GNNs) by distilling graph representations and knowledge from GNNs to MLPs. However, the existing methods face two notable issues: (1) the graph representation module of the methods neglects the importance of nodes, resulting in student MLPs lacking the ability to focus on significant nodes; (2) they ignore that G2M distillation is a cross-architecture knowledge distillation involving a model gap between teacher and student models, leading to low knowledge transferability. To address the above issues, this paper proposes a novel method called Hybrid Graph Representation and Latent Space Translation (GRLT). The proposed GRLT captures richer graph information and enhances knowledge transferability, thereby improving the performance of the student model. Specifically, we incorporate graph prior knowledge into the hybrid graph representation module to obtain structural embeddings of the graphs and the aggregated features of nodes. In addition, we introduce a latent space translation feature distillation module, which transforms conventional feature map distillation into a two-stage process by first aligning the consensus representations between the teacher and the student via the Generalized Procrustes Analysis (GPA) and then performing feature distillation through cosine similarity. This design reduces the model gap and enhances knowledge transferability. GRLT has been experimentally verified on seven datasets and achieves superior overall performance. When SAGE is used as the teacher model, the distilled student MLP obtains average performance improvements of 1.72%, 3.72%, and 22.41% over the state-of-the-art method NOSMOG, the teacher GNN, and the vanilla MLP, respectively.