Pyramid Graph Neural Network Knowledge Distillation with Pre-Trained Language Model for Medical Question Answering

The rapid growth of medical data and increasing complexity of clinical decisions highlight the need for advanced technologies in medical question answering. Pre-trained language models perform well in natural language processing but struggle with specialized medical datasets due to insufficient domain-specific knowledge, resulting in hallucinations and factual errors. This paper proposes Pyramid Graph Neural Network Knowledge Distillation (PyGNN-KD), a framework that distills knowledge from a fine-tuned pre-trained language model to enhance a Pyramid Graph Neural Network (PyGNN) for medical question answering. PyGNN-KD builds a joint knowledge subgraph centered on context nodes for efficient multi-hop reasoning, uses task-specific fine-tuning for domain-aligned node features, and integrates shallow and deep layer features via a pyramid network with adaptive gating to address over-smoothing. An annealing distillation strategy optimizes graph neural network learning by assimilating probability distributions from large language models, improving medical semantic understanding. These innovations advance artificial intelligence through knowledge distillation and adaptive features, with applications in clinical decision support. Evaluations on five medical question answering datasets and five language models show a 3.01% overall improvement. Notably, PyGNN-KD achieves 6.27% average gain on the medical subset of the Massive Multitask Language Understanding (MMLU) dataset and 2.74% on the Medical Question Answering for the United States Medical Licensing Examination (MedQA-USMLE) dataset, aiding accurate disease diagnosis and treatment planning.