Low-resource Security Patch Identification Based on Cumulative Learning and Knowledge Distillation

Security patch identification is an essential task in software maintenance, which aims to identify vulnerability-related patches from open-source software (OSS) repositories. Previous works tend to adopt pre-trained models to improve the model performance. These models are often built on billions of parameters and require sufficient computing resources and enough data for supervised fine-tuning, to be applied on downstream classification tasks. However, existing studies mainly address class imbalance through re-sampling or re-weighting strategies. Re-sampling may introduce synthetic noise or disrupt the original distribution of commit data, while fixed re-weighting schemes lack dynamic awareness of the model’s learning status, leading to over-smoothing, unstable optimization, and underfitting. Moreover, increasingly large pre-trained models require substantial computational resources for training and deployment, limiting their practical use in low-resource scenarios. In this work, we propose to boost security patch identification with cumulative learning and knowledge distillation, which aims to address security patch identification in low-resource scenarios while maintaining high-quality identification performance. To alleviate the data imbalance problem, inspired by cumulative learning, we designed a novel adaptive smoothing loss, which dynamically controls the smoothing coefficient a through the training epoch. In the early stage, a higher smoothing coefficient is settled to balance the influence of the majority class, while it is gradually reduced to ensure that the predictions of the minority classes are closely consistent with their true labels in the later stages. To bridge the gap between high-quality identification performance and low computational resources, based on the teacher-student network theory, we adopted the Kullback–Leibler divergence to minimize the difference between the teacher’s distribution and the student’s. The teacher model is further used to generate pseudo-labels for unlabeled data, thereby enhancing the performance of the student model. Experiment results on two open-source datasets demonstrate that our proposed approach not only improves the best baselines by 14.14% and 8.81% on minority class, but also reduces 68% model size. The code and data are shared in https://github.com/AppleMax1992/low-resource-commit-classification.