APT: Adaptive Prefix-Tuning on Pretrained Models for Code Intelligence

In the field of code intelligence, pretrained models exhibit impressive performance. However, it is imperative to modify all model parameters and maintain full copies for various tasks. Moreover, the effectiveness of fine-tuning a pretrained model depends on the availability of data, which can be constrained in practical settings. Prefix-tuning, a novel approach in NLP for addressing the aforementioned issue, has demonstrated promising results in numerous tasks. This paper aims to investigate the potential of prefix-tuning in the field of code intelligence, an area that has been relatively understudied. We also acknowledge the critical role of initialization in the performance of prefixtuning, which has been neglected in prior research. To address this gap, we introduce a novel method termed "adaptive prefixtuning". This approach entails training the prefix parameters on adapting tasks, followed by subsequent tuning on downstream tasks. We perform prefix-tuning and adaptive prefix-tuning using well-known pretrained models, CodeT5 and CodeGPT, and conduct experiments on four code intelligence tasks, namely, defect detection, code completion, code summarization, and code translation. Our experiments revealed that, in situations with restricted data availability, adaptive prefix-tuning yields substantial performance enhancements compared to fine-tuning, whereas prefix-tuning does not exhibit evident advantages. Furthermore, adaptive prefix-tuning has demonstrated superior performance relative to prefix-tuning in scenarios with comprehensive datasets. In certain cases, it has even outperformed traditional fine-tuning methods. Our findings indicate that within the field of code intelligence, adaptive prefix-tuning can serve as an effective substitute for fine-tuning, especially in situations with constrained data availability.