On the Effectiveness of Large Language Models in Domain-Specific Code Generation
arxiv(2023)
摘要
Large language models (LLMs) such as ChatGPT have shown remarkable
capabilities in code generation. Despite the great achievement, they rely on
enormous training data to acquire a broad spectrum of open-domain knowledge.
Besides, their evaluation revolves around open-domain benchmarks like
HumanEval, which primarily consist of programming contests. Therefore, it is
hard to fully characterize the intricacies and challenges associated with
particular domains (e.g., web, game, and math). In this paper, we conduct an
in-depth study of the LLMs in domain-specific code generation. Our results
demonstrate that LLMs exhibit sub-optimal performance in generating
domain-specific code, due to their limited proficiency in utilizing
domain-specific libraries. We further observe that incorporating API knowledge
as prompts can empower LLMs to generate more professional code. Based on these
findings, we further investigate how to efficiently incorporate API knowledge
into the code generation process. We experiment with three strategies for
incorporating domain knowledge, namely, external knowledge inquirer,
chain-of-thought prompting, and chain-of-thought fine-tuning. We refer to these
strategies as a new code generation approach called DomCoder. Experimental
results show that all strategies of DomCoder lead to improvement in the
effectiveness of domain-specific code generation under certain settings. The
results also show that there is still ample room for further improvement, based
on which we suggest possible future works.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要