Knowledge-Aware Code Generation with Large Language Models
CoRR(2024)
摘要
Large Language Models (LLMs) perform well on basic programming problems.
However, they encounter challenges when dealing with complex tasks involving
the use of diverse algorithmic and data structure skills, particularly
programming competition-level problems. Notably, ChatGPT exhibits proficient
performance on problems it has encountered during its pre-training phase, but
this performance deteriorates when faced with novel problems. Consequently,
enhancing the ability of LLMs to address unfamiliar problems has emerged as a
pivotal research focus. The problem-solving process of LLMs mirrors human
programmers' approach to a certain extent. When confronted with new programming
tasks, human programmers engage in task planning and code writing with the
previously acquired knowledge about algorithms and data structures. Despite
having learned such knowledge, LLMs struggle to effectively apply it when faced
with specific new problems. To address this issue, we constructed a novel
dataset, CodeF, which contains a portion of programming problems that ChatGPT
has not previously encountered. Furthermore, we developed a Knowledge Library
tailored for Python programming contest problems and introduced the concept of
Knowledge-Aware Code Generation (KareCoder). KareCoder bolsters the models'
understanding and problem-solving capabilities by integrating prompt and
knowledge from the library into the LLMs' code generation reasoning process,
especially on Pass@1 metrics. Upon testing on the CodeF and APPS datasets,
KareCoder demonstrated outstanding performance in handling novel problems
previously unencountered by LLMs. In contrast with the code directly generated
by ChatGPT, KareCoder achieved a relative improvement of 23.3
metric on the CodeF post2021-9 dataset. Additionally, it performs well compared
to other methods when dealing with problems that LLMs have previously
encountered.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要