Improving LLM Classification of Logical Errors by Integrating Error Relationship into Prompts
Generative Intelligence and Intelligent Tutoring Systems Lecture Notes in Computer Science(2024)
摘要
LLMs trained in the understanding of programming syntax are now providing
effective assistance to developers and are being used in programming education
such as in generation of coding problem examples or providing code
explanations. A key aspect of programming education is understanding and
dealing with error message. However, 'logical errors' in which the program
operates against the programmer's intentions do not receive error messages from
the compiler. In this study, building on existing research on programming
errors, we first define the types of logical errors that can occur in
programming in general. Based on the definition, we propose an effective
approach for detecting logical errors with LLMs that makes use of relations
among error types in the Chain-of-Thought and Tree-of-Thought prompts. The
experimental results indicate that when such logical error descriptions in the
prompt are used, the average classifition performance is about 21
the ones without them. We also conducted an experiment for exploiting the
relations among errors in generating a new logical error dataset using LLMs. As
there is very limited dataset for logical errors such benchmark dataset can be
very useful for various programming related applications. We expect that our
work can assist novice programmers in identifying the causes of code errors and
correct them more effectively.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要