Rocks Coding, Not Development–A Human-Centric, Experimental Evaluation of LLM-Supported SE Tasks
CoRR(2024)
摘要
Recently, large language models (LLM) based generative AI has been gaining
momentum for their impressive high-quality performances in multiple domains,
particularly after the release of the ChatGPT. Many believe that they have the
potential to perform general-purpose problem-solving in software development
and replace human software developers. Nevertheless, there are in a lack of
serious investigation into the capability of these LLM techniques in fulfilling
software development tasks. In a controlled 2 x 2 between-subject experiment
with 109 participants, we examined whether and to what degree working with
ChatGPT was helpful in the coding task and typical software development task
and how people work with ChatGPT. We found that while ChatGPT performed well in
solving simple coding problems, its performance in supporting typical software
development tasks was not that good. We also observed the interactions between
participants and ChatGPT and found the relations between the interactions and
the outcomes. Our study thus provides first-hand insights into using ChatGPT to
fulfill software engineering tasks with real-world developers and motivates the
need for novel interaction mechanisms that help developers effectively work
with large language models to achieve desired outcomes.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要