Evaluating the Effectiveness of LLMs in Introductory Computer Science Education: A Semester-Long Field Study
arxiv(2024)
摘要
The integration of AI assistants, especially through the development of Large
Language Models (LLMs), into computer science education has sparked significant
debate. An emerging body of work has looked into using LLMs in education, but
few have examined the impacts of LLMs on students in entry-level programming
courses, particularly in real-world contexts and over extended periods. To
address this research gap, we conducted a semester-long, between-subjects study
with 50 students using CodeTutor, an LLM-powered assistant developed by our
research team. Our study results show that students who used CodeTutor (the
experimental group) achieved statistically significant improvements in their
final scores compared to peers who did not use the tool (the control group).
Within the experimental group, those without prior experience with LLM-powered
tools demonstrated significantly greater performance gain than their
counterparts. We also found that students expressed positive feedback regarding
CodeTutor's capability, though they also had concerns about CodeTutor's limited
role in developing critical thinking skills. Over the semester, students'
agreement with CodeTutor's suggestions decreased, with a growing preference for
support from traditional human teaching assistants. Our analysis further
reveals that the quality of user prompts was significantly correlated with
CodeTutor's response effectiveness. Building upon our results, we discuss the
implications of our findings for integrating Generative AI literacy into
curricula to foster critical thinking skills and turn to examining the temporal
dynamics of user engagement with LLM-powered tools. We further discuss the
discrepancy between the anticipated functions of tools and students' actual
capabilities, which sheds light on the need for tailored strategies to improve
educational outcomes.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要