Towards Deriving Programming Competencies from Student Errors

Learning outcomes are more and more defined and measured in terms of competencies. Many research projects are conducted that investigate combinations of knowledge and skills that students might learn. Yet, it is also promising to analyze what students might fail to learn, which provides information about the absence of certain competencies. For this purpose, we are evaluating the outcomes of automatic assessment tools that provide automatic feedback to the participating students. In particular, we analyzed the errors of the students that participated in an introductory programming course. The 604 students participating in the course had to solve six tasks during the semester, resulting in a total of 12274 submissions. The error analysis is done by evaluating the data from the automatic assessment tool JACK, which provides automatic feedback on programming tasks. To derive information about prospective competencies, we conducted a qualitative analysis of the different errors the students made in their solutions. The results provide interesting insights into missing competencies. In further research our findings have to be validated by investigating the cognitive processes involved during programming.