Exploring the Design Space of Automatically Synthesized Hints for Introductory Programming Assignments.

For massive programming classrooms, recent advances in program synthesis offer means to automatically grade and debug student submissions, and generate feedback at scale. A key challenge for synthesis-based autograders is how to design personalized feedback for students that is as effective as manual feedback given by teachers today. To understand the state of hint-giving practice, we analyzed 132 online Q&A posts and conducted a semi-structured interview with a teacher from a local massive programming class. We identified five types of teacher hints that can also be generated by program synthesis. These hints describe transformations, locations, data, behavior, and examples. We describe our implementation of three of these hint types. This work paves the way for future deployments of automatic, pedagogically-useful programming hints driven by program synthesis.