Mutation testing (MT) is a powerful technique that evaluates test suite quality by introducing intentional faults (mutations) into code. In junior-level Data Structures and Algorithms (DSA) courses, students are tasked with developing programming projects with multi-week life cycles where MT could improve their test suites and coding skills. However, teaching students to effectively use MT is challenging. This study introduces a modified version of MT feedback combined with training content as an alternative to traditional code coverage for improving student test suites. We conducted a comparative analysis across three semesters that used different testing approaches: code coverage, MT with standard feedback from a widely used MT implementation (PIT), and MT with feedback modified and training to better support student understanding. We support our findings using 38 semi-structured interviews and quantitative analysis of IDE activity data and students' code submissions. Initial implementation of standard PIT MT with a reduced set of mutation operators showed mixed results, as the stock feedback failed to meet student needs. We developed an improved approach incorporating additional documentation and training with practical examples, client-side support for MT, and improved feedback messages. The revised feedback and training was associated with positive results in student understanding and application of MT principles. This research provides a framework for effectively integrating MT in programming courses, contributing to improved student test suite development, better student code, and better project outcomes. The findings offer practical guidelines for instructors teaching testing in CS courses, bridging the gap between industry tools and educational needs.

IntroductionSelf-regulated learning skills are necessary for academic success. While not all students entering post-secondary education are proficient at many of these critical skills, they can be improved upon when practiced. However, self-regulation tends to be highly internal, making it difficult to measure. One form of measurement comes from using data traces collected from educational software. These allow researchers to make strong empirical inferences about a student's internal state. Automatically captured data traces also make it possible to provide automated interventions that help students practice and master self-regulated learning skills.Methods/resultsUsing an experimental methodology we created a set of promising data traces that are grounded in theory to study self-regulated learning within a typical Computer Science course. Extra attention is given to studying the skill of help-seeking, which is both a key to success in CS and requires unobtrusive observation to properly measure.DiscussionWe also make the case for taking a broader perspective with our data collection efforts. The traces identified in this paper are not from one source, but the full ecosystem of software tools common to CS courses.

The rise of interactive eTextbooks opens new opportunities for enhancing student engagement and learning outcomes. However, analyzing student interactions within these digital platforms remains challenging. This study examines student engagement profiles in OpenDSA, an interactive eTextbook for data structures and algorithms courses. Using session-level interaction data, we categorize engagement into four distinct engagement states, defined as types of student activities: Reading, Visualization, Proficiency Exercises, and Multiple Choice Exercises. We then apply clustering techniques to identify distinct engagement profiles, characterized by the frequency of transitions and total engagement time spent in each engagement state. Our research addresses two key questions: (1) What engagement profiles can be identified from students' interactions across these four engagement states? (2) How do these engagement profiles correlate with students' academic performance? Our findings reveal four distinct engagement profiles: Highly Engaged Learners, exhibiting frequent transitions and high engagement across all engagement states; Moderately Engaged Learners, characterized by sporadic interactions and below-average overall engagement; Balanced Learners, maintaining consistent and moderate engagement across engagement states, and Minimally Engaged Learners, demonstrating limited engagement and infrequent state transitions. Statistical analysis confirms that students in profiles with frequent and diverse engagement significantly outperform minimally engaged learners academically. These results underline the critical role of active, varied engagement in student success and underline the potential of session-level data for monitoring and optimizing student engagement. We believe our findings will be valuable to eTextbook developers, providing actionable insights to guide the design of digital content and targeted interventions that improve student engagement and performance.

Mutation testing (MT) is a powerful technique for evaluating the quality of software test suites. MT introduces faults or "mutations" into the code and checks whether the tests then fail as appropriate. While MT is known to be more effective than code coverage as a measure of test quality, its computational cost makes it challenging to deploy in educational settings. In this paper, we show the effects of this computational demand on an auto-grading system when MT was used in a junior-level Data Structures and Algorithms (DSA) course. Through a comparative study spanning semesters with and without MT, we observed a noticeable increase on the auto-grader's processing time and feedback turnaround time (about 30-50 s, which represents roughly a tripling in per-submission processing time) for students whose projects are graded with MT. This additional load raises concerns that it might overload the server, causing delays for students in other courses. However, with suitable mitigation strategies in place, the only measurable impact on other students was a higher variance in feedback turnaround times during peak use. One such mitigation strategy is the use of a local MT plug-in which helped to reduce the total number of submissions to the auto-grader. Overall, we find the effects on server load from a carefully chosen set of mutations combined with moderate use of local MT to have an acceptable computational cost on the system load while improving student test suite quality.

When students use an online eTextbook with content and interactive graded exercises, they often display aspects of two types of behavior: credit-seeking and knowledge-seeking. A student might behave to some degree in either or both ways with given content. In this work, we attempt to detect the degree to which either behavior takes place and investigate relationships with student performance. Our testbed is an eTextbook for teaching Formal Languages, an advanced Computer Science course. This eTextbook uses Programmed Instruction framesets (slideshows with frequent questions interspersed to keep students engaged) to deliver a significant portion of the material. We analyze session interactions to detect credit-seeking incidents in two ways. We start with an unsupervised machine learning model that clusters behavior in work sessions based on sequences of user interactions. Then, we perform a fine-grained analysis where we consider the type of each question presented within the frameset (these can be multi-choice, single-choice, or T/F questions). Our study involves 219 students, 224 framesets, and 15,521 work sessions across three semesters. We find that credit-seeking behavior is correlated with lower learning outcomes for students. We also find that the type of question is a key factor in whether students use credit-seeking behavior. The implications of our research suggest that educational software should be designed to minimize opportunities for credit-seeking behavior and promote genuine engagement with the material.

Background and ContextSelf-regulated learning (SRL) skills are critical aspect of learning to program and are predictive of academic success. Early college students often struggle to use these skills, but can improve when given targeted instruction. However, it is not yet clear what skills are best to prioritize.ObjectiveWe seek to create a shortlist of SRL skills that help computer science students with an ultimate goal of giving future researchers and educators a set of promising targets for future study.MethodThis work details a three-phase Delphi-Process study wherein a panel educators to reach consensus on SRL skills needed for success in CS.FindingsWe identified fourteen skills broadly categorized into five areas: Planning, Executing, Monitoring, Responding, and Reflecting. There are several factors to indicate these skills are all highly valuable areas for future work.ImplicationsWe highlight Knowing When to Seek Help especially for future research within the Computing Education Research community.

Mutation testing is used to gauge the quality of software test suites by introducing small faults, called "mutations'', into code to assess if a test suite can detect them. Although it has been applied extensively in the software industry, mutation testing's use in programming courses faces both computational and pedagogical barriers. This study examines the impact of mutation testing on student performance in a post-CS2 Data Structures and Algorithms course with 3-4 week life-cycle programming projects. We collected a semester of data with projects using only code coverage (control group) and another semester that used mutation testing (experimental group). We investigated three aspects of mutation testing impact: the quality of student-written test suites, the correctness and complexity of students' solution code, and the degree of incremental test writing. Our findings suggest that students using mutation testing, as a group, demonstrated higher quality test suites and wrote better solution code compared to students using traditional code coverage methods. Students using mutation testing were more likely to exhibit incremental testing practices.

Many SIGCSE attendees are either developing or using online educational tools, and all will benefit from better interoperability among these tools and better analysis of the clickstream data coming from those tools. New tools for analyzing big data leveraged by AI (e.g., deep learning for assessment) in turn improve both content and pedagogy, thus setting up a virtuous cycle fueling learning discoveries and leveraging innovation in AI: Online technologies → big data analysis → better online technologies. This NSF-supported workshop is the latest in a series of SPLICE workshops, and is a continuation of our event at SIGCSE 2023, where the SPLICE-Portal, a dedicated socio-technical research infrastructure for Computing Education Research, was presented. This year, we continue the work with several new SPLICE community working groups, including those on Dashboards, Large Language Models, Parsons Problems, and Smart Learning Content Protocols. We continue to build upon our existing collaborations developed over the course of the project to engage more members of the community in tasks that will advance the project agenda.

The content in Formal Languages courses is mathematical in nature, and requires students to engage with proofs and algorithms to grasp core concepts. Conventional textbooks on Formal Languages predominantly employ textual explanations, with assignments often entailing manual problem solving. Some educators incorporate tools like JFLAP, which helps students construct models and apply algorithms to enhance interaction with the subject matter. However, students must put considerable effort into reading and solving problems manually to reach comprehension. Drawing inspiration from the Programmed Instruction (PI) teaching methodology, we have developed an innovative eTextbook for Formal Languages that facilitates better understanding of these ideas. The PI approach requires students to read a bit, ideally a sentence or paragraph, and then answer a question or complete an exercise related to that information. Depending on their response, students can progress to subsequent information frames or re-attempt the exercise. Our objective is to present the entirety of a Formal Languages curriculum through the PI approach. To evaluate the pedagogical effectiveness of our new eTextbook, we conducted a survey to ask students for their feedback on their experience with the Programmed Instruction etextbook. We also conducted performance evaluations on two offerings of the Formal Languages course. Students' grades are compared to assess learning gains between visualizations with exercises, and with PI frames. The evaluation shows that the Programmed Instruction ebook improved students' grades in almost all topics covered by the ebook.

Student perceptions of the complete online transition of two CS courses in response to the COVID-19 pandemic Due to the COVID-19 pandemic, universities across the globe switched from traditional Face-to-Face (F2F) course delivery to completely online. Our university declared during our Spring break that students would not return to campus, and that all courses must be delivered fully online starting two weeks later. This was challenging to both students and instructors. In this evidence-based practice paper, we present results of end-of-semester student surveys from two Spring 2020 CS courses: a programming intensive CS2 course, and a senior theory course in Formal Languages and Automata (FLA). Students indicated course components they perceived as most beneficial to their learning, before and then after the online transition, and preferences for each regarding online vs. F2F. By comparing student reactions across courses, we gain insights on which components are easily adapted to online delivery, and which require further innovation. COVID was unfortunate, but gave a rare opportunity to compare students' reflections on F2F instruction with online instructional materials for half a semester vs. entirely online delivery of the same course during the second half. The circumstances are unique, but we were able to acquire insights for future instruction. Some course components were perceived to be more useful either before or after the transition, and preferences were not the same in the two courses, possibly due to differences in the courses. Students in both courses found prerecorded asynchronous lectures significantly less useful than in-person lectures. For CS2, online office hours were significantly less useful than in-person office hours, but we found no significant difference in FLA. CS2 students felt less supported by their instructor after the online transition, but no significant difference was indicated by FLA students. FLA students found unproctored online exams offered through Canvas more stressful than in-person proctored exams, but the opposite was indicated by CS2 students. CS2 students indicated that visual materials from an eTextbook were more useful to them after going online than before, but FLA students indicated no significant difference. Overall, students in FLA significantly preferred the traditional F2F version of the course, while no significant difference was detected for CS2 students. We did not find significant effects from gender on the preference of one mode over the other. A serendipitous outcome was learning that some changes forced by circumstance should be considered for long term adoption. Offering online lab sessions and online exams where the questions are primarily multiple choice are possible candidates. However, we found that students need to feel the presence of their instructor to feel properly supported. To determine what course components need further improvement before transitioning to fully online mode, we computed a logistic regression model. The dependent variable is the student's preference for F2F or fully online. The independent variables are the course components before and after the online transition. For both courses, in-person lectures were a significant factor negatively affecting students' preferences of the fully online mode. Similarly, for CS2, in-person labs and in-person office hours were significant factors pushing students' preferences toward F2F mode.

Computer Science programs have seen high enrollments in recent years, which contributed to widening the capacity gap. One way to address this problem is to hire more teaching-focused faculty at both research and non-doctoral granting institutions. Although this kind of hiring has already been taking place in several institutions, PhD-granting CS departments have not been able to produce enough PhDs to meet the increasing demand, especially for PhD holders with interest in - and capacity for - teaching. In this paper, we describe our experience with the initial phase of building a training program within our (large, land grant, R1) institution, targeting graduate students interested in pursuing an academic teaching-focused career in CS. Through a semester-long set of meetings, conversations, and activities, we worked with participants on improving their teaching skills and applying effective pedagogies in the classroom. At the end of the semester, we surveyed participants about the value of those meetings to them, ideas for improvement, and perspectives for future directions. Most participants rated the meetings positively in terms of content relevance and usefulness, and the opportunity to connect and interact with other participants and invited faculty members. We also discuss the lessons learned and best practices, which can be widely applied by other departments looking to better prepare their graduate students for a CS teaching-focused faculty position.

Repeated deliberate practice has been shown to be vital to developing mastery in engineering problem solving. Online tutoring systems have enhanced learning experiences, and delivered content tailored for specialized fields. Motivated by the aim of improving students' problem solving skills, we created an interactive system for use in an undergraduate introductory engineering mechanics course required for many engineering disciplines. Our system provides an intuitive, visual framework that allows students to rapidly solve problems that require building systems of equations in multiple steps. Built within the OpenDSA eTextbook system, these exercises can be served directly through a learning management system such as Canvas, allowing the exercises to be integrated seamlessly with other content. In this paper, we describe the key design choices for our system, present important features and the student workflow, and describe support for targeted feedback and analysis for the instructors. We present our plans to evaluate the system, and discuss the results of a preliminary usability study.

As more students interact with online learning platforms and eTextbooks, they generate massive amounts of data. For example, the OpenDSA eTextbook system collects clickstream data as users interact with prose, visualizations, and interactive auto-graded exercises. Ideally, instructors and system developers can harness this information to create better instructional experiences. But in its raw event-level form, it is difficult for developers or instructors to understand student behaviors, or to make testable hypotheses about relationships between behavior and performance. In this study, we describe our efforts to break raw event-level data first into sessions (a continuous series of work by a student) and then to meaningfully abstract the events into higher-level descriptions of that session. The goal of this abstraction is to help instructors and researchers gain insights into the students’ learning behaviors. For example, we can distinguish when students read material and then attempt the associated exercise, versus going straight to the exercise and then hunting for the answers in the associated material. We first bundle events into related activities, such as the events associated with stepping through a given visualization, or with working a given exercise. Each such group of events defines a state. A state is a basic unit that characterizes the interaction log data, and there are multiple state types including reading prose, interacting with visual contents, and solving exercises. We harnessed the abstracted data to analyze studying behavior and compared it with course performance based on GPA. We analyzed data from the Fall 2020 and Spring 2021 sections of a senior-level Formal Languages course, and also from the Fall 2020 and Spring 2021 sections of a data structures course. Analyzing Student Session Data in an eTextbook

The material taught in a Formal Languages and Automata (FLA) course is mathematical in nature and requires students to practice proofs and algorithms to understand the content. Traditional FLA textbooks are heavy on prose, and homework typically consists of solving many paper exercises. Instructors often make use of Finite State Machine simulators like the JFLAP package. JFLAP allows students to interactively build models and apply different algorithms to these models, providing both a more interactive and a more visual approach. However, course materials have still traditionally relied largely on prose and hand-graded exercises, limiting both the interaction and the amount of practice. In this paper, we propose an eTextbook with integrated tools (simulators and auto-graded exercises) that allow for greater interactivity and levels of engagement. To evaluate the pedagogical effectiveness of our approach, we conducted performance evaluations across different offerings of an FLA course. Results indicate that students using the integrated eTextbook performed better than did a control group using a traditional textbook approach. Students gave positive feedback regarding the usefulness of the auto-graded exercises for practicing different FLA concepts.

COVID-19 caused universities to switch from traditional face-to-face (F2F) course delivery to completely online in Spring 2020. This transition took place on short notice in the middle of the semester. We present results from surveys of students in two CS courses offered at Virginia Tech. Results indicate differing perceptions in the two courses regarding the usefulness of course components before and after the transition for each course. A logistic regression model indicates that for each course, different course components both before and after the transition significantly affect students' preferences for course modality.

Students often skip through instructional material without paying enough attention to the content, resulting in less understanding. Inspired by the Programmed Instruction (PI) technique, we implemented extensions to an eTextbook system to support instructional slideshows with a large number of interspersed questions. Students must answer a question correctly to pass to the next slide. This completely changes how students interact with the material. Our initial results show a significant increase in students' grades when they used PI for a Formal Languages course.

When students interact with an eTextbook, it typically logs their interactions while engaged in activities like watching a visualization, attempting to solve an exercise, or refreshing the page. These event logs allow instructors and researchers to evaluate students’ engagement level and approaches to us-ing the artifacts. We predict that the way students use the book and the artifacts affects their performance on the exercises, their learning gains, and their performance in other aspects of the course. In this paper, we describe a data set gathered from a complete semester course on Formal Languages. This includes all student interactions with the Formal Languages eText-book. The book contains a set of auto-graded exercises and visualizations about Formal Languages course contents in the form of slideshows.

As incorporating software testing into programming assignments becomes routine, educators have begun to assess not only the correctness of students’ software, but also the adequacy of their tests. In practice, educators rely on code coverage measures, though its shortcomings are widely known. Mutation analysis is a stronger measure of test adequacy, but it is too costly to be applied beyond the small programs developed in introductory programming courses. We demonstrate how to adapt mutation analysis to provide rapid automated feedback on software tests for complex projects in large programming courses. We study a dataset of 1389 student software projects ranging from trivial to complex. We begin by showing that although the state-of-the-art in mutation analysis is practical for providing rapid feedback on projects in introductory courses, it is prohibitively expensive for the more complex projects in subsequent courses. To reduce this cost, we use a statistical procedure to select a subset of mutation operators that maintains accuracy while minimizing cost. We show that with only 2 operators, costs can be reduced by a factor of 2–3 with negligible loss in accuracy. Finally, we evaluate our approach on open-source software and report that our findings may generalize beyond our educational context.

Textbooks for theory courses in CS tend to be heavy on prose and mathematics. We find that students do not engage such material, and skip or rush through it without understanding. To increase students level of engagement, we developed support within the OpenDSA eTextbook system support for creating materials based on the Programmed Instruction pedagogical paradigm. This requires near-constant activity by the student, who must read a little, ideally a sentence or a paragraph, and then answer a question or complete an exercise related to that information. Based on the question response, students are permitted to continue, or must retry to solve the exercise. Versions of the eTextbook have been used to teach the senior-level Formal Languages course at Virginia Tech for two semesters. In this demonstration, we show how students interact with material developed using the Programmed Instruction approach.