Background and Context. Computer programming often involves extended periods of sustained activity and mobile phone notifications introduce frequent opportunities for interruption. Prior work demonstrates that suppressing phone notifications may reduce these disruptions. Objectives. Our primary research question is: How does suppressing phone notifications affect students' task engagement and productivity while programming? Method. We report on a replication and methodological extension study conducted in a CS1 course involving 22 students. Using a within-subject design, selected programming assignments were randomly designated for enabling notification suppression. Phone state logs were synchronized with millisecond-resolution IDE keystroke data to measure student attention and focus when in the control and notification-suppression conditions. Findings. Assignments completed with notification suppression enabled significantly lower break rates and longer intervals of focus compared to assignments completed in the control condition for many, but not all, students. This study provides evidence that notification suppression is associated with measurable differences in programming engagement and behavior. We also find a remarkable bimodality in the effect across students – many students are positively affected, a small number are negatively affected, and very few experience little or no effect. This finding is consistent with other studies in diverse disciplines. Implications. Our results show that, for many students, phone notification suppression tools, such as Do Not Disturb, can improve attention and focus. Implications apply to educational settings (do-not-disturb as an intervention) and scholarship (understanding the effects of phone distraction).

Background and Context: Large Language Models (LLMs) are more accessible and accurate than ever before, raising significant concerns for computing educators. One major concern is students using LLMs to bypass the effort needed to understand concepts and metacognitive strategies essential for success in computer science. Objectives: We contribute a unique approach to assessing and building up student understanding through weekly oral code review assessments. These formative assessments incentivize students to understand their submitted code, regardless of whether or not the code was generated by AI tools. We also use a flipped classroom to provide time for students to learn concepts outside of class and provide ample time for students to schedule code review interviews. Methods: For this paper, we collected data from three semesters. We analyze student exam scores, keystroke logs, and surveys to understand how the new course policies affected student learning, behavior, and attitudes. Findings: Pairwise comparison of exam results reveals a statistically insignificant increase in average scores for Fall 2025 compared to previous semesters. Keystroke logs show a significant increase in characters pasted per total characters input into coding assignments in Fall 2025, pointing towards higher AI usage. Survey results show positive student sentiment towards code reviews at the end of Fall 2025, with nearly all negative feedback being addressable through better scheduling and more rigorous TA training. Implications: Oral code reviews with a flipped classroom appear to be effective at mitigating harms of LLM use while providing space for students to freely experiment with these tools. Our work suggests that students in Fall 2025 still show adequate understanding of material covered in written exams, despite dramatic increases in LLM usage for coding assignments.

Learning the process of writing a computer program is an important part of introductory computer programming education. Many educators attempt to teach programming process implicitly through live coding. One major obstacle preventing more direct instruction on process is that we have no empirically derived terms to describe it. In this paper, we begin to build a set of programming process descriptors by classifying topics and concepts that students pay attention to as they watch replays of other students writing code. We conducted think-aloud interviews with students in a CS1 course while they watched reconstructed playbacks of peers completing a programming task. We observe that, surprisingly, students talk about the temporal aspects of writing code almost twice as often as the code itself. Some participants did not recognize their own code-writing process when played back to them. We also find that students are surprisingly interested in what the programmer is thinking while writing. We identify clusters of similar phrases that students use to describe the evolution of code, such as being written "top to bottom" or "jumping back and forth". These descriptive features will, in future work, be used to develop and validate quantifiable measures of programming process that are grounded in empirical measures of student behavior.

Self-efficacy, sense of belonging, and imposter phenomenon are all phenomena related to an individual's perception of themselves and others. They are well-studied with mature, validated surveys, and they all have implications on personal performance, including in the CS education context. One question that none of the phenomena address, however, is how a student perceives their own performance relative to their peers. For example, none of these accepted measures considers a student who thinks that "everyone in this classroom is smarter than me." In this paper, we propose a survey to measure a previously unexplored phenomenon experienced by computer science (CS) students. We call this phenomenon relative self-efficacy, and it measures how highly a student perceives their ability relative to their peers. No prior attempt has been made to measure how students rate themselves relative to others or to understand the causes and effects of these comparisons. We find that the underlying factors of our survey are distinct from those that measure sense of belonging. We also find that women are more likely to experience lower relative self-efficacy than men. Lastly, we find that GPA is weakly correlated with relative self-efficacy and suggest that there may be stronger, less obvious influences on relative self-efficacy.

Background and Context: Problem decomposition is a fundamental computational thinking skill that novice programmers struggle to develop. Understanding and improving decomposition skills remains challenging for educators. Objective: We investigate how students use a "natural language functions" (NLFs) tool that generates callable functions from natural language prompts, examining their decomposition behaviors compared to students without access to this generative AI tool. Method: We conducted a mixed-methods study combining thinkaloud protocols, automated metrics collection, and qualitative analysis. In a quasi-experiment, an experimental group (n=6) used NLFs to solve two programming tasks, while a control group (n=5) solved the same tasks without NLFs. Findings: Students with NLFs access exhibited significantly more decomposition behavior, creating approximately three times as many functions as the control group. We observed a shift from verbal articulation to written expression through the prompt interface, with experimental students spending less time verbalizing intent and more time crafting prompts. This suggests prompting may serve as a proxy for traditional programming behaviors. Implications: Students who articulate functionality in natural language demonstrate enhanced decomposition behaviors. Tools like NLFs can serve as valuable pedagogical scaffolds that encourage structured thinking through natural language articulation. By requiring students to explicitly describe discrete functions before implementation, such tools may make metacognitive processes more visible and teach decomposition skills that are otherwise difficult to convey, potentially improving learning outcomes in introductory programming courses.

The third version of the Regional Atmosphere and Land (RAL3) science configuration is documented. Developed through international partnerships, RAL configurations define settings for the Unified Model atmosphere and Joint UK Land Environment Simulator (JULES) when applied across timescales with kilometre and sub-kilometre-scale model grids. The RAL3 configuration represents a major advance compared to previous versions by delivering a common science definition suitable for application to tropical and mid-latitude regions. Developments within RAL3 include the introduction of a double-moment microphysics scheme and a bimodal cloud scheme, replacing use of a single-moment scheme and different cloud schemes for mid-latitudes and tropics in previous versions. Updates have been implemented to the boundary layer scheme and a consolidation of land model settings to be more consistent with global atmosphere and land (GAL) science configurations. Physics developments aimed to address priorities for model performance improvement identified by users. This paper documents the RAL3 science configuration, including a series of iterative revisions delivered since its first release, and their characteristics. Evidence is provided from the variety of assessments of RAL3, relative to the previous version (RAL2). Collaborative development and evaluation across organizations have enabled evaluation across a range of domains, grid spacing and timescales. The analysis indicates more realistic precipitation distributions, improved representation of clouds and of visibility, a continued trend to more realistic representation of convection, and reduced near-surface wind speeds but a persistent cold-temperature bias. Overall the convective-scale verification scores and climatological model distributions relative to observations improve for the majority of variables. Ensemble results show improvements to the spread-error relationship. User feedback from subjective assessment activities has also been positive. Differences between RAL3 revisions and RAL2 are further illustrated through a process-based analysis of a convective system over the UK. The latest RAL3 configuration (RAL3.3) is therefore recommended for research, operational numerical weather prediction, and climate production at kilometre and sub-kilometre scales.

Digital simulations are powerful instructional tools for physics education. They are often designed to visualize canonical physical phenomena, with adjustable parameters for influencing the system. While this is sufficient for developing conceptual and qualitative intuitions, it does little to help physics students build connections between physical systems and the mathematical models and equations that represent them. We present PhysMath, a suite of interactive physics simulations for use in upper-division courses. These simulations allow students to explore connections between mathematical equations and the phenomena they represent by inputting, modifying, and observing changes in system behavior. In this paper, we describe our first simulation—the Bead-On-Hoop for Classical Mechanics—and report findings from pilot interviews with intermediate physics students interacting with the simulation. Our findings validate the simulations’ design and highlight its potential for scaffolding students’ mathematical sensemaking.

Background and Context: Learning programming syntax is challenging for novice programmers. One promising approach to teaching syntax in introductory computer science (CS) courses is assigning syntax exercises: simple, repetitive problems where students are asked to make a small modification to code or practice recalling and typing syntax. Prior work has shown that syntax exercises, as an addition to a course, are beneficial for all students, and perhaps especially helpful for students without prior CS experience. Still, it is unclear how syntax exercises compare to other types of practice problems. Objective: The goal of our study was to investigate whether syntax exercises are a beneficial intervention when compared to other programming practice problems. We evaluated if syntax exercises impact students’ (1) average grade on programming problems, (2) syntax skills, (3) CS self-efficacy, or (4) beliefs about CS, and if they had a differential benefit for students without prior CS experience. Method: We conducted a randomized-controlled trial in an undergraduate introductory CS course. As part of their weekly homework, all students completed practice programming problems. Students in the treatment group were also assigned to complete syntax exercises, whereas students in the control group were assigned to complete additional practice programming problems. Findings: When controlling for students’ prior CS experience, we did not find any statistically significant differences between students assigned to syntax exercises versus practice programming problems on (1) average grade on programming problems, (2) syntax skills, (3) CS self-efficacy, or (4) beliefs about CS. We also did not find any differential benefit of syntax exercises for students without prior CS experience. Implications: Our findings suggest that syntax exercises may not provide additional benefits compared to practicing programming problems. The results also shed light on how design choices in syntax exercises and other course factors may affect outcomes. Our work introduces a framework to describe variation in the broader class of syntax-focused practice activities, which can guide future research of interventions to support students’ early learning of programming skills.

The ability of a human to retrospectively estimate the amount of time spent on a task is largely only understood when the period of time is seconds- or minutes-long. The lack of research into estimation of longer periods of time can be attributed, in part, to the difficulty of measuring ground truth durations when the task is broken up by other activities in a natural, day-to-day setting. An empirically based model of engagement was recently proposed that statistically estimates time-on-task for computer programming assignments in an introductory computer programming course. Computer programming assignments can be completed in many sessions across days or weeks and, based on recorded keystroke data, an objective ground truth of task duration can be measured. In this work, we take advantage of this new measurement method to explore duration estimation of tasks lasting hours that are spread out over multiple days in a natural setting. Subjects in our study overestimated time-on-task 78% of the time and reported a median of 1.45 hours worked for every actual hour spent on task. We find that self-reports are more accurate when students score higher on their assignments in our data, suggesting the accuracy of estimated time is correlated with task performance.

Computing educators and researchers have used programming process data to understand how programs are constructed and what sorts of problems students struggle with. Although such data shows promise for using it for feedback, fully automated programming process feedback systems have still been an under-explored area. The recent emergence of large language models (LLMs) have yielded additional opportunities for researchers in a wide variety of fields. LLMs are efficient at transforming content from one format to another, leveraging the body of knowledge they have been trained with in the process. In this article, we discuss opportunities of using LLMs for analyzing programming process data. To complement our discussion, we outline a case study where we have leveraged LLMs for automatically summarizing the programming process and for creating formative feedback on the programming process. Overall, our discussion and findings highlight that the computing education research and practice community is again one step closer to automating formative programming process-focused feedback.

How novice programmers navigate through errors, length of pauses, and states of compilation while writing a program can provide valuable information in computing education research. In this paper, we analyze keystroke datasets collected from assignments of the CS1 course from 44 students from Utah State University. We propose a metric called Average Known Recovery (AKR) to measure and understand the efficiency of students who could resolve their code faster after knowing their program is in an uncompilable state. Surprisingly, we discovered longer pauses were more common in the executable states instead of error states. This suggests that longer pauses are not only driven by the cognitive load or frustration of being in an uncompilable state. Visualization of cursor positions for each event, along with states of compilability, helps to explore the programming flow of students for a particular assignment. These findings can be helpful in designing programming pedagogy and intervention strategies to help novice programmers.

Pausing behavior in introductory Computer Science (CS1) courses has been related to course outcomes and could be linked to a student's cognitive load. Using Cognitive Load Theory and Vygotsky's Zone of Proximal Development as a theoretical framework, this study empirically analyzes keystroke latencies, or pause times between keystrokes, with the goal of better understanding what types of assignments need more scaffolding than others. We report the characteristics of eleven assignments, introduce a method to analyze pausing behavior, and investigate how pausing behavior changes with assignment characteristics (e.g., introducing new programming constructs, engaging creativity through Turtle graphics, etc). We find evidence that pausing behavior does change based on the assignment characteristics and that assignments with particular characteristics, such as object-oriented principles, may be more likely to have excessive demands on student working memory. We also find evidence that assignment completion time may not be an accurate measure of assignment difficulty.

The following paper addresses student attitudes towards keystroke logging and plagiarism prevention measures. Specifically, the paper concerns itself with changes made to the “ShowYourWork” plugin, which was implemented to log the keystrokes of students in Utah State University's introductory Computer Science course, CS1400. Recent work performed by the Edwards Lab provided insights into students' feelings towards keystroke logging as a measure of deterring plagiarism. As a result of that research, we decided to investigate how students' perception of keystroke logging and plagiarism changes when they are given more control over their data. This paper introduces the work done to enhance student privacy and feelings of security and an evaluation of the effectiveness of the changes made to ShowYourWork.

The rapid advancement of artificial intelligence (AI) and the expanding integration of large language models (LLMs) have ignited a debate about their application in education. This study delves into university instructors' experiences and attitudes toward AI language models, filling a gap in the literature by analyzing educators' perspectives on AI's role in the classroom and its potential impacts on teaching and learning. The objective of this research is to investigate the level of awareness, overall sentiment towardsadoption, and the factors influencing these attitudes for LLMs and generative AI-based tools in higher education. Data was collected through a survey using a Likert scale, which was complemented by follow-up interviews to gain a more nuanced understanding of the instructors' viewpoints. The collected data was processed using statistical and thematic analysis techniques. Our findings reveal that educators are increasingly aware of and generally positive towards these tools. We find no correlation between teaching style and attitude toward generative AI. Finally, while CS educators show far more confidence in their technical understanding of generative AI tools and more positivity towards them than educators in other fields, they show no more confidence in their ability to detect AI-generated work.

Coding Bootcamps are a relatively new phenomenon in computer science education. As such, few academic studies have assessed their effectiveness in preparing students for programming careers. Completed studies maintain a focus on the short-term results and outcomes of these programs. This paper describes a meta-analysis of multiple short-term outcomes studies and proposes the combined results as a baseline dataset for future studies. It also describes a study which gathered long-term outcomes for coding bootcamps and certificate programs and identify common factors among graduates who either successfully transitioned into software engineering, or achieved higher salaries. Lastly, it compares the long-term data to the short-term baseline dataset.

Tools based on generative artificial intelligence (AI), such as ChatGPT, have quickly become commonplace in education, particularly in tasks like programming. We report on a study exploring how students use a tool similar to ChatGPT, powered by GPT-4, while working on Introductory Computer Programming (CS1) assignments, addressing a gap in empirical research on AI tools in education. Utilizing participants from two CS1 class sections, our research employed a custom GPT-4 tool for assignment assistance and the ShowYourWork plugin for keystroke logging. Prompts, AI replies, and keystrokes during assignment completion were analyzed to understand the state of students' programs when they prompt the AI, the types of prompts they create, and whether and how students incorporate the AI responses into their code. The results indicate distinct usage patterns of ChatGPT among students, including the finding that students ask the AI for help on debugging and conceptual questions more often than they ask the AI to write code snippets or complete solutions for them. We hypothesized that students ask conceptual questions near the beginning and debugging help near the end of program development do not find statistical evidence to support it. We find that large numbers of AI responses are immediately followed by the student copying and pasting the response into their code. The study also showed that tools like these are widely accepted and appreciated by students and deemed useful according to a student survey - suggesting that the integration of AI tools can enhance learning outcomes and positively impact student engagement and interest in programming assignments.

This paper reports an analysis of incremental development: a process in computer science education where students code a little and run their code regularly making continual forward progress. We use a recently published measure of incremental development to evaluate reconstructed submissions collected from CS1 students using keystroke and IDE data. An element that was lacking in the original paper was an analysis of correlation of incremental development with academic outcomes which our study does. We mark submissions as incrementally developed or not and we then analyze the keystroke behaviors between these submissions and final assignment performance. We study if incremental development is correlated with grades, number of assignment interaction events, time spent on the assignment, number of coding sessions to complete the assignment, as well as other metrics. We present evidence that incremental development is correlated with time spent on the assignment, number of times the code was executed by the student, and similar measures. We do not find evidence that incremental development is correlated with assignment score or final grade in the course. Our analysis is consistent with the belief that incremental development will help students complete programming assignments more quickly but it does not support the idea that incremental development will result in better grades.

Emerging technologies like generative AI tools, including ChatGPT, are increasingly utilized in educational settings, offering innovative approaches to learning while simultaneously posing new challenges. This study employs a survey methodology to examine the policy landscape concerning these technologies, drawing insights from 102 high school principals and higher education provosts. Our results reveal a prominent policy gap: the majority of institutions lack specialized guide-lines for the ethical deployment of AI tools such as ChatGPT. Moreover,we observed that high schools are less inclined to work on policies than higher educational institutions. Where such policies do exist, they often overlook crucial issues, including student privacy and algorithmic transparency. Administrators overwhelmingly recognize the necessity of these policies, primarily to safeguard student safety and mitigate plagiarism risks. Our findings underscore the urgent need for flexible and iterative policy frameworks in educational contexts.