Numerous studies demonstrate the importance of self-regulation during learning by problem-solving. Recent work in learning analytics has largely examined students' use of SRL concerning overall learning gains. Limited research has related SRL to in-the-moment performance differences among learners. The present study investigates SRL behaviors in relationship to learners' moment-by-moment performance while working with intelligent tutoring systems for stoichiometry chemistry. We demonstrate the feasibility of labeling SRL behaviors based on AI-generated think-aloud transcripts, identifying the presence or absence of four SRL categories (processing information, planning, enacting, and realizing errors) in each utterance. Using the SRL codes, we conducted regression analyses to examine how the use of SRL in terms of presence, frequency, cyclical characteristics, and recency relate to student performance on subsequent steps in multi-step problems. A model considering students' SRL cycle characteristics outperformed a model only using in-the-moment SRL assessment. In line with theoretical predictions, students' actions during earlier, process-heavy stages of SRL cycles exhibited lower moment-by-moment correctness during problem-solving than later SRL cycle stages. We discuss system re-design opportunities to add SRL support during stages of processing and paths forward for using machine learning to speed research depending on the assessment of SRL based on transcription of think-aloud data.

In today's data-driven educational technologies, algorithms have a pivotal impact on student experiences and outcomes. Therefore, it is critical to take steps to minimize biases, to avoid perpetuating or exacerbating inequalities. In this paper, we investigate the degree to which algorithmic biases are present in two learning analytics models: knowledge estimates based on Bayesian Knowledge Tracing (BKT) and carelessness detectors. Using data from a learning platform used across the United States at scale, we explore algorithmic bias following three different approaches: 1) analyzing the performance of the models on every demographic group in the sample, 2) comparing performance across intersectional groups of these demographics, and 3) investigating whether the models trained using specific groups can be transferred to demographics that were not observed during the training process. Our experimental results show that the performance of these models is close to equal across all the demographic and intersectional groups. These findings establish the feasibility of validating educational algorithms for intersectional groups and indicate that these algorithms can be fairly used for diverse students at scale.

Digital learning games have been increasingly adopted in classrooms to facilitate learning and to promote learning outcomes. Contrary to common beliefs, many digital learning games can be more effective for female students than male students in terms of learning and affective outcomes. However, the in-game learning mechanisms that explain these differences remain unclear. In the current study, we re-analyze three retrospective data sets drawn from three studies conducted in different years. These data sets, which involved 213, 197, and 287 students, were collected from a digital learning game that teaches late elementary and middle school students decimal concepts. We re-analyzed these data sets to understand how female and male students differ in the rates of gaming the system, a behavioral measure that reflects a form of disengagement while playing the game. Rates of gaming the system are compared between female and male students within each of the game’s two core instructional activities (i.e. problem-solving and self-explanation) as well as tested in a game vs. non-game condition. We found that female students game the system significantly less than male students in the self-explanation step in the game condition, in all three studies. This difference in the rates of gaming mediates the relationship between gender and learning outcomes, a pattern in which female students tend to learn more than male students, across all three studies. These results suggest that future design iterations of the game could focus on reducing gaming behaviors for male students, which might improve learning outcomes for female students as well. Understanding gender-based differences in game behaviors can inform future game design to promote better learning outcomes for all students.

When students procrastinate on programming assignments, it can hinder the quality of their code and negatively impact their grades. In contrast, when students actively delay working on assignments to prepare to code (e.g., reading or seeking help), it can be an effective self-regulated learning (SRL) strategy beneficial to programming performance. However, distinguishing active delay from procrastination is methodologically challenging. To address this, we tracked what students did when they behaviorally delayed starting an assignment. Most students prepared to code by using multiple course resources across programming assignments. We found that many students delayed starting to code by seeking help in the Q&A platform, and this was beneficial to the quality of their code. Also, some pre-coding activities were related to behavioral delay in starting to code, but benefitted students' grades, and thus may indicate active delay, but not all pre-coding activities were beneficial. By considering pre-coding activities, we gain a comprehensive view of students' approach to coding in CS education.

Programming courses can be challenging for first year university students, especially for those without prior coding experience. Students initially struggle with code syntax, but as more advanced topics are introduced across a semester, the difficulty in learning to program shifts to learning computational thinking (e.g., debugging strategies). This study examined the relationships between students' rate of programming errors and their grades on two exams. Using an online integrated development environment, data were collected from 280 students in a Java programming course. The course had two parts. The first focused on introductory procedural programming and culminated with exam 1, while the second part covered more complex topics and object-oriented programming and ended with exam 2. To measure students' programming abilities, 51095 code snapshots were collected from students while they completed assignments that were autograded based on unit tests. Compiler and runtime errors were extracted from the snapshots, and three measures -- Error Count, Error Quotient and Repeated Error Density -- were explored to identify the best measure explaining variability in exam grades. Models utilizing Error Quotient outperformed the models using the other two measures, in terms of the explained variability in grades and Bayesian Information Criterion. Compiler errors were significant predictors of exam 1 grades but not exam 2 grades; only runtime errors significantly predicted exam 2 grades. The findings indicate that leveraging Error Quotient with multiple error types (compiler and runtime) may be a better measure of students' introductory programming abilities, though still not explaining most of the observed variability.

Ethical issues matter for artificial intelligence in education (AIED). Simultaneously, there is a gap between fundamental ethical critiques of AIED research goals and research practices doing ethical good. This article discusses the divide between AIED ethics (i.e., critical social science lenses) and ethical AIED (i.e., methodologies to achieve ethical goals). This discussion contributes paths toward informing AIED research through its fundamental critiques, including improving researcher reflexivity in developing AIED tools, describing desirable futures for AIED through co-design with marginalized voices, and evaluation methods that merge quantitative measurement of ethical soundness with co-design methods. Prioritizing a synthesis between AIED ethics and ethical AIED could make our research community more resilient in the face of rapidly advancing technology and artificial intelligence, threatening public interest and trust in AIED systems. Overall, the discussion concludes that prioritizing collaboration with marginalized stakeholders for designing AIED systems while critically examining our definitions of representation and fairness will likely strengthen our research community.

Think-aloud protocols are a common method to study self-regulated learning (SRL) during learning by problem-solving. Previous studies have manually transcribed and coded students' verbalizations, labeling the presence or absence of SRL strategies and then examined these SRL codes in relation to learning. However, the coding process is difficult to scale, as it is time-consuming and laborious. This aspect potentially limits the ability to measure SRL comprehensively on a larger scale. Recent advancements in language models offer the potential to infer SRL from automated think-aloud transcriptions, which could enhance the efficiency of SRL measurement, complementing log data-based approaches to studying SRL. Therefore, this study explores the possibility of leveraging large language models (LLMs) and machine learning to automatically detect SRL in machine-transcribed student think-aloud transcripts. Specifically, we experimented with two LLMs (Universal Sentence Encoders and OpenAI’s text-embedding-3-small) to predict four SRL categories (processing information, planning, enacting, and realizing errors) in students' verbalizations, collected from three intelligent tutoring systems, covering stoichiometry chemistry and formal logic. We found that these models are reliable at predicting the SRL categories, with AUC scores ranging from 0.696 to 0.915. Models that use embeddings from the text-embedding-3-short model performed significantly better at predicting SRL, including transfer from open-ended to highly scaffolded ITS systems. However, we note limitations in transferring models from the chemistry to logic domain, potentially due to the differences in domain-specific vocabulary. We discuss the practical implications of these models, highlighting the opportunity to analyze think-aloud transcripts at scale to facilitate future SRL research.

In qualitative data analysis, codebooks offer a systematic framework for establishing shared interpretations of themes and patterns. While the utility of codebooks is well-established in educational research, the manual process of developing and refining codes that emerge bottom-up from data presents a challenge in terms of time, effort, and potential for human error. This paper explores the potentially transformative role that could be played by Large Language Models (LLMs), specifically ChatGPT (GPT-4), in addressing these challenges by automating aspects of the codebook development process. We compare four approaches to codebook development – a fully manual approach, a fully automated approach, and two approaches that leverage ChatGPT within specific steps of the codebook development process. We do so in the context of studying transcripts from math tutoring lessons. The resultant four codebooks were evaluated in terms of whether the codes could reliably be applied to data by human coders, in terms of the human-rated quality of codes and codebooks, and whether different approaches yielded similar or overlapping codes. The results show that approaches that automate early stages of codebook development take less time to complete overall. Hybrid approaches (whether GPT participates early or late in the process) produce codebooks that can be applied more reliably and were rated as better quality by humans. Hybrid approaches and a fully human approach produce similar codebooks; the fully automated approach was an outlier. Findings indicate that ChatGPT can be valuable for improving qualitative codebooks for use in AIED research, but human participation is still essential.

Understanding indicators in self-regulated learning (SRL) that affect mathematical success using quantitative techniques such as epistemic networks hold potential for providing effective scaffolds that draw directly from the learner’s perspective. Tied to learning success, SRL provides a range of frameworks for identifying students' affective, cognitive, and metacognitive performance in a computer-based learning environment. This research can investigate how ENA can contribute as a visualization device to understanding of the metacognitive aspect of math learning. With the aim, we collected text responses from an online math problem-solving environment that encouraged reflections on self-regulated learning patterns that differ by the rate of correctness and familiarity with the educational tool. Student responses consisted of their explanations of strategies and solutions after the scaffolding instructions. Our team deductively designed detectors reflecting on assembling and translating operations (Winne’s SMART model) to examine differences in the learner’s self-regulated learning behaviors. We then leveraged Epistemic Network Analysis (ENA) using these detected indicators as codes to compare the results within two categories: performance on correctness and familiarity developed over time. Models show stronger co-occurrence between numerical representation and contextual representation and highlight the critical impact of outcome orientation on learner success. When the final answer is correct, or learners are more familiar with the educational tool, there is a strong outcome orientation connected to contextual representation within SRL operations.

Pedagogical agents offer significant promise for engaging students in learning. In this paper, we investigate students’ conversational interactions with a pedagogical agent in a game-based learning environment for middle school science education. We utilize word embeddings of student-agent conversations along with features distilled from students’ in-game actions to induce predictive models of student engagement. An evaluation of the models’ accuracy and early prediction performance indicates that features derived from students’ conversations with the pedagogical agent yield the highest accuracy for predicting student engagement. Results also show that combining student problem-solving features and conversation features yields higher performance than a problem solving-only feature set. Overall, the findings suggest that student-agent conversations can greatly enhance student models for game-based learning environments.

This study uses Knowledge Engineering (KE) to develop an automated model of problem-solving strategy invention (PSSI) behavior (defined as inventing a new strategy for solving a math problem, outside of system-offered default strategies). The PSSI model identified the students inventing new strategies, and examined the relationship between PSSI behavior and existing fine-grained detectors of self-regulation. The findings suggest that students inventing new strategies to use for problems, are more likely to transform the information provided in the question, and to reason around the problem’s contextual information.

In past work, time management interventions involving prompts, alerts, and planning tools have successfully nudged students in online courses, leading to higher engagement and improved performance. However, few studies have investigated the effectiveness of these interventions over time, understanding if the effectiveness maintains or changes based on dosage (i.e., how often an intervention is provided). In the current study, we conducted a randomized controlled trial to test if the effect of a time management intervention changes over repeated use. Students at an online computer science course were randomly assigned to receive interventions based on two schedules (i.e., high-dosage vs. low-dosage). We ran a two-way mixed ANOVA, comparing students' assignment start time and performance across several weeks. Unexpectedly, we did not find a significant main effect from the use of the intervention, nor was there an interaction effect between the use of the intervention and week of the course.

Peer review, a commonly-used pedagogy in contemporary education has been found to positively influence student learning, benefitting both feedback provider and recipient.However, the quality of the feedback may vary, and lower-quality feedback (e.g., lacking specificity), is less likely to be implemented by the recipient, leading to suboptimal outcomes.Although recent work has used criteria to scaffold feedback to ensure quality, it is often difficult to monitor whether students follow these criteria.In this study, we develop models that automatically detect the attributes of student feedback, reflecting the presence of three pedagogically relevant constructs: 1) commenting on the process, 2) commenting on the answer, and 3) relating to self.We find models employing sentence embeddings produce the best results, with AUC ROCs ranging from .90-.96, and are robust to algorithmic bias.

The effects of educational anxiety have been observed across multiple disciplines; anxiety negatively influences cognition, self-regulation, performance, and educational outcomes.However, there has been limited research on anxiety within the context of interactive learning environments.In the current research, we expand this by assessing whether and how trait-level anxiety (assessed as a pre-and post-measure in a year-long study) is related to students' self-regulated learning strategies, behaviors, belief, and achievement in the context of an open-ended math problem-solving platform, called CueThink.Results indicate that anxiety is negatively related to key constructs involving math achievement.Altogether, our findings generally imply that students with higher anxiety may avoid interacting with their stressors, in this case, math content, effectively contributing to poorer outcomes.We discuss our findings within the context of research and pedagogical and system design.

Self-regulated learning (SRL) is a critical component of mathematics problem solving. Students skilled in SRL are more likely to effectively set goals, search for information, and direct their attention and cognitive process so that they align their efforts with their objectives. An influential framework for SRL, the SMART model, proposes that five cognitive operations (i.e., searching, monitoring, assembling, rehearsing, and translating) play a key role in SRL. However, these categories encompass a wide range of behaviors, making measurement challenging – often involving observing individual students and recording their think-aloud activities or asking students to complete labor-intensive tagging activities as they work. In the current study, we develop machine-learned indicators of SMART operations, in order to achieve better scalability than other measurement approaches. We analyzed student’s textual responses and interaction data collected from a mathematical learning platform where students are asked to thoroughly explain their solutions and are scaffolded in communicating their problem-solving process to their peers and teachers. We built detectors of four indicators of SMART operations (namely, assembling and translating operations). Our detectors are found to be reliable and generalizable, with AUC ROCs ranging from .76-.89. When applied to the full test set, the detectors are robust against algorithmic bias, performing well across different student populations.

The effort to learn and the regulation of learning are key to successful learning. Voluntary practice has been shown to improve learning and is associated with having generally good self-regulated learning. At the same time, procrastination often slows the learning process and is associated with less than ideal regulation of learning. In this paper, we present the results of a study exploring the impact of voluntary practice and procrastination on the learning outcomes of novice programmers. We used data from an introductory programming course (CS1) at a large university and found that most students engaged in voluntary practice. However, students with higher prior performance and non-procrastinators were more likely to participate in the voluntary practice. We also found that participating in the voluntary practice did not have a significant impact on course performance. Furthermore, the study showed a weak negative correlation between procrastination and time spent on the homework and a weak negative correlation between procrastination and distributed practice. Finally, we found that non-procrastinators performed significantly better than procrastinators on the majority of homeworks.

As a cognitive and affective state, interest promotes engagement, facilitates self-regulated learning, and is positively associated with learning outcomes. Research has shown that interest interacts with prior knowledge, but few studies have investigated these issues in the context of adaptive game-based learning environments. Using three subscales from the User Engagement Scale, we examine data from middle school students (N = 77) who interacted with Crystal Island in their regular science class to explore the relationship between interest, knowledge, and learning. We found that interest is significantly related to performance (both knowledge assessment and game completion), suggesting that students with high interest are likely to perform better academically, but also be more engaged in the in-game objectives. These findings have implications both for designers who seek to identify students with lower interest and for those who hope to create adaptive supports.

Previous studies on the accuracy of knowledge tracing models have typically considered the performance of all student actions. However, this practice ignores the difference between students’ initial and later attempts on the same skill. To be effective for uses such as mastery learning, a knowledge tracing model should be able to infer student knowledge and performance on a skill after the student has practiced that skill a few times. However, a model’s initial performance prediction – on the first attempt at a new skill – has a different meaning. It indicates how successful a model is at inferring student performance on a skill from both their performance on other skills and from the difficulty and other properties of the first item the student encounters. As such, it may be relevant to differentiate prediction in these two contexts when evaluating a knowledge tracing model. In this paper, we describe model performance at a more granular level and examine the consistency of model performance across the number of student instances on a given skill. Results from our research show that much of the difference in performance between classic algorithms such as BKT (Bayesian Knowledge Tracing) and PFA (Performance Factors Analysis), as compared to a modern algorithm such as DKVMN (Dynamic Key-Value Memory Networks), comes down to the first attempts of a skill. Model performance is much more comparable by the time the student reaches their third attempt at a skill. Thus, while there are many benefits to using contemporary knowledge tracing algorithms, they may not be as different as previously thought in terms of mastery learning.