Rendering understanding as a general causal account is problematic. This leads us toward accusations of partial understanding, or illusory understanding, characterized by understanding that lacks depth. The discourse comprehension literature offers an alternative: rendering understanding as the construction of a situation model, a model that integrates knowledge, purpose, and context. On this account, the question is never whether understanding is complete, but whether it is adequate for what it aims to achieve.

We conducted a ManyClasses study in 30 different classes, where student participants (n = 1,571) were assigned to watch two online lecture videos, and one of these videos (order randomized) was immediately preceded by ungraded multiple-choice prequestions, with no feedback or answers shown. In the laboratory, decades of research have observed that prequestions reliably improve learning from subsequent instruction when compared to instruction not preceded by prequestions, but evidence in authentic education settings has been elusive. The current experiment, embedded in authentic classes ranging from 6th grade through senior-level college, with class-specific videos and questions created by teachers, confirmed that prequestions improve average student learning performance, measured by delayed classroom assessments that included the prequestions. Further, we examined how prequestions affected students' interactions with the online videos. Despite the generalizable average benefit of prequestions for student learning, we found that prequestions also caused some students to disengage, skipping the assigned video entirely. Among students who did watch the videos, there was no increase in the amount viewed after answering prequestions compared to videos with no prequestions. These findings lead us to suggest that prequestions do not increase attention to the subsequent learning materials, but rather, prequestions cause learners to activate a relevant mental model in advance of instruction. This account is corroborated by the finding that answering prequestions correctly was a significant moderator of the benefits of prequestions, suggesting that the prequestions, while beneficial on average, may cause a rich-get-richer scenario when implemented in authentic education settings.

When making comparisons between groups of students, a common technique is to analyze whether there are statistically significant differences between the means of each group. This convention, however, is problematic when data are negatively skewed and bounded against a performance ceiling, features that are typical of data in education settings. In such a situation, we might be particularly interested to observe group differences in the left tail, specifically among students who have room to improve, and conventional analyses of group means have limitations for detecting such differences. In this article, an alternative to these conventions is presented. Rather than comparing the means of two groups, we can instead compare how closely student data are concentrated toward the modes of each group. Bayesian methods provide an ideal framework for this kind of analysis because they enable us to make flexible comparisons between parameter estimates in custom analytical models. A Bayesian approach for examining concentration toward the mode is outlined and then demonstrated using public data from a previously reported classroom experiment. Using only the outcome data from this prior experiment, the proposed method observes a credible difference in concentration between groups, whereas conventional tests show no significant overall differences between group means. The present article underscores the limitations of conventional statistical assumptions and hypotheses, especially in school psychology and related fields, and offers a method for making more flexible comparisons in the concentration of data between groups.

For researchers seeking to improve education, a common goal is to identify teaching practices that have causal benefits in classroom settings. To test whether an instructional practice exerts a causal influence on an outcome measure, the most straightforward and compelling method is to conduct an experiment. While experimentation is common in laboratory studies of learning, experimentation is increasingly rare in classroom settings, and to date, researchers have argued it is prohibitively expensive and difficult to conduct experiments on education in situ. To address this challenge, we present Terracotta (Tool for Education Research with RAndomized COnTrolled TriAls), an open-source web application that integrates with a learning management system to provide a comprehensive experimental research platform within an online class site. Terracotta automates randomization, informed consent, experimental manipulation of different versions of learning activities, and export of de-identified research data. Here we describe these features, and the results of a live classroom demonstration study using Terracotta, a preregistered replication of McDaniel et al. (Journal of Applied Research in Memory and Cognition, 1(1), 18–26, 2012). Using Terracotta, we experimentally manipulated online review assignments so that consenting students alternated, on a weekly basis, between taking multiple-choice quizzes (retrieval practice) and reading answers to these quizzes (restudy). Students' performance on subsequent exams was significantly improved for items that had been in retrieval practice review assignments. This successful replication demonstrates that Terracotta can be used to experimentally manipulate consequential aspects of students’ experiences in education settings.

A common activity in learning analytics research is to demonstrate a new analytical technique by applying it to data from a single course. We explore whether the value of an analytical approach might generalize across course contexts. Accordingly, we conduct a conceptual replication of a well-cited temporal modeling study using self-regulated learning (SRL) taxonomies. We attempt to conceptually replicate this previous work through the analysis of 411 students across 19 courses' trace event data. Using established SRL categorizations, learner actions are sequenced to identify regular clusters of interaction through hierarchical clustering methods. These clusters are then compared with the entire data corpus and each other through the development of first-order Markov models to develop process maps. Our findings indicate that, although some general patterns of SRL can generalize, these results are more limited at higher scales. Comparing these clusters of interaction along students' performance in courses also indicates some relationships between activity and outcomes, though this finding is also limited in relation to the complexity introduced by scaling out these methods. We discuss how these temporal models should be viewed when making descriptive and qualitative inferences about students' activity in digital learning environments.

Participants performed a categorization training task, where each trial presented an example scenario in which an individual makes a claim based on an observation, and participants marked which fallacy or bias, if any, the individual in the scenario was committing. In two studies, we measure the effect of this training task on critical thinking, measured using an open-ended critical thinking assessment, both pre- and post-training. In Study 1, we pilot these materials in an online college course across a full academic semester and observe credible improvements in critical thinking performance. In Study 2, we conduct a pre-registered randomized controlled experiment using online research participants and observe credible improvements in critical thinking relative to no training, and relative to comparable learning activities focused on conventional curricular content. We infer that the categorization training task facilitated inductive learning of patterns of biased and flawed reasoning, which improved participants’ ability to detect and identify such patterns in the delayed open-ended critical thinking assessment. Such categorization training shows promise as an effective and practical method for improving learners’ resistance to online disinformation.

Immediate feedback has been considered a cornerstone of online language learning platforms. However, a closer reading of relevant research reveals that the definition of the term "immediate feedback" is inconsistent. Furthermore, findings from the STEM literature have not been well supported by other fields. As a result, clarification is required to assess which type of immediate feedback improves students' performance in a computer-assisted learning environment. Moreover, research on the effects of immediate feedback outside of STEM classes should provide an enhanced understanding of whether the findings can be generalized. Therefore, this study investigated the effects of immediate feedback timing in online language learning exercises. The following three conditions were examined: no feedback, end-of-question feedback, and end-of-assignment feedback. A planned contrast test revealed that with a pretest as the covariate, students in the end-of-question feedback condition received significantly higher grades in the posttest compared with those in the end-ofassignment feedback condition. Furthermore, students with lower pretest scores required more attempts, although their learning progress was not significantly superior to that of students with higher prior knowledge. This study's findings provide insights into the use of immediate feedback for improving learning as part of foreign language classroom instruction.

Learning analytics defines itself with a focus on data from learners and learning environments, with corresponding goals of understanding and optimizing student learning. In this regard, learning analytics research, ideally, should be characterized by studies that make use of data from learners engaged in education systems, should measure student learning, and should make efforts to intervene and improve these learning environments. However, a common concern among members of the learning analytics research community is that these standards are not being met. In two analysis waves, we review a large and comprehensive sample of research articles from the proceedings of the three most recent Learning Analytics and Knowledge conferences, the premier conference venue for learning analytics research, and from articles published during the same time in the Journal of Learning Analytics (over the years of 2020, 2021, and 2022). We find that 37.4% of articles do not analyze data from learners in an education system, 71.1% do not include any measure of learning, and 89.0% of articles do not attempt to intervene in the learning environment. We contrast these findings with the stated definition of learning analytics and infer, like others before us, that scholarship in learning analytics research presently lacks clear direction toward its stated goals. We invite critical discussion of these findings from the learning analytics community, through open peer commentary.

In this commentary we present an analogy between Johann Wolfgang Von Goethe’s classic poem, The Sorcerer’s Apprentice, and institutional learning analytics. In doing so, we hope to provoke institutions with a simple heuristic when considering their learning analytics initiatives. They might ask themselves, “Are we behaving like the sorcerer’s apprentice?” This would be characterized by initiatives lacking faculty involvement, and we argue that when initiatives fit this pattern, they also lack consideration of their potential hazards, and are likely to fail. We join others in advocating for institutions to, instead, create ecosystems that enable faculty leadership in institutional learning analytics efforts.

Learning analytics defines itself with a focus on learner data, with corresponding goals of understanding and optimizing student learning. In this regard, learning analytics research, ideally, should be characterized by studies that make use of data from learners engaged in education systems, should measure student learning, and should make efforts to intervene and improve these learning environments. However, a common concern among members of the learning analytics research community is that these standards are not being met. In the current study, we review a large and comprehensive sample of research articles from the proceedings of two recent Learning Analytics and Knowledge conferences, the premier conference venue for learning analytics research. We find that 36.3% of articles do not analyze data from learners in a formal education system, 70.5% do not include any measure of learning, and 91.4% of articles do not attempt to intervene in the learning environment. We contrast these findings with the stated definition of learning analytics, and infer, like others, that scholarship in learning analytics research presently lacks clear direction toward its stated goals.

While a variety of learning technologies are presently available to facilitate student-to-student peer interactions and collaborative learning online, recent research suggests that students' opportunities to interact with their peers were significantly reduced following the abrupt transition to remote instruction due to coronavirus disease.This raises concerns because peer interaction is known to be a key ingredient in effective online learning environments, and during remote instruction, the primary connection between a student and their identity as a member of a college community would have been online courses.In this study, we investigate whether and how collaborative technologies supported peer interaction, and students' learning, during remote instruction.Specifically, we used results from a multicampus survey of students and instructors, as well as data from our online learning management system, to explore the use of collaborative tools at a large scale and their associations with student outcomes.Findings indicate that instructors, as was typical before the pandemic, generally favored individual learning activities over collaborative activities during campus closure.But in those situations where collaborative activities were present during remote instruction, triangulation analyses indicate that their use was related to improved performance as measured by instructors' survey responses, by students' performance in their courses, and by an increased sense of belonging among students.

Emphasizing the predictive success and practical utility of psychological science is an admirable goal but it will require a substantive shift in how we design research. Applied research often assumes that findings are transferable to all practices, insensitive to variation between implementations. We describe efforts to quantify and close this practice-to-practice gap in education research.

Considering that the central theme of learning analytics is data, there are uniquely strong benefits to sharing data, materials, and analysis code in this field. These open research practices facilitate transparency in research, and transparency can improve the quality of learning analytics methods, improve the reliability of findings, and improve the scope of impact of new developments. The Society for Learning Analytics research recently adopted a motion to advance on these goals. To inform this effort, we conducted a systematic review of articles published in the past two proceedings of the Learning Analytics and Knowledge conference, measuring the frequency of openly-shared data, materials, and analysis code. We find that 7.5% of articles made their data available, 13.7% made materials available, and 5.5% made analysis code available. We discuss these findings in the context of possible barriers to open practices, and suggest that the principal barrier to improved transparency is researcher’s reluctance to share, rather than privacy or legal constraints.

Similar to how attention can enhance or suppress a visual target in a specific location in space, attention can also tune the perception of a visual event at a specific moment in time. One way to observe this temporal tuning is to present an auditory stimulus aligned with a visual target event. In general, past research reported that a co-occurring auditory stimulus decreases the amplitude of the N1 component of the visual event-related potential (ERP), possibly suppressing bottom-up attention evoked by a visual target. However, an independent line of research found that an expected auditory stimulus, such as one embedded in a rhythmic pattern, increases N1 amplitude to a coincident visual target event. Here, we present a novel preregistered demonstration of this dissociation using the flash-lag paradigm. Specifically, we observe opposite effects of rhythmic (expected) and non-rhythmic (unexpected) sounds on the visual ERP to a coincident flash. Results suggest that expectation for an upcoming sound reverses the sound’s typical suppression of bottom-up visual attention. We discuss these findings in the context of the multisensory tuning of attention.

In visual search tasks, physically large target stimuli are more easily identified among small distractors than are small targets among large distractors. The present study extends this finding by presenting preliminary evidence of a new search asymmetry: stimuli that symbolically represent larger magnitude are identified more easily among featurally equivalent distractors that represent smaller magnitude. Participants performed a visual search task using line-segment digits representing the numbers 2 and 5, and the numbers 6 and 9, as well as comparable non-numeric control stimuli. In three experiments, we found that search times are faster when the target is a digit that represents a larger magnitude than the distractor, although this pattern was not evident in one additional experiment. The results provide suggestive evidence that the magnitude of a number symbol can affect perceptual comparisons between number symbols, and that the semantic meaning of a target stimulus can systematically affect visual search.

Positive feedback has known benefits for improving task performance, but it is not clear why. On the one hand, positive feedback may direct attention to the task and one’s motivations for having performed the task. On the other hand, positive feedback may direct attention to the task’s value for achieving a future goal. This ambiguity presents a challenge for the design of automated feedback interventions. Specifically, it is unclear whether positive feedback will more effectively influence behavior when it praises the recipient for having performed an action, or when it highlights the action’s value toward a goal. In the present study, we test these competing approaches in a large-scale field experiment (n = 1,766). Using a mobile app, we assigned college students to receive occasional notifications immediately upon submitting online assignments that either praised them for having submitted their coursework, or that highlighted the value of submitting coursework for academic success, or to a no-treatment control group. We find that only praise messages improved submission rates and course performance, suggesting that drawing attention to the feedback-eliciting task is necessary and sufficient for influencing behavior at scale.

As institutions of higher education increasingly utilize online learning management systems, college students are asked to submit more assignments online. Under this regime, when most assignments are posted and submitted online, it is possible to know if a student is missing a submission for an imminent deadline, and to intervene proactively to reduce missed assignments and improve student outcomes. Toward this goal, we designed and evaluated a scalable targeted intervention: a mobile app that would deploy push notifications when students were missing submissions for assignments with imminent deadlines. Results from two experimental pilots demonstrate that this intervention system significantly decreased missed assignments compared with control notifications about instructor announcements to the class (in Experiment 1), and improved assignment adherence and course grades compared with courses that were not using the app (in Experiment 2). In this article, we discuss the benefits and theoretical implications of this behavioral guide rail, a purely informative proactive intervention to mitigate risk in advance of a negative outcome.

Under normal circumstances, when students invest more effort in their schoolwork, they generally show evidence of improved academic achievement. But when universities abruptly transitioned to remote instruction in Spring 2020, instructors assigned rapidly-prepared online learning activities, disrupting the normal relationship between effort and outcomes. In this study, we examine this relationship using data observed from a large-scale survey of undergraduate students, from logs of student activity in the online learning management system, and from students’ estimated cumulative performance in their courses (n = 4,636). We find that there was a general increase in the number of assignments that students were expected to complete following the transition to remote instruction, and that students who spent more time and reported more effort carrying out this coursework generally had lower course performance and reported feeling less successful. We infer that instructors, under pressure to rapidly put their course materials online, modified their courses to include online busywork that did not constitute meaningful learning activities, which had a detrimental effect on student outcomes at scale. These findings are discussed in contrast with other situations when increased engagement does not necessarily lead to improved learning outcomes, and in comparison with the broader relationship between effort and academic achievement.