While predictive models are widely used in learning analytics, several studies have shown that the performance of these models can exhibit biases against different demographic groups of students. The first step to audit for and mitigate these group biases is to accurately estimate them. However, the current practices for identifying and measuring group bias often suffer from reliability issues. In this paper, we use simulations and real-world data analysis to explore statistical factors that impact the reliable estimate of group bias and suggest approaches to improve their statistical robustness. Our analysis revealed that small group sizes lead to high variability in group bias estimation due to sampling error—an issue that is more likely to impact students from historically marginalized communities. We then suggest statistical approaches, such as bootstrapping, to construct confidence intervals for a more reliable estimation of group bias. Based on our findings, we encourage future learning analytics researchers to ensure sufficiently large group sizes, construct confidence intervals rather than relying on p-values, use at least two metrics, and move beyond the dichotomy of the presence or absence of bias for a more comprehensive evaluation of group bias.
Professional learning environments offer a unique opportunity for exploring professional competencies due to the use of workplace data, which captures important aspects of professional performance. This data is different from that used in formal learning analytics contexts, as it is not collected by systems designed to understand learning. Instead, workplace data represents the digital fingerprint professionals leave behind when interacting with technologies to do their jobs. Harnessing this data to support workplace learning is challenging for a range of reasons, including being able to identify meaningful metrics to identify individual performance and scaffold the use of this data to support professional learning interventions on knowledge and performance. Generative AI (GenAI) has great potential to enhance professional learning analytics by addressing some of the challenges inherent in using workplace data. These challenges include needing to process large amounts of unstructured data to understand individual performance and transform this data into interfaces and interventions that can support learning. In our paper, we modify Clow’s learning analytics cycle to inform a modified framework describing the intersection of learning analytics with professional learning. Subsequently, we illustrate the potential power of GenAI for supporting professional learning across the framework through three case studies in health professions education.
Online discussion forums are a key data source in STEM education for analyzing student interaction, knowledge construction, and social learning. From a learning analytics perspective, understanding the temporal dynamics of instructional behaviours, for example, the differences between instructors and peers, can offer valuable insights into how instructional support unfolds in online discussions. However, few studies have examined these interactions with attention to their sequential and temporal organization. This study adopts a learning analytics approach to investigate the behavioural dynamics of instructional support in a large-scale online math discussion forum. We analyzed 83,569 posts from Math Nation, a widely used online learning platform supporting K–12 math education in the United States. Instructional behaviours were automatically coded based on a theoretically grounded coding scheme using transformer-based language models, and their temporal patterns were modelled using multilevel vector autoregression. Network visualizations were employed to illustrate the dynamic relationships among instructional strategies. Our findings show that peers posted more frequently than instructors and were more likely to engage in acknowledgement and feedback behaviours. When examining behaviour proportions and temporal transitions within discussion threads, peer interactions were associated with more diverse behavioural sequences, whereas instructor interactions exhibited more focused and directive patterns. In mixed-participant threads, instructors often responded following peer contributions, which was associated with a reduced likelihood of subsequent direct peer interventions. Rather than making claims about instructional effectiveness or learning outcomes, this work contributes to the field of learning analytics by demonstrating how natural language processing (NLP)-based behaviour modelling can be integrated with temporal interaction analysis to characterize role-based instructional support dynamics in asynchronous, discussion-based STEM learning environments.
Assessment is foundational to learning analytics, especially in evaluating instructional interventions and guiding improvement in online learning environments. With the growing use of large language models (LLMs) to score open-ended responses, questions arise about the reliability of these model-generated scores, particularly in short pre-post formats where learners are expected to improve. This study introduces a novel method for estimating test reliability that adjusts for learning gains using a Rasch-based split-half approach. We validated this approach through simulation under realistic conditions of missing data and score change, showing tangible improvements in reliability estimation compared to baseline methods. Applying this method to a dataset of 985 tutors completing 12 online lessons, we find that GPT-4-based scoring achieves satisfactory reliability, with open-ended responses (0.733) outperforming multiple-choice items (0.652). Both item types jointly yielded the highest reliability (0.774). Hence, as few as 14 open-ended items (across an average of 3-4 completed lessons) were sufficient to surpass common reliability thresholds of 0.7 or higher. Principal component analysis revealed a skill structure with a strong primary dimension shared across almost all lessons and interpretable subdimensions—socio-emotional, cognitive, and fairness-related tutoring skills—supporting a bifactor-like model. These findings demonstrate that GPT-4 and similar LLMs can be effectively used for formative assessment of complex instructional skills in online and personalized learning contexts, provided their reliability is empirically verified. This study contributes an open-source, learning-aware framework for scalable and reliable AI-supported assessment in learning analytics contexts.
This study expands the concept of curricular complexity as defined in the Curricular Analytics framework (Heileman et al., 2018) by providing qualitative evidence of the suitability of three new metrics, concerning timing of course offerings, extended time-to-degree, and credit loss, that more adequately address curricular challenges encountered by transfer students. Curricular Analytics is a method for analyzing a curriculum that enables practitioners and researchers to quantify and systematically analyze the impacts of course sequencing in a plan of study on student outcomes. However, the original conceptualization falls short of capturing the substantive challenges faced by transfer students who enter an undergraduate program at various points in the curricular sequence. This study was guided by the following research question: “How do three new measures of curricular complexity (i.e., inflexibility factor, transfer delay factor, and credit loss) align with transfer professionals’ perceptions of curricular barriers for transfer students?” Using a grounded theory approach, we conducted seven focus groups with 38 transfer professionals across the United States. We presented these transfer experts with each new measure and prompted them to reflect on its validity based on their experiences supporting transfer students. We found transfer professionals resonated strongly with all three new metrics, suggesting strong initial construct and content validity.
