Background Large language models (LLMs) have shown potential not only as content generators but as evaluators capable of providing nuanced feedback. However, much of the current application of LLMs in education treats them as static graders rather than dynamic participants in formative assessment processes. Purpose This study explores how rubric-guided prompting and role-aware feedback simulations can enable LLMs to approximate human evaluative reasoning across dimensions critical to design-based learning. Scope/Method Using situated learning theory, iterative design pedagogy, and cognitive models of scientific and engineering thinking, the research developed a framework wherein LLMs were trained to align with expert judgment. A stratified sample of student design artifacts was evaluated across different roles (instructor, peer reviewer, grant reviewer) using targeted prompting. Feedback outputs were coded for tone and evaluation focus. Results Rubric engineering was found to substantially improve LLM-human agreement in cognitively complex categories. LLMs demonstrated role-sensitive feedback variation, and final rubric-tuned LLM ratings achieved high consistency with human ratings (Cronbach’s Alpha > 0.75). Figures and tables illustrate how role-specific emphasis and tone were reliably modulated. Conclusions When properly scaffolded, LLMs can serve as dynamic co-evaluators and rubric co-design partners. These findings advance the use of AI from automation to pedagogical emulation, offering scalable, reflective feedback ecosystems for design-rich learning environments.
While generative artificial intelligence (GenAI) is increasingly used in language education, empirical evidence about how educators perceive, adopt, and learn to integrate these tools remains fragmented. This systematic review examines language educators’ adaptation to GenAI technologies through an analysis of 23 empirical studies published in SSCI-indexed journals between December 2022 and September 2024. Using PRSIMA guidelines, we examined educators across pre-service, in-service, and higher education contexts to address four questions: their perceptions of GenAI, factors influencing adoption, competency gaps, and professional development (PD). Across contexts, educators expressed cautious, selective adoption: GenAI was valued for lesson planning, materials creation, and writing support, yet classroom-facing use lagged due to concerns about academic integrity, role redefinition, and technostress. Reported competency gaps spanned episteme (limited understanding of AI capabilities/limits), techne (prompting, AI-enhanced task/assessment design, content evaluation), and phronesis (ethical judgment, bias/privacy, adaptive expertise). Only three studies reported structured PD, but each signaled gains in knowledge, confidence, and identity reframing. Findings underscore the need to pair technical skill-building with practical wisdom to enable effective, ethical GenAI integration in language education.
Managing blended learning environments, which combine traditional face-to-face and online learning, can be challenging for teachers as it requires adapting and orchestrating both components effectively. Learning analytics dashboards (LAD) can provide teachers with insights into students’ self-studying habits in the online component. While recent advances in machine learning (ML) enable the identification of meaningful behavioral patterns, existing LADs mostly focus on providing aggregated information. Reasons for this are manifold, including the potential lack of trust in ML processes and the intricate nature of their visualizations. In this work, we investigate how to make ML findings accessible to teachers using a teacher-centered, mixed-method approach. We first design multiple visualizations and assess their perceived clarity, appeal, and actionability in a user study with 100 teachers. We then implement these visualizations in a dashboard to monitor students’ self-regulated learning behavior and adapt it to two different learning contexts: Reflective Writing and Flipped Classrooms. We evaluate the effectiveness and applicability of our dashboard through semi-structured interviews with 19 teachers. Our findings suggest that the visualization preferences, requirements, uses, and concerns of LADs differ considerably between the two contexts. Our study contributes to understanding teachers’ design preferences in LADs and the integration of ML-derived insights into classrooms.
In recent years, knowledge tracing has been widely studied in the field of educational data mining as a means of accurately modeling students’ learning processes. Knowledge tracing is the task of modeling a student’s current knowledge state based on their previous learning history. This knowledge state is eventually used to predict the students’ upcoming performances for a knowledge concept. Recently, Deep Neural Network based models using a spectrum of data features and prediction strategies like Attention Mechanism, Q-Matrices have been employed for this purpose. Our novel approach to knowledge tracing introduces a new perspective in university-level engineering programs that follow the Outcome-Based Education (OBE) system. By leveraging the OBE concept ‘Affinity mappings’, we ensure the interconnectedness of knowledge concepts across the entire curriculum. With this unique feature, we introduce the Outcome-Based Knowledge Tracing (OKT) model, a Recurrent Neural Network that tracks students’ knowledge states based on their interactions with course and program outcomes. This technique quantifies the inherent relationships between outcomes, providing more clear understanding of how different knowledge pieces link and influence learning. We enhance OKT with Memory Augmented Neural Networks (MANN), which allows to analyze the specific contribution of each outcome to student knowledge progress. Our model achieves an impressive 89.81% AUC on a live Learning Management System, outperforming the common baseline RNN models such as DKT, DKVMN and SimpleKT and EKT.
Auto-marking, also referred to as auto-scoring or auto-grading, is the automated assessment of student responses using computational methods. Applying auto-marking to short answer questions in science education has long posed significant challenges due to the domain-specific knowledge required and the diverse ways students express understanding. The emergence of transformer-based models from 2017 onwards represents a potential breakthrough in addressing these challenges. This scoping review explored the performance of early transformer-based auto-markers during this foundational period (2017-2024), providing insights that remain relevant as the technology continues to evolve.We followed a systematic process, adhering to relevant PRISMA guidelines. Our review included literature from 2017 to early 2024, focusing on English natural language responses on science content in an educational assessment context. A final set of 21 articles was reviewed and coded in depth to answer our research questions which explored the types of auto-marking models being used, the datasets used to fine-tune and test them, and their performance.The most commonly used models in this context were BERT models and BERT variants, which increased in frequency reaching a peak in 2021. After 2021, papers using GPT models started to appear. The SciEntsBank dataset was the most commonly used to test auto-markers but several other datasets (e.g., ASAP SAS, Beetle) also featured in our review. BERT models generally performed better than previous models on the SciEntsBank dataset. As of yet, GPT models have not been evaluated on SciEntsBank but there was one study in the review that directly compared GPT-3.5 and BERT base and found that GPT-3.5 outperformed BERT base across different items and item types. The review also shows that models that utilise additional forms of data like textbooks and marking rubrics seem to consistently outperform models without these, and that auto-markers may present issues in terms of low reliability, lack of explainability and bias.By examining auto-markers in this foundational period, this review identifies enduring patterns, critical gaps, and methodological challenges that will inform the development of future auto-marking systems regardless of specific model architectures.
Calculated Formula Questions (CFQs) are a prevalent and critical assignment type in engineering courses to help students practice solving real-world problems. However, providing timely and personalized feedback on CFQ assignments remains challenging in large classrooms. Recent development of artificial intelligence (AI) offers unprecedented opportunities to deliver timely feedback through empowering intelligent tutoring systems (ITSs). Nevertheless, existing efforts have been constrained to assignments with readily available structured digital data, creating a gap in supporting unstructured CFQs. This study introduces a life-cycle framework that enables the development of an AI-powered ITS for CFQs, from data curation and model training to student-facing system deployment. Using graded CFQ assignments from undergraduate Engineering Economics courses as a case study, we built a digitalized dataset and applied a novel random masking technique to augment small-scale and imbalanced data. For each CFQ, we trained an eXtreme Gradient Boosting (XGBoost) model as its AI backbone. The model functions to predict potential mistakes in the solution using only each student’s submitted numerical answers, bypassing access to full written solutions. Our experiments demonstrate the feasibility of AI models in solution diagnosis, achieving an average precision of 0.81, a recall of 0.79, and an accuracy of 0.65 in predicting mistakes. To balance feedback efficiency and accuracy, we implemented a dialogue-based interaction scheme within a student-facing web interface. This scheme adaptively gathers additional inputs from students when the AI model’s predictions have close probabilities. Together, the AI backbone models and the web interface form an AI-powered ITS (Arthur) that delivers real-time and personalized feedback. Our framework offers a scalable pathway for building AI-powered ITS across engineering courses.
This study investigates the pedagogical impact of an AI-enhanced critical media literacy program implemented with fourth-grade students in a public primary school. Using an explanatory sequential mixed-methods design, the research combined quantitative data from the Critical Media Literacy Scale for Primary Students with qualitative insights from interviews, student artifacts, and classroom observations. The 18-hour intervention followed the 5E instructional model and integrated generative AI tools such as ChatGPT and Grammarly. Aligned with the national Turkish and Social Studies curricula, the program ensured curricular coherence and contextual relevance. Quantitative findings revealed significant gains in students’ media literacy skills, particularly in reading and writing. Qualitative data highlighted developments in digital safety, source verification, online ethics, and media critique. Students also showed enhanced awareness of ethical digital behavior and reflective media use. These results underscore the potential of AI-supported instruction to foster critical thinking, digital citizenship, and media literacy in early education settings.
Estimating item difficulty through field-testing is often resource-intensive and time-consuming. As such, there is strong motivation to develop methods that can predict item difficulty at scale using only the item content. Large Language Models (LLMs) represent a new frontier for this goal. The present research examines the feasibility of using an LLM to predict item difficulty for K-5 mathematics and reading assessment items (N = 5170). Two estimation approaches were implemented: (a) a direct estimation method that prompted the LLM to assign a single difficulty rating to each item, and (b) a feature-based strategy where the LLM extracted multiple cognitive and linguistic features, which were then used in ensemble tree-based models (random forests and gradient boosting) to predict difficulty. Overall, direct LLM estimates showed moderate to strong correlations with true item difficulties. However, their accuracy varied by grade level, often performing worse for early grades. In contrast, the feature-based method yielded stronger predictive accuracy, with correlations as high as r = 0.87 and lower error estimates compared to both direct LLM predictions and baseline regressors. These findings highlight the promise of LLMs in streamlining item development and reducing reliance on extensive field testing and underscore the importance of structured feature extraction. We provide a seven-step workflow for testing professionals who would want to implement a similar item difficulty estimation approach with their item pool.
Artificial Intelligence (AI), in particular the novel possibilities supplied by GPT-4, is increasingly being utilized in educational settings, yet its impact on student activation and learning outcomes remains controversial. We conduct a lab-in-the-field experiment in undergraduate tutorial classes in macroeconomics at the university level. Over the course of one semester, we asked students to answer eight open-ended questions. Depending on the treatment, students received lecturer feedback at the classroom level only (LF), additional individual feedback from their peers (PF), or from the AI (AIF). We find that AIF has a significant positive effect on student activation. Compared to LF, continuous participation in the tasks is significantly higher in AIF, and it induces the longest written answers across the eight tasks. In terms of learning outcomes, AIF stands out for producing the most significant improvement in content. For the style of answers, we find no effects. We attribute the larger effects of AIF compared to PF to the higher reliability and quality of feedback provision of the AI. Our findings demonstrate that AI, specifically GPT-4, can offer a scalable and consistent solution for providing individual feedback in educational settings.
The integration of new literature into the English curriculum remains a challenge since educators often lack scalable tools to rapidly evaluate readability and adapt texts for diverse classroom needs. This study proposes to address this gap through a multimodal approach that combines transformer-based text classification with linguistic feature analysis to align texts with UK Key Stages. Eight state-of-the-art Transformers were fine-tuned on segmented text data, with BERT achieving the highest unimodal F1 score of 0.75. In parallel, 500 deep neural network topologies were searched for the classification of linguistic characteristics, achieving an F1 score of 0.392. The fusion of these modalities shows a significant improvement, with every multimodal approach outperforming all unimodal models. In particular, the ELECTRA Transformer fused with the neural network achieved an F1 score of 0.996. Unimodal and multimodal approaches are shown to have statistically significant differences in all validation metrics (accuracy, precision, recall, F1 score) except for inference time. The proposed approach is finally encapsulated in a stakeholder-facing web application, providing non-technical stakeholder access to real-time insights on text complexity, reading difficulty, curriculum alignment, and recommendations for learning age range. The application empowers data-driven decision making and reduces manual workload by integrating AI-based recommendations into lesson planning for English literature.
With the increasing integration of Artificial Intelligence (AI) in our lives, including education, it is necessary to consider and address the ethical questions raised by this process. While the ethics of AI is widely researched, little attention has been paid to the ethics of AI in education (AIED). This systematic literature review aims to identify the main ethical values (EVs) and ethical norms (ENs) for AIED in literature published before 2010 and available in English. Using database search and backward snowballing in November 2022, 25 articles were included and analysed. In order to identify the EVs, the definitions found in literature were collected and reported. Thematic analysis and grouping were performed based on common terms. It was found that there are six main EVs for AIED: non-discrimination, data stewardship, human oversight, goodwill, explicability, and educational aptness. The ENs found in the literature were grouped as per the stakeholder sets that they were relevant for and per main EV. Following this, these two groupings were combined into a matrix with ENs for stakeholder sets to follow in order to implement specific main EVs. Identifying the main components of ethics of AIED is an initial step that can pave the way for future research aimed at creating ethical frameworks or regulation to ethically guide the domain.
In highly immersive, team-based healthcare simulations, students must collaborate effectively to complete open-ended learning tasks. These often require students to move among multiple locations within the physical learning space, requiring the development of effective communication tactics to manage complex team dynamics and multiple concurrent dialogue segments. Analysing such distributed team dialogue using only audio data poses significant challenges as it is hard to distinguish how teams temporarily break into smaller groups. This study investigates the integration of audio and spatial data to model embodied team communication tactics in highly-dynamic healthcare simulation scenarios. We employ ordered network analysis (ONA) to examine the order of communication behaviours across coded co-located dialogue segments of teams of nursing students. We also explore the potential of ONA to enhance teaching through a qualitative assessment of its usability with teachers. Results reveal that high-performing teams employ effective team tactics by prioritising primary tasks (i.e., deteriorating patients) and performing critical tasks timely and correctly, while low-performing teams struggle to develop prioritisation tactics (i.e., focusing on stable patients) and with team coordination. Interviews with teachers found high-level information in ONA networks interpretable but faced challenges with sequential details and contextualisation. This research offers insights into improving team-based learning and assessment in healthcare education.
Responding to national calls to integrate generative AI into mathematics teacher education, this study developed Student ChatGPT, a custom chatbot designed as an interactive role-playing simulation for preservice secondary mathematics teachers (PSMTs). Specifically, Student GPT simulated a middle school student with misconceptions in ratio reasoning, providing opportunities for PSMTs for practice-based teaching experience. Through qualitative analysis of interactions between PSMTs and Student ChatGPT, an analytic framework was developed focusing on three domains: affective, communicative, and technical, to assess the behavior, strengths, and weaknesses of Student GPT. The results showed that the Student GPT demonstrates strengths in positive expression, clarity, relevance, error types, knowledge, and consistency. These advantages suggest that Student GPT can serve as a practice-based training tool to help PSMTs deepen their pedagogical content knowledge for teaching. However, limitations were noted in negative expression, language, acquisition, and role confusion. Despite these limitations, this study proposed that Student GPT is a valuable tool offering PSMTs low-risk, personalized, and interactive teaching experiences resembling real teacher-student interactions. Implications for mathematics teaching and directions for future research were discussed.
Students often misjudge their understanding of learning material, which can lead to the use of ineffective learning strategies and result in suboptimal learning outcomes. However, it remains unclear how misjudgments relate to the use of metacognitive strategies in online learning settings, which is essential context for developing effective interventions that support students in making (and using) accurate judgments of their performance. To address this, we analyze data from 210 college students using a computer-based learning environment, investigating the relationships among calibration discrepancy, judgments, and strategies, as well as the factors affecting shifts in metacognitive judgments during learning. Students who overestimated their pretest retrospective judgments engaged less in metacognitive strategies, particularly in preparatory actions before quizzes (b = -9.100, p < .001). Notably, pretest retrospective judgments—rather than actual pretest scores—significantly predicted students’ engagement in these metacognitive strategies (b = -9.841, p < .001). Furthermore, increased engagement in repeated quiz-taking was a significant negative predictor of changes in metacognitive judgments (b = -1.792, p = .036), indicating that students engaging in repeated quizzes tended to adjust their judgments more conservatively. These results highlight the role of pretest retrospective judgments in shaping engagement with metacognitive strategies, underscoring the importance of correcting early calibration discrepancies. Our findings advocate for early, proactive metacognitive support tools that go beyond merely presenting information, offering guidance on interpreting feedback, and implementing strategies to better align students’ judgments with their actual performance.
Intelligent Tutoring Systems (ITS) for psychomotor skills provide an accessible, scalable, and efficient solution, compared to tutors. Despite successes in the past, the research progress seems stagnating. Part of the reasons can be due to developing ITS for a very specific skill or task. In consequence, skills and the applications of those in their entirety are not presented. Therefore, we conducted a systematic literature review, and examined existing ITS using Harrow’s taxonomy, a skill continua framework, and dimensions of psychomotor skill learning. We observed a lack in consideration of offering different tasks to promote skill proficiency. Skills supported by ITS are majorly fine, closed, internally paced, discrete, individual, and simple skills. ITS focus majorly on technical, thus coordination aspects of motor abilities. Feedback and repetition are key methods to promote psychomotor skill learning. There is potential considering other physical activities to promote skill proficiency. Similarly, it might be worth exploring ITS for skills that, for example, are gross, and open. Integrating tasks that target motor abilities, such as strength or flexibility, can be part of it. The integration of theories in ITS from related research fields, such as training periodization, can be investigated.
Integrating generative artificial intelligence tools like ChatGPT and GitHub Copilot into programming education presents notable opportunities to enhance learning but also raises critical challenges in balancing innovation with preserving foundational programming skills. Through a systematic review of 40 empirical studies guided by PRISMA 2020 and Kitchenham’s methodologies, this study evaluates how effectively GenAI was incorporated into programming education and its impact on preserving higher-order thinking skills and foundational programming logic. The Findings reveal that successful integration hinges on intentional teaching strategies, thoughtfully designed assessments, and structured integration processes. However, barriers such as GenAI tools’ limited accessibility features, insufficient bias mitigation, and a tendency to prioritize tool availability over curriculum alignment often disrupt seamless adoption. Additionally, the potential for students to become over-reliant on AI risks diminishing higher-order thinking skills and programming logic. To address these gaps, this paper proposes the GenAI-Ped framework, a structured approach combining self-regulated learning, universal design principles, and iterative feedback to harmonize GenAI-driven support with skill development. The findings emphasized the necessity of strategic implementation, educator training, and inclusive practices to maximize GenAI’s potential in programming education.
Theoretical models of self-regulated learning emphasize the role of metacognitive monitoring and regulation in deploying accurate cognitive, affective, metacognitive, motivational, and social processes to improve learning outcomes. Many of these models identify metacognition as a dynamic, recursive, conditional, and contextual process. However, empirical work has largely neglected how past metacognitive judgments influence future judgments. Furthermore, prior work has primarily focused on describing what learners did during learning but has not advanced toward predicting future behavior on an individual level. Our work addresses this gap by comparing three modeling approaches to predict the accuracy of ease-of-learning (EOL) and retrospective confidence judgments (RCJs) from 55 undergraduates based on the accuracy of previous metacognitive judgments as participants learned about the human circulatory system using Metatutor-IVH a multimedia environment. We found that when fit on the individual level, the most frequently fit models were random-walk models, followed by ARIMAs, and then moving average models. Additionally, we observed consistency in the types of model fits across judgment types within individuals. When examining the accuracy in the predictions our individually fit models, ARIMA models outperformed the others in predicting RCJs. Moreover, they achieved higher accuracy that the traditional group-level models in which one model was trained on the entire sample and then fit to each individual, that our individual approach outperformed the traditional method. Finally, individuals who showed autoregressive tendencies in their EOL judgment accuracies outperformed their peers. Overall, our results make a strong argument for an individualized time-series modeling approach over traditional aggregate measures. Additionally, we found evidence that future EOL judgment accuracy may be temporally dependent on past judgments, and those that show autoregressive behaviors may be indicative of more efficient learning and metacognitive monitoring. We conclude by discussing the theoretical and practical implications for designing intelligent adaptive learning environments.
