Writing assessments, including curriculum-based measurement (CBM), face feasibility challenges because manual scoring becomes time- and resource-intensive when large numbers of writing samples must be evaluated. Automated scoring offers a promising complement, and rapid advances in generative artificial intelligence (AI) further expand its potential. However, empirical evidence on automated scoring of brief CBM writing samples remains limited, and potential scoring bias remains underexplored. We evaluated automated CBM scores generated by a feature-based system (writeAlizer) and a generative AI system (Gemini) using three-minute passage-level CBM Story Prompt responses produced by elementary students identified as having intensive early writing needs. We examined (a) consistency and agreement between automated scores and trained raters’ hand-calculated scores, (b) criterion-related validity in relation to a standardized writing assessment, and (c) subgroup bias. Overall, scoring performance varied across CBM metrics, with stronger agreement observed for simpler metrics. Automated and hand-calculated scores demonstrated comparable criterion-related validity, though correlations with the standardized measure were modest. No systematic differences in scoring accuracy were found across student race/ethnicity, sex, or English Learner status. Findings highlight the potential of automated scoring and the need for continued research before it can be recommended for scoring brief CBM writing samples in beginning writers.
AI chatbots are increasingly used as writing aids and feedback providers; however, their classroom implementation remains limited due to inconsistent feedback accuracy, highlighting the need for teachers to act as feedback curators who refine and contextualize AI-generated feedback. However, no study has systematically examined the scope, nature, and focal areas of teachers’ refinements of AI-generated feedback. The present study documents teacher interventions using color-coded annotations and structured review forms, examines teachers’ evaluations of feedback accuracy through accuracy ratings, and explores their perceptions of classroom usability through interviews. The findings indicated that AI frequently overlooked meaning-level problems and instead focused on surface-level errors; in densely errored papers, major meaning-related issues often remained unresolved. While error-coded feedback was dominated by surface-level corrections, neglecting clarity and structural concerns, content feedback showed limited ability to detect repetitive ideas and organizational flaws, which were readily identified by teachers. Consequently, teachers described the filtering process of error-coded feedback as burdensome, whereas content feedback was perceived as more useful and easier to enhance through teacher-led prioritization adjustments addressing students’ actual needs. The findings highlighted the key areas of teacher refinements and offer insights for improving the accuracy and classroom applicability of AI-generated feedback through teacher prioritization strategies.
Recent research has highlighted the importance of peer feedback literacy in academic writing, yet its longitudinal development among graduate students remains underexplored. This longitudinal multiple-case study traced the development of peer feedback literacy among four Chinese graduate students working in two fixed provider–receiver dyads across two peer feedback cycles. Drawing on drafts, peer comments, multimodal interaction records, and interviews, the analysis identified cross-case commonalities and role-related variation. The findings revealed a shared three-stage trajectory, moving from initial constraints on productive participation in peer feedback, through the recalibration of feedback orientations and practices, to more coordinated, strategic, and collaborative feedback practices. Within this trajectory, development took different forms across cases: one receiver showed growing confidence in evaluating feedback, whereas the other developed greater trust in the feedback relationship; among the providers, one produced increasingly explicit and actionable feedback, whereas the other adopted a more adaptive and relationally sensitive evaluative stance. The findings suggest that peer feedback literacy is a role-differentiated and socially co-constructed configuration of capacities, dispositions, and practices shaped by affective, relational, and interactional conditions. They offer pedagogical implications for creating emotionally safe peer-feedback environments, scaffolding feedback provision, and structuring opportunities for digitally mediated dialogue.
The formative value of written feedback (WF) depends partly on whether learners can interpret and apply metalinguistic information, yet feedback language remains underexamined in EFL contexts. This quantitative-priority mixed-methods quasi-experimental study compared revision uptake, subsequent target-feature accuracy, and learner perceptions across English-only WF and pedagogical translanguaging written feedback (PT-WF). Seventy Saudi EFL university students at CEFR B1 completed the pre-test and post-test, and 55 completed the four-week delayed post-test. Sixty-two participants provided questionnaire ratings, and eight joined two focus groups. Both conditions received focused WF on subject–verb agreement, articles, and cohesive markers. Every PT-WF marginal comment combined Arabic and English, whereas English-only WF provided the same feedback types entirely in English. Revision uptake did not differ between conditions for subject–verb agreement or articles. However, cohesive-marker uptake under PT-WF (M = 0.86) was more than double that under English-only WF (M = 0.41). Longitudinally, total errors declined under PT-WF and remained lower at the delayed post-test, whereas they increased under English-only WF. Participants rated PT-WF as easier to understand and apply, although some preferred English-only WF. The findings position feedback language as a formative-assessment design variable whose value may depend on the linguistic and interpretive demands of the target feature.
This study examined rater scoring in L2 writing assessment under two conditions of onscreen marking: scanned handwritten scripts and word-processed versions of the same scripts. Fifty-three raters marked both versions of 30 scripts presented to them on a computer screen, using an analytical rating scale. A cross-classified multilevel modelling approach was employed to analyse both the overall score and four subscores. The findings indicated that the word-processed scripts were, on average, scored significantly lower than their handwritten counterparts by 0.05–0.11 SDs. Given that the desired mode effect is 0 SD, these differences could be considered substantial in high-stakes examination contexts. Lower scores for word-processed scripts were not attributable to specific scripts being penalised, but rather reflected variability in individual rater scoring behaviour. There was significant variation among raters in the degree to which they scored word-processed scripts more severely than handwritten scripts across both the overall and four subscale scores, with the grammar criterion showing the widest spread in rater-specific mode effects. Practical implications of the findings are discussed in relation to rating conditions, rater training, and automated scoring.
