Assessing source-based writing is challenging for raters due to the complexity of language usage and source integration (Plakans & Ohta, 2021). During the rating process, raters draw upon their prior experience, but despite a substantial body of research (Barkaoui, 2010a; Lim, 2011), no consensus has been reached regarding the impact of experience on assessment quality. This explanatory mixed-methods study compares rating performance of six novice and four experienced raters, who rated text products from integrated writing tasks using an analytic rating scale. The quantitative strand examines differences between the groups in terms of interrater agreement, severity, and halo effect. For the qualitative strand, think-aloud protocol and focus group interview data were collected. A set of qualitative variables that could help to explain the quantitative findings was identified based on prior research (e.g., Crisp, 2008; Zhang, 2016; Wolfe et al., 1998). Concerning interrater agreement and severity, the study results indicate greater variance between individual raters than between groups. In the case of the halo effect, we found that experienced raters were more susceptible to this effect than novices. The findings enhance our understanding of the role of rater experience and rater training for scoring approaches and rating quality in the context of integrated writing assessment.
Second language (L2) writing performance is shaped not only by linguistic proficiency but also by cognitive, motivational, and affective learner-related factors. Although individual differences have long been acknowledged in second language acquisition research, relatively limited attention has been given to how these dimensions jointly interact within task-based L2 writing assessment contexts. Addressing this gap, the present study examined the relationships among perceived cognitive load, academic writing motivation, writing anxiety, and task-based L2 writing performance within a Task-Based Language Assessment (TBLA) framework. The participants consisted of 326 undergraduate EFL students enrolled in academic writing courses at a university in Türkiye. Participants completed three self-report instruments measuring perceived cognitive load, academic writing motivation, and writing anxiety, in addition to two task-based writing assessments. The data were analyzed using Pearson correlation analyses and structural equation modeling (SEM). The findings revealed that perceived cognitive load and academic writing motivation were positively associated with task-based L2 writing performance, whereas writing anxiety demonstrated a negative relationship with writing performance. The findings support a multidimensional understanding of task-based L2 writing performance by highlighting the interactive roles of cognitive, motivational, and affective learner-related variables within assessment-oriented writing contexts.
Writing assessments, including curriculum-based measurement (CBM), face feasibility challenges because manual scoring becomes time- and resource-intensive when large numbers of writing samples must be evaluated. Automated scoring offers a promising complement, and rapid advances in generative artificial intelligence (AI) further expand its potential. However, empirical evidence on automated scoring of brief CBM writing samples remains limited, and potential scoring bias remains underexplored. We evaluated automated CBM scores generated by a feature-based system (writeAlizer) and a generative AI system (Gemini) using three-minute passage-level CBM Story Prompt responses produced by elementary students identified as having intensive early writing needs. We examined (a) consistency and agreement between automated scores and trained raters’ hand-calculated scores, (b) criterion-related validity in relation to a standardized writing assessment, and (c) subgroup bias. Overall, scoring performance varied across CBM metrics, with stronger agreement observed for simpler metrics. Automated and hand-calculated scores demonstrated comparable criterion-related validity, though correlations with the standardized measure were modest. No systematic differences in scoring accuracy were found across student race/ethnicity, sex, or English Learner status. Findings highlight the potential of automated scoring and the need for continued research before it can be recommended for scoring brief CBM writing samples in beginning writers.
AI chatbots are increasingly used as writing aids and feedback providers; however, their classroom implementation remains limited due to inconsistent feedback accuracy, highlighting the need for teachers to act as feedback curators who refine and contextualize AI-generated feedback. However, no study has systematically examined the scope, nature, and focal areas of teachers’ refinements of AI-generated feedback. The present study documents teacher interventions using color-coded annotations and structured review forms, examines teachers’ evaluations of feedback accuracy through accuracy ratings, and explores their perceptions of classroom usability through interviews. The findings indicated that AI frequently overlooked meaning-level problems and instead focused on surface-level errors; in densely errored papers, major meaning-related issues often remained unresolved. While error-coded feedback was dominated by surface-level corrections, neglecting clarity and structural concerns, content feedback showed limited ability to detect repetitive ideas and organizational flaws, which were readily identified by teachers. Consequently, teachers described the filtering process of error-coded feedback as burdensome, whereas content feedback was perceived as more useful and easier to enhance through teacher-led prioritization adjustments addressing students’ actual needs. The findings highlighted the key areas of teacher refinements and offer insights for improving the accuracy and classroom applicability of AI-generated feedback through teacher prioritization strategies.
Recent research has highlighted the importance of peer feedback literacy in academic writing, yet its longitudinal development among graduate students remains underexplored. This longitudinal multiple-case study traced the development of peer feedback literacy among four Chinese graduate students working in two fixed provider–receiver dyads across two peer feedback cycles. Drawing on drafts, peer comments, multimodal interaction records, and interviews, the analysis identified cross-case commonalities and role-related variation. The findings revealed a shared three-stage trajectory, moving from initial constraints on productive participation in peer feedback, through the recalibration of feedback orientations and practices, to more coordinated, strategic, and collaborative feedback practices. Within this trajectory, development took different forms across cases: one receiver showed growing confidence in evaluating feedback, whereas the other developed greater trust in the feedback relationship; among the providers, one produced increasingly explicit and actionable feedback, whereas the other adopted a more adaptive and relationally sensitive evaluative stance. The findings suggest that peer feedback literacy is a role-differentiated and socially co-constructed configuration of capacities, dispositions, and practices shaped by affective, relational, and interactional conditions. They offer pedagogical implications for creating emotionally safe peer-feedback environments, scaffolding feedback provision, and structuring opportunities for digitally mediated dialogue.
