In this Viewpoint, we respond to Owen and Xu’s critique of our justice-oriented antiracist validity (JAV) framework, first advanced in “Our validity looks like justice. Does yours?” (Randall et al.). We argue that their response rests on a false dichotomy: that we must either repair educational measurement incrementally or abandon it altogether. JAV, we clarify, is not a replacement for validity theory but a critical extension of it, building explicitly on Kane’s interpretation-use argument and Mislevy’s sociocognitive foundations. We address four of Owen and Xu’s critiques. First, we take issue with their critique of QuantCrit, which confuses methodology with epistemology. As we explain, QuantCrit was never proposed as a new set of statistical techniques; it is a critical orientation toward the production, interpretation, and use of quantitative evidence. Second, we concur with Owen and Xu that a more expansive view of the history of testing is warranted. Unintended consequences of testing are often the result of a test’s long-term use, which ends up advantaging those with more power. Cultural systems of evaluation are always tied to systems of power, and in tracing the influence of assessment systems, we must always place them in their historical context to understand who is being advantaged and disadvantaged by them. Third, we concur that within the brief space of a Viewpoint article, it is impossible to provide a sustained treatment of Rawls’s contributions. Rawls alone is insufficient; for that reason, in our research we place his work within a broader conversation that includes Iris Marion Young, Charles W. Mills, Martha Nussbaum, bell hooks, Kimberlé Crenshaw, and Patricia Hill Collins. Finally, JAV was developed from a North American use case. Acknowledging that North America is not the only standpoint, we have worked with other researchers to develop and apply new models of principled design and validation across a range of contexts within and beyond the United States and Canada. JAV is not the totalizing project described by Owen and Xu. It is about procedures and consequences, a provocation made in service to those who are impacted by the work we do.
This study provides a bibliometric analysis of International English Language Testing System (IELTS) research from 1989 to 2024, incorporating 641 research documents. Among these, 482 were obtained from five online indices (Web of Science, Scopus, ERIC, EBSCOhost, Google Scholar) and 159 were identified manually (including 146 studies sponsored by the IELTS co-owners). The analysis focuses on patterns and trends in research topics, methodological approaches, publications and disciplinary areas, author-affiliated institutions and countries, and co-authorship. The results show that topics which were well-covered by researchers extended beyond psychometrics and validation of the testing system to include consequential factors, such as teaching and learning, academic/professional contexts, test preparation, and non-linguistic constructs (e.g., stakeholders’ attitudes, beliefs). Quantitative studies constituted the most common approach but were exceeded by (often well-cited) mixed-methods inquiries sponsored by the co-owners. Research authorship diversified from Anglophone to English as a Foreign Language (EFL)/English as a Second Language (ESL) contexts, reflecting a broader view of IELTS’s validity, warranting further exploration. Implications of the findings are discussed both for IELTS and language assessment research more generally.
This study examined how multiple factors, independent of research quality, influence the scientific impact of language assessment articles. A total of 447 papers published between 2018 and 2022 were investigated using a path analysis approach to identify both direct and indirect ways in which extrinsic factors affect citation counts. A conceptual path model was constructed based on existing literature and assessed using a comprehensive bibliometric dataset. The final model showed an excellent fit to the data (χ 2 = 65.6, df = 52, p = .097; standardized root mean square residual [SRMR] = .034, root mean square error of approximation [RMSEA] = .024, comparative fit index [CFI] = .980, Tucker–Lewis index [TLI] = .970) and accounted for 57.75% of the variance in citation counts ( R² = .5775, p < .001). Among nine examined factors, seven demonstrated significant direct or indirect effects on citation counts: number of self-citations, article age, journal CiteScore, number of countries, number of authors, open access, and research topic. Based on these findings, the study deepens our understanding of citation practices and offers broader implications for citation-based research evaluation in language assessment.
When different tests are used for the same purpose, score requirements should be comparable so that examinees cannot obtain an unfair advantage simply because of the test they chose. Drawing on our experience conducting a large-scale concordance study to allow for an empirical comparison of IELTS Academic and TOEFL iBT test scores, we review Knoch and Fan’s evaluative framework, explore methodological best practices and challenges, and offer future directions for score concordance research. We emphasize the importance of methodological rigor in collecting test-taker score data, transparency in analyzing such data to build score concordance tables, and a reasonable degree of construct comparability as a prerequisite for conducting a score concordance study, while also highlighting the limitations of concordance tables as standalone tools for admissions decisions. We note that some aspects of Knoch and Fan’s good practice principles are more straightforward to implement in practice than others. The good practice principles could be updated or adjusted after real-world application, which we describe with a view to furthering best practice in concordance research. We conclude this viewpoint with recommendations for decision-making that are based on fair score requirements, irrespective of which test the examinees chose.
Recent initiatives have sought to integrate dynamic assessment and diagnostic assessment to facilitate language learning. Extending this line of innovation, the present study uses the example of refusals to illustrate the development and validation of a computerized dynamic diagnostic assessment of pragmatic competence (CDDA-P). To enhance the authenticity of assessed pragmatic performance, a bottom-up approach was adopted in item design, in which response options were empirically derived from a corpus of productions by 335 language users. The effectiveness of the CDDA-P was examined from three key perspectives: its ability to diagnose learners' strengths and weaknesses, to identify their zones of proximal development (ZPDs) and learning potential, and to promote the development of learners' pragmatic performance. A pretest-immediate posttest-delayed posttest design was employed to track changes in 66 Chinese learners' performance before and after the implementation of the CDDA-P. Findings reveal that the CDDA-P can provide a fine-grained diagnosis of learners' strengths and weaknesses in performing L2 refusals and identify their diverse ZPDs and learning potential. Furthermore, learners demonstrated significant improvement after mediation. This study presents both theoretical and methodological implications for the development of integrated dynamic and diagnostic language assessment, while also offering insights into promoting L2 pragmatic competence through assessment.
How to optimize ordering the parts of a language test (e.g., items, tasks, and sections) is often overlooked or assumed to be self-evident. However, ordering choices are an essential component of test design as they may impact test takers’ objective performance, perceived performance, or affective states. In this paper, we report on a systematic review of test component ordering research, focusing on 88 studies from 1933 to 2023. We provide a narrative synthesis, describing typical outcome variables (e.g., test-taker performance), independent variables (e.g., order of difficulty), and mediating variables (e.g., language proficiency). Key findings, mostly from higher educational contexts, indicate that easy-to-hard ordering may lead to better performance, though these effects are mediated by various test-taker characteristics, especially anxiety and proficiency. While section ordering by language skills is common practice in language testing, there is scant empirical support for this approach, or for organizing tests by content or format. We discuss the implications of these findings for language test developers and suggest avenues for future research, particularly the need for more studies on ordering effects in language assessment contexts.
To support use of tests in academic contexts, it is critical to demonstrate that test scores and test performances are associated with performance in academic settings—an inferential link referred to as extrapolation in argument-based validation frameworks. TOEFL Essentials is a newer test designed to measure both general and academic English and is intended for use in higher education. The TOEFL Essentials speaking section consists of Virtual Interview, Read Aloud, and Listen & Repeat tasks, the latter two of which elicit highly constrained responses that may be less reflective of academic speaking tasks. In this study, we examined correlations of scores and linguistic characteristics across TOEFL Essentials speaking performances and (a) lab-based academic tasks (graph description, lecture response) for 149 students and (b) an authentic course-based speaking task for 65 students. Strong correlations (.65 < r < .80) were found between TOEFL Essentials speaking scores and evaluations of academic speaking. Among linguistic characteristics, fluency and accuracy variables demonstrated the largest and most consistent correlations across test and non-test tasks. Findings provide evidence relevant to the extrapolation of TOEFL Essentials speaking performances, which are based in part on highly constrained tasks, to academic settings and help inform decisions about test use.
A test taker may underperform for reasons not fully attributable to language proficiency, including psychological or contextual influences such as anxiety or illness. IELTS One Skill Retake (OSR) was launched in 2022, allowing test takers to retake, within 60 days, a single component in which their initial performance may have been affected by extenuating circumstances. This study examines outcomes associated with OSR by analysing test-taking patterns and score changes among over 20,000 OSR test takers from its launch through Spring 2024. It also reports survey findings from 578 OSR test takers on their experiences, including whether they achieved target scores and their perceived reasons for not achieving the desired score on the original full test. Across skills, average OSR component scores were higher than the corresponding scores on the original full test, and overall-band changes among OSR test takers were comparable to those observed among short-interval full-test repeaters (⩽60 days). Survey responses commonly cited factors such as insufficient preparation, stress and anxiety, and fatigue and lack of focus as perceived contributors to underperformance on the initial test. The findings contribute empirical evidence relevant to the interpretation and use of one-skill retake scores and to ongoing discussions of the validity and equity implications of retake policies in large-scale language testing.
This three-level meta-analysis investigated the human–machine correlation of automated speech evaluation (ASE) systems. Sixty-seven studies representing 392 effect sizes were included. The results indicated a positive overall correlation ( r = .654, p < .001) between machine and human scoring of speech. Pooled effect sizes across speaking constructs showed the highest correlation for delivery ( r = .784), followed by overall speaking proficiency ( r = .686), fluency ( r = .618), pronunciation ( r = .606), content ( r = .574), and grammar and vocabulary ( r = .499). A clear upward trend was observed in ASE studies, from the traditional machine learning stage ( r = .597), to the deep learning application stage ( r = .641), and further to the transformer-driven stage ( r = .680). Moderator analysis revealed significant moderating effects on the overall human–machine correlation from 10 variables: publication year, publication type, unit of sample, age group, level of task constraints, rater expertise, inter-rater reliability, system developer, feature engineering , and algorithm type . However, no significant moderating effects were observed for level of task integration, scoring method, system architecture , and automated speech recognition (ASR) accuracy . Key considerations for future ASE development are proposed, offering insights for educators and policymakers in integrating ASE into education.
This study presents a methodological framework for applying corpus linguistics to systematically evaluate the authenticity of chatbot production in relation to (spoken) production in a general target language use domain. We demonstrate the approach through data drawn from the development cycle of a low-stakes formative assessment system in which learners interact with a ChatGPT-powered bot. A Chatbot Corpus containing approx. 290,000 words from 600 simulations of target ChatGPT production was created, representing two GPT versions (3.5 and 4), and three temperature settings. This corpus was then compared with relevant subcorpora in the British National Corpus 2014, which contains 100 million words of British English collected in naturalistic settings. Analyses were conducted at macro- (multi-dimensional analysis), meso- (comparative frequency analysis), and micro-levels (occurrence of specific pragmatic feature analysis). Results showed that the ChatGPT-powered chatbot production was systematically more similar to genres of written rather than spoken communication: output demonstrated higher lexical density and was characterised by a relatively low occurrence of features typical of spoken communication such as stance and pragmatic markers. We argue that the methodological framework is applicable across different chatbot models, allowing researchers and developers to use this approach with newer, more refined AI-powered conversational agents in the future.
A key psychometric phenomenon in educational testing is differential item functioning (DIF), which evaluates whether specific test items function differently across subgroups of examinees who have the same level of the underlying (latent) ability. DIF happens when examinees with the same latent trait have different probabilities of correctly responding to a test item, influenced by their subgroup membership. This study aims to illustrate the application of the recursive partitioning Rasch tree model, shortly called the Rasch tree, to examine DIF in a large-scale German reading comprehension test for fifth-grade elementary students across gender, socioeconomic status, immigration status, personality disposition, and the need for cognition. Unlike the conventional DIF detection methods, the Rasch tree does not require a pre-specification of groups for exploring DIF, and continuous covariates can be easily included in the analysis. To investigate DIF of the test, item responses of 4,252 students were analyzed. The Rasch tree analysis generated nine non-predefined nodes, with slightly different patterns of item difficulties. Eleven items were flagged as exhibiting moderate and large DIF in the four splitting nodes. No splits were produced based on the need for cognition. The results indicated that the combination of the covariates impacted students’ test performance.
Language testing is both an evaluative practice and a commercial enterprise shaped by market forces. Within this context, test developers have a responsibility to ensure transparency with test users, particularly when scores inform high-stakes decisions. This Viewpoint contrasts transparency in communicating the truth about language tests with the persuasiveness of argument-based validity, noting that persuasiveness, although central to such arguments, is not equivalent to transparency. Two measures are proposed to strengthen transparency. First, test developers should publish test-form-specific validity reports detailing content, psychometric properties, and interpretation guidelines for each form. Second, they should clearly explain a test's limitations to the public, especially when scores are used in high-stakes settings, such as immigration or university admission, without adequate validation. The latter measure draws on regulatory norms in the pharmaceutical industry, where transparency can protect consumers from potential misuse. Specific steps are outlined to support these measures. Overall, these proposals aim to shift the emphasis from justification and persuasion toward transparency and align language testing practices more closely with open science principles.
This study investigates ChatGPT's performance as an Automated Writing Evaluation (AWE) system by comparing its scoring with that of human raters and examining learners' perceptions of its feedback. Six ChatGPT models were developed using different prompt configurations. Sixty English writing samples produced by Korean university English as a Foreign Language (EFL) learners were evaluated by two human raters and the six models. A multifaceted Rasch model, Spearman's correlation, and intraclass correlation were used to examine reliability, severity, and bias. Learners' perspectives on the models' feedback were collected through open-ended surveys and analyzed thematically. The results indicate that prompt design plays a central role in shaping ChatGPT's scoring behavior. Prompts combining Chain-of-Thought reasoning with Fill-in-the-blank scaffolding were associated with higher scoring consistency, while predefined personas and few-shot exemplars tended to moderate scoring severity. However, no stable patterns were observed for either bias or rating scale use, suggesting that prompt design alone cannot fully control domain-level bias. In particular, reasoning-intensive writing domains showed substantial divergence from human judgment, highlighting the need for human oversight. In parallel, learners generally viewed ChatGPT's feedback positively, while also noting areas for improvement. Overall, the study demonstrates the potential of prompt-calibrated ChatGPT-based AWE as a supplementary tool for writing assessment and instruction.
In most Western European countries, aspiring citizens must pass both a second language (L2) test and a knowledge of society (KoS) test, often administered in the L2. While several studies highlighted the social inequalities such tests may create, few have quantified these disparities. This study, anchored in the IMPECT project, analyses the test results of 79,794 migrants in Norway to explore the implications of these requirements, particularly for adult L2 learners with limited formal education in their first language (L1). Logistic regression analyses reveal that (1) learners with limited formal L1 education have significantly lower probabilities of passing the tests compared to those with higher education; (2) disparities are greater for KoS tests than for the oral A2 language test; (3) L2 scores strongly predict the likelihood of passing KoS tests, substantiating the assumption that KoS tests are implicit language tests; and (4) the impact of language skills on passing KoS tests varies by schooling level, with L2 scores being a stronger predictor for learners with limited education than for those with tertiary degrees. These findings highlight barriers faced by learners with limited L1 schooling in the citizenship acquisition process and underscore the need for targeted policy and support mechanisms.
Previous research has established the essential role of aural lexical knowledge (ALK) in second language (L2) listening comprehension; however, from a measurement perspective, it remains underexplored how best to measure ALK and which word frequency bands show the strongest predictive power for L2 listening comprehension. This study examines the effectiveness of three item formats in an aural vocabulary test (Yes/No, first language meaning-recall, or meaning-recognition) in predicting variance in L2 listening comprehension among 176 English as a Foreign Language (EFL) learners. Following a vocabulary levels test (VLT) format, 300 words representing the first 5000 flemmas of a television and movie transcript-based word frequency list were tested across the three item formats. The results indicated that the meaning-recall VLT was significantly more difficult than the Yes/No and meaning-recognition formats. Overall scores for all three tests significantly correlated with scores on a standardized L2 English listening comprehension test (TOEIC (R)), with the strongest correlations observed for the meaning-recall (r = .63) and recognition formats (r = .65). The vocabulary bands that provide significant predictive power across each format are identified, and implications for language teachers and vocabulary knowledge testing are discussed.
While ethical and moral philosophy underpins core concepts in language testing, few publications in the field engage with moral philosophy in a sustained or systematic way. A notable exception is the International Language Testing Association's (ILTA) Code of Ethics (COE), ratified in 2000, which continues to guide professional practice. As ILTA initiates a revision of the COE, this paper examines the objectives and philosophical foundations of the original COE. Drawing on the document itself and key writings by its principal author, Alan Davies, and contemporary authors, we offer an analysis that reveals both the strategic rationale behind the Code and the unresolved tensions within it. Our aim is to deepen the field's understanding of the ethical frameworks that have shaped the language testing profession as ILTA traces a way forward in revising the COE.
