The answer-until-correct (AUC) method is an alternative response format for multiple-choice questions. It allows respondents to answer repeatedly until they provide the correct answer. The number of attempts required to find a solution can be used as a measure of partial knowledge. For educational purposes, AUC is useful because it provides the correct answer to each question immediately after the response is given. Previous studies, which used mainly correlational designs, found that the AUC procedure improved the reliability of knowledge tests; however, the results regarding its validity were inconclusive. We report two studies in which we experimentally manipulated participants' knowledge to provide a robust external criterion for validation. In the first study, conducted with primary and secondary school children, we obtained more valid test scores using a paper-and-pencil version of the AUC procedure compared to a parallel multiple-choice format. In a second online study, we replicated this finding and found that the improvement in validity depended on how well the available distractors represented varying degrees of partial knowledge. Taken together, these findings offer the first experimentally grounded validation-using a strong external criterion-that the AUC format, especially when items are well constructed, can improve validity relative to the traditional MC format.
Textual analyses have grown increasingly common as an approach to quantify the high-dimensional information in documents such as reflective essays, customer reviews and survey responses. Sentiment analysis offers a way to capture the emotion present in these texts, which can then be used in statistical analyses to assess change. The current study applies a sentence-level sentiment analysis approach to two cohorts of pre-service teachers with the goal of understanding what, if any, changes in emotion are observed in their reflective essays. We investigate the extent to which sentiment scores calculated at the sentence level provide evidence of differences across time and across two cohorts of teachers. We find that while there were significant differences in certain emotion dimensions, the limited variability in the distribution of sentiment scores raises important challenges and implications for quantitative analyses of sentiment data.
Many schools utilize universal behavioral screening to quickly evaluate all students to determine who may need additional support. Most commonly, a single teacher rates all students in their class. Like many other school-based assessments, screening data has a nested structure and is typically ordinally scaled and non-normally distributed. In this study, we simulated data with characteristics similar to behavioral screening data with varied numbers of items, levels of inter-item correlation, and average factor loadings. We estimated single-level and multi-level reliability per Lai () to examine the impacts of nesting and ignoring nesting when estimating reliability. With this type of data, single-level alpha and omega were more like within-class reliability estimates than between-class reliability estimates. 5-point ordinal scales were more like the continuous reliability estimates than the 3- or 4-point scales. We discuss the implications of these findings for screening data used in schools and offer recommendations for estimating reliability for future studies of behavioral screening tools.
Large Language Models (LLMs) have shown promise in automated writing evaluation that provides estimates of writing proficiency ratings and feedback, yet validation studies concentrate on score agreement, overlooking whether human and LLM raters attend to the same aspects in rubrics. We evaluate comparability between a human reference benchmark and an LLM rater using both agreement indices and Many-Facet Rasch Measurement (MFRM) in ratings and feedback, and propose the psychometric approach that separates what theme the rater emphasizes in feedback and how strongly the rater evaluates within a theme. We use an argumentative writing assessment with 300 examinees, scored by human raters and an LLM under multiple prompting conditions. For feedback, 26 rubric codes are mapped into five theory-grounded themes and modeled via dichotomous MFRM for theme detection and partial-credit MFRM for code-level severity. Results show that the surface agreement was improved with few-shot prompting, but systematic rater differences persist in where score and feedback attention are allocated and how severity is applied across themes based on a stable rater-by-domain interaction effect, indicating structured divergence. These findings demonstrate why agreement is insufficient and illustrate how a score and feedback-focused MFRM can identify systematic discrepancies that matter for assessment result interpretation.
