Block-position effects in large-scale assessments are often modeled as common difficulty shifts, although variation may differ across administration stages and across examinee-item ability-difficulty relationships. This study develops a staged two-parameter logistic framework: M0 has no position effect, M1 adds a common linear shift, M2 permits stage-specific nonlocalized shifts, and M3 allows those shifts to vary with ability-difficulty proximity. The framework was evaluated in controlled balanced and TIMSS-structured sparse simulations and applied to TIMSS 2019 Chinese Taipei Grade 8 mathematics data. Across both simulations, M1 was sufficient for linear effects, M2 for nonlinear but nonlocalized stage effects, and M3 added information when localization was genuinely present. M3 primarily improved recovery of position-effect and probability-effect functions and threshold-proximal calibration, whereas sparse administration reduced precision and weakened identification of the nonlocalized boundary. In the TIMSS application, the common linear trend was small, whereas the empirical administration design separated the items into two disconnected position families, Positions 1 and 4 versus Positions 2 and 3. M3 yielded a proximity-varying fitted pattern aligned with these two families and concentrated near the baseline response threshold. Because no items connected the two families, this between-family pattern cannot be attributed uniquely to block position; the directly connected within-family contrasts did not show clear differences. Global held-out predictive differences were very small. The fitted family-aligned pattern persisted under a weaker item-difficulty prior, although amplitude and localization width were prior-sensitive. The framework therefore supports staged diagnosis of position-related difficulty variation, with M3 used to assess additional localization rather than general predictive superiority.
This study evaluates the extent to which synthetic response generation methods reproduce the psychometric properties of Likert-type social-emotional assessment data. We compare four approaches: machine learning (ML), text-embedding models, zero-shot large language models (LLMs), and fine-tuned LLMs, in generating responses to 10 field-test items from the Devereux Student Strengths Assessment (DESSA). Synthetic responses were evaluated using prediction accuracy, graded response model (GRM) parameter recovery, response-style indices, and person-level response patterns. At an aggregate level, ML and text-embedding approaches achieved higher accuracy and more stable recovery of threshold parameters than LLM-based methods. However, closer examination revealed systematic distortions across all methods, including shifts in scale-use behavior and reduced variability in person-level response patterns, particularly for LLM-generated data. These findings suggest that while synthetic methods can approximate surface-level response distributions, they do not consistently reproduce the response processes underlying ordinal data. Implications for the use of synthetic data in early-stage item development and psychometric evaluation are discussed.
Structure-Regularized Composite Scoring (SRCS) was developed as a person-scoring extension for calibrated multidimensional item response theory (MIRT) item banks. Given a Q-matrix, reference distribution, and operational mastery cuts, SRCS combines the calibrated MIRT measurement score with a soft structural score while leaving the item response functions fixed. SRCS was compared with MIRT post hoc classification, MIRT with re-estimated DINA profile scoring (MIRT-reDINA), and Sequential ICC-Induced Diagnostic Scoring (SIDS) across 6 simulation conditions and an empirical illustration using the ECPE grammar data. The first three procedures retained identical conventional MIRT proficiency estimates but differed in their profile-scoring distributions; SRCS alone modified continuous person scoring. In simulation, SRCS incurred small losses in proficiency recovery, classification accuracy, probability calibration, and specificity, while increasing sensitivity and slightly reducing posterior-weighted measurement-structural discrepancy. The soft structural representation was consistently closer to the calibrated MIRT response surface than the hard representation. Empirically, SRCS closely preserved MIRT scores and operational profiles and moderately concentrated profile probabilities, but it did not reduce measurement-structural discrepancy. Overall, SRCS provides a controlled scoring-stage mechanism for incorporating attribute-based structure while preserving a calibrated MIRT measurement system.
Comparative judgment (CJ) is an assessment method in which assessors compare pairs of essays and judge which is of higher quality. While CJ produces valid and reliable results, it often requires many judgments before quality scores become reliable. In this study, we examine whether natural language processing (NLP) can improve the efficiency of CJ by warm-starting the estimation of quality scores and the selection of essay pairs. We predict essay quality by fine-tuning a transformer model and use these predictions to construct informative priors in a Bayesian Bradley-Terry-Luce model. We then introduce three selection rules based on predicted scores and essay embeddings. In this study, we focus on Dutch writing assessments in secondary and higher education in two settings: one-off assessments and recurrent assessments. Warm-start estimation reduced the number of judgments required to reach a reliability of .70 by approximately 40%-68% compared with likelihood-based cold-start estimation and by approximately 24%-64% compared with Bayesian cold-start estimation. To reach a reliability of .90, warm-start estimation yielded smaller reductions of around 15% compared with cold-start estimation. The warm-start selection rules we constructed yielded smaller additional gains. In practice, a reliability of .70 was reached after two to four judgments per essay, and a reliability of .90 after seven to fourteen judgments. Overall, warm-starting CJ substantially reduces assessor workload and improves the practical applicability of CJ as a summative assessment method.
