Background Adaptive instruction tailors learning to individual students by responding to their evolving ideas. Aims This study uses principles of Knowledge Integration (KI) and cognitive activation to inform the design of adaptive instruction that integrates system-driven and teacher-led activities and examines how students’ conceptual understanding and disciplinary ideas progress during implementation. Participants Two middle school science teachers and 272 students. Method In regular classrooms, students engaged with an adaptive thermodynamics unit. Based on real-time formative assessment scores (T1), students received condition-specific activities featuring three adaptive elements (example response, guiding question, and interactive model), revised their explanations (T2), participated in teacher-led instruction, and revised again (T3). Explanations were automatically scored for the level of knowledge integration and disciplinary ideas at three time points. A pre-post item is used to assess overall learning gains. Learning process data is quantified for additional insight. Results Linear mixed-effects models showed that students with emerging understanding at T1 benefited most from the adaptive activities. Students expressing connected ideas at T1 decreased their knowledge integration level across the revisions of the formative assessment but improved most from pre-to posttest. Analysis of student-generated improvement suggestions revealed that many students identified target domain ideas during the adaptive activity. Conclusions The adaptive design successfully integrated automated and teacher-led instruction and supported the development of condition-specific target ideas. However, students need help to retain, rather than replace, relevant ideas. Future studies should evaluate the design against a non-adaptive control group to determine the added value of adaptivity.
In educational assessment, automated scoring systems increasingly rely on deep learning and large language models (LLMs). However, these systems face significant risks of bias amplification, where model prediction gaps between student groups become larger than those observed in training data. This issue is especially severe for underrepresented groups such as English Language Learners (ELLs), as models may inherit and further magnify existing disparities in the data. We identify that this issue is closely tied to representation bias: the scarcity of minority (high-scoring ELL) samples makes models trained with empirical risk minimization favor majority (non-ELL) linguistic patterns. Consequently, models tend to under-predict ELL students who even demonstrate comparable domain knowledge but use different linguistic patterns, thereby undermining fairness in automated scoring. To mitigate this, we propose BRIDGE, a Bias-Reducing Inter-group Data GEneration framework designed for low-resource assessment settings. Instead of relying on the limited minority samples, BRIDGE synthesizes high-scoring ELL samples by “pasting” construct-relevant (i.e., rubric-aligned knowledge and evidence) content from abundant high-scoring non-ELL samples into authentic ELL linguistic patterns. We further introduce a discriminator model to ensure the quality of synthetic samples. Experiments on California Science Test datasets demonstrate that BRIDGE effectively reduces prediction bias for high-scoring ELL students while maintaining overall scoring performance. Notably, our method achieves fairness gains comparable to using additional real human data, offering a cost-effective solution for ensuring equitable scoring in large-scale assessments.
Person-fit statistics are widely used to detect aberrant response patterns in educational and psychological measurement. Snijders suggested an asymptotically correct standardization for a broad class of such statistics. This paper presents an alternative derivation of this standardization. The derivation yields several advantages including a simpler formula and simpler description of several person-fit statistics including the l z * statistic and a theoretical explanation of simulation findings reported by Snijders and van Krimpen-Stoop and Meijer, among others.
Large language models (LLMs) are increasingly used to support automated writing evaluation (AWE), both for purposes of scoring and feedback. However, LLMs present challenges to interpretability, making it hard to evaluate the construct validity of scoring and feedback models. BIOT (best interpretable orthogonal transformations) is a new method of analysis that makes dimensions of an embedding interpretable by aligning them with external predictors. It was originally developed to improve the interpretability of multidimensional scaling models. However, This paper shows that BIOT can be used to align LLM embeddings with an interpretable writing trait model developed using multidimensional analysis of classical NLP features to measure latent dimensions of writing style and writing quality. This makes it possible to determine whether an AWE model built using an LLM is aligned with known (and construct-relevant) dimensions of textual variation, supporting construct validity. Specifically, we examine the alignment between the hidden layers of deBERTA, a small LLM that has been shown to be useful for a variety of natural language processing applications, and a writing trait model developed through factor analysis of classical features used in existing AWE models. Specific dimensions of transformed deBERTA layers are strongly correlated with these classical factors. When the transformation matrix derived using BIOT is applied to token vectors, it is also possible to visualize which tokens in the original text contributed to high or low scores on a specific dimension.
Large language models (LLMs) are becoming widely deployed as personal AI assistants with access to sensitive user data, making privacy a major challenge for their design and evaluation. Prior work focuses mainly on individual-level risks, overlooking \textbf{interdependent privacy (IDP)}--where one person's data may be revealed by others without their knowledge or consent. We address this gap by introducing \textbf{IDP-Bench}: the first LLM benchmark for IDP scenarios, grounded in the Contextual Integrity (CI) framework. We evaluate eight open-source LLMs on their understanding of IDP scenarios across three levels of IDP reasoning using two LLM judges. Results show strong co-ownership recognition (6/8 models exceed 90\%) but persistent weaknesses in identifying CI parameters (information attribute, primary subject) and IDP-specific parameters such as secondary subjects, where 7/8 models score below 74\%. Models also struggle to judge sharing appropriateness (5/8 scoring below 77\%). While the ability to judge the appropriateness of sharing improves with scale, performance tends to decline in smaller models, and prompt sensitivity remains high on IDP-specific questions--highlighting the need for more targeted study of IDP in LLM privacy research. Data \& code available \href{https://github.com/tisl-lab/Interdependent_Privacy_Bench}{here}.
