Quality Assurance in Digital-First Assessments

Computational psychometrics, a blend of theory-driven psychometrics and data-driven algorithms, provides the theoretical underpinnings for the design and analysis of the new generation of high-stakes, digital-first assessments that can be taken anytime and anywhere in the world, and their scores impact test takers’ lives. The unprecedented flexibility, complexity, and high-stakes nature of these digital-first assessments pose enormous quality assurance challenges. In order to ensure these assessments meet both “the contest and the measurement” requirements of high-stakes tests, it is necessary to conduct continuous pattern monitoring and to be able to promptly react when needed. In this paper, we illustrate the development of a quality assurance system for a high-stakes and digital-first assessment. To build the system, educational data from continuous administrations of the assessments are mined, modeled and monitored. In particular, five categories of statistics are monitored to assure the quality of the assessment, including scores, test taker profiles, repeaters, item analysis and item exposure. Various control charts and models were applied to detect and flag the abnormal changes in the assessment statistics. The monitoring results and alerts were communicated with the stakeholders via an interactive dashboard. The paper concludes with a discussion on how the automatic quality assurance system is combined with the human review process in real-world application.