A Generalized Framework for Describing Question Randomization

The rise of online assessments has motivated the development of randomized question banks, with randomization referring to the generation of different variants of a question. Although not all randomization efforts are equally effective in generating question isomorphs, the current classification of questions solely as randomized or not fails to address the varying degrees of randomization. To address this limitation in describing the diversity of randomization designs, we introduce a framework that outlines six distinct randomization levels. Additionally, we designed practical guides to assist educators in effectively using the framework, aligning with their pedagogical objectives. Through our application of this framework to classify around 200 questions from two courses, we further highlight the generalizability of the framework and reveal insights into the considerations and challenges associated with incorporating question randomization into computer science curricula.