Evaluating Response Processes Underlying Multiple‐Choice and Short‐Answer Questions Using Eye‐Tracking Data

Abstract The analysis of eye‐tracking data has been shown to be useful in supporting inferences about the underlying processes of test takers responding to multiple‐choice questions (MCQs; Yaneva et al.), especially when framed as one piece of evidence within an overall validity argument. The same type of eye‐tracking data can also be collected with other item formats such as short‐answer questions (SAQs). In the present study, we leveraged eye‐tracking data to evaluate the extent to which test takers engage in different processes when responding to MCQs and SAQs. We randomly assigned medical students matched test items presented in either MCQ or SAQ format and collected eye‐tracking data as they responded to each item. We then compared various eye‐tracking measures (sequencing order, location, and duration of visual fixations) across the two item formats. The results provided little evidence of differences in the response process for the two item formats.