Observing Testing Situations: Validation As Jazz

Observations of real-life testing situations can provide important insights into test validation and assessment response processes. We consider observations of face-to-face interaction as a starting point to investigate how variation in assessment performance is informed by the ecological characteristics of the testing situation. This perspective offers a contrast to conventional 'think aloud' protocols. Observations of real-life testing situations capture in-vivo perspectives on assessment interaction and performance as it occurs, rather than as if occurring in-vitro. Our aim is to deliberately capture and consider the "noisy" ecological characteristics in the testing situation (e.g., interaction, setting, affect) that may influence response processes, but that are not part of the trait under investigation. Observations of real-life testing situations reveal observable structures and patterns of behaviour, but every performance is somewhat different. Like jazz, the testing situation involves elements of improvisation. This is illustrated using transcripts from the OECD Survey of Adult Skills (PIAAC).