Validity CHAPTER 18–Issues and Concerns for Technology-based Performance Assessments

Advancements in technology have led to a revolution in assessments. No longer limited to the bubble-andbooklet approach of the 20th century, today’s assessments may involve rich, interactive exercises, assess new and complex constructs, and automatically record and score evidence of skills. Still, the inferences drawn from technology-rich assessments must adhere to the same principles of good measurement as do traditional assessments. In other words, the inferences or claims made about test-takers (eg,“this learner is ready to move on to the next unit”) must be valid: having sufficient empirical, theoretical, and logical backing. The process of collecting evidence that supports a particular interpretation of assessment results, in the context of a particular use of those results, is called validation.The purpose of this chapter is to introduce the concepts of assessment validity and validation evidence, with the hope that this material serves as a framework for understanding how validity theory and practice may help the research and development of intelligent tutoring systems (ITSs). Our focus is on technology-based assessment, but the reader should interpret “assessment” broadly to include any task–from multiple-choice questions to essays to complex, interactive performance tasks, such as solving algebraic equations, conducting scientific experiments using a simulation-based environment, discussing a story with a virtual peer, or playing an online game. As long as the tasks produce information that allows someone (eg, parent, teacher, test-taker, administrator) or a system (eg, an ITS) to make a decision about test-takers (students), we have a …