Detecting Rater Bias in Mixed-Format Assessments

Mixed-format assessments made up of multiple-choice (MC) items and constructed response (CR) items that are scored using rater judgments include unique psychometric considerations. When these item types are combined to estimate examinee achievement, information about the psychometric quality of each component can depend on that of the other. For example, the presence of differential item functioning (DIF) in MC items could compromise the sensitivity of rater effect indicators, such as differential rater functioning (DRF) indices. Likewise, the presence of DRF could compromise the sensitivity of DIF indices. We used real data and a simulation study to consider the impact of DIF on DRF indices, and the impact of DRF on DIF indices. Results indicate some interaction between DIF and DRF that varies across data collection designs. We consider the implications of these results for research and practice.