Validating the Use of LMS-Derived Rubric Structural Features to Facilitate Automated Measurement of Rubric Quality

Rubrics are widely used throughout postsecondary education as means for aiding in instruction and evaluation. However, despite their broad global adoption, very little is known about the quality of rubrics in use. We develop two measures to assess the quality of rubrics: (1) a checklist identifying criteria of high-quality rubrics based on analytic rubric design best practices and (2) a set of LMS-derived features that are hypothesized to represent structural components that are, in general, necessary but not sufficient for high quality rubrics. The validity of using the feature-generated scores as proxies for identifying rubric quality is evaluated through several means. First, the feature-generated scores are calculated for a set of external exemplary rubrics of known high quality. Second, the feature-scores for a subset of internal rubrics are compared to average human rater scores of rubric quality based on the checklist. We discuss the results, practical applications, and a larger research program surrounding the feature-generated scores.