Challenges to assessing motivation in MOOC learners: An application of an argument-based approach.

While data for examining learner activity are abundant, there are relatively little frameworks for principled interpretation of the behavior. Through application of Kane's argument-based approach to assessment validation, the authors conducted several analyses to combine motivation assessment with online behavioral data for further validating inferences made about MOOC learners. The EVC motivation scale was administered to learners in three advanced engineering MOOCs, and event log data was collected from the online learning platform. The EVC items were comprehensively examined (n = 661) through factor analysis, item response theory, and linear regression. Results indicated that the instrument retained a three-factor structure as well as strict and structural invariance across age groups and education levels, but the Expectancy and Value items suffered from significant ceiling effects, a key difficulty in assessing motivation using this approach. As part of a regression model with learners' intentions, EVC scores did not account for a significant amount of variance in learners' assessment outcomes and course behavior. Challenges remain for adjusting to learners' high expectancy and value of courses at the beginning, and for fully understanding how scores relate to learner behavior.