Measuring Teachers' Civic Online Reasoning in a MOOC with Virtual Simulations and Automated Feedback Systems

In an increasingly polarized digital landscape, evaluating online information has become a critical media literacy skill. Many individuals have difficulty distinguishing satirical from legitimate news sources [1]. One useful strategy is lateral reading, looking up information about a website or social media account in order to judge its credibility from an outside source [2]. We developed a short lateral reading task where we asked users to evaluate the trustworthiness of the satire account @GOPTeens and explain their response in a short text response. We developed a natural language processing classifier to detect whether users correctly identified the account as a satire account, which would indicate that they employed lateral reading to evaluate the trustworthiness of the account. This classifier examines a very specific case and the NLP classifier was highly accurate with a Macro F1 statistic of 0.96 (Overall Macro F1 = 0.96, Yes F1 = 0.99, No F1 = 0.94). In future work, we will employ the classifier to provide targeted feedback to users and will explore the effects of facilitative versus directive feedback on performance with lateral reading tasks.