How instructional context can impact learning with educational technology: Lessons from a study with a digital learning game

There is a tendency to think of the impact of educational technology in a vacuum. However, it is likely that the instructional context in which educational technology is used affects student learning. For instance, outcomes may differ when using educational technology in a classroom versus at home, in a quiet versus noisy environment, or in a context where support is readily available versus not available. The COVID-19 pandemic provided an unexpected opportunity to explore this issue. Intending to explore how providing hints and feedback within a digital learning game (Decimal Point) impacts mathematics learning, we instead found ourselves exploring a new question: How did learning with the game differ between classrooms and at home? After two of five middle schools had participated in our classroom experiment, we switched to at-home use of the Internet-based game for the final three schools due to the pandemic. The different instructional settings led to significantly different completion rates, likely due to students in the classroom (N = 151) being monitored by experimenters and teachers (completion rate of 88.8%), while students at home (N = 126) were not monitored nor strictly required to finish (completion rate of 56.5%). In addition, the two versions of the game, one that provided students with onrequest hints and error feedback (Hint condition) and one that did not (No-Hint condition), led to different classroom versus at-home results. On the delayed posttest, students in the No-Hint condition did significantly better in the classroom, while there was no significant difference between conditions at home. In addition, students in the Hint condition used significantly more hints in the classroom than they did at home. There was also a significant effect of gender in the classroom, with female students out-performing male students on the immediate posttest, but with no effect of gender remotely. We performed post-hoc analyses to better understand students' learning processes and gameplay behaviors. In summary, our study clearly illustrates how educational technology can be sensitive to instructional context, yet just cracks open the door to much more research on this topic.