Instructional designers (IDs) are charged with what some might argue is the impossible: implementing innovative technologies and new pedagogical approaches within complex systems that often implicitly discourage change. This 6-year case study examined key factors influencing the exploration, multi-course pilot, and ultimate end of an adaptive learning (AL) project. AL has been recognized as a potentially transformative approach for online learning that individualizes instruction to each student based on demonstrated competencies. Past research has considered performance comparisons, learner characteristics, course design, and teaching practices, but what has been largely overlooked is an examination of the ways in which the university context itself impacts, and is impacted by, AL. Our study showed that while IDs and instructors easily recognize the potential benefits of AL, it is significantly more difficult to gain sustained administrative leadership advocacy. Primary challenges encountered were (1) AL’s disruption to the online teaching and learning status quo and (2) ongoing tensions between institutional structure, culture, and best practices for AL. The findings of this study suggest that a systems-based approach to organizational change is necessary for the successful implementation of multifaceted approaches like AL. The paper offers IDs greater insight into what can happen “behind the scenes” to support or challenge the success of educational technology implementations. While they may not always be able to impact these factors directly, having more awareness about the system complexity can allow them to be more strategic in asking for resources and buy-in from leadership.

Psychology researchers have long attempted to identify educational practices that improve student learning. However, experimental research on these practices is often conducted in laboratory contexts or in a single course, which threatens the external validity of the results. In this article, we establish an experimental paradigm for evaluating the benefits of recommended practices across a variety of authentic educational contexts—a model we call ManyClasses . The core feature is that researchers examine the same research question and measure the same experimental effect across many classes spanning a range of topics, institutions, teacher implementations, and student populations. We report the first ManyClasses study, in which we examined how the timing of feedback on class assignments, either immediate or delayed by a few days, affected subsequent performance on class assessments. Across 38 classes, the overall estimate for the effect of feedback timing was 0.002 (95% highest density interval = [−0.05, 0.05]), which indicates that there was no effect of immediate feedback compared with delayed feedback on student learning that generalizes across classes. Furthermore, there were no credibly nonzero effects for 40 preregistered moderators related to class-level and student-level characteristics. Yet our results provide hints that in certain kinds of classes, which were undersampled in the current study, there may be modest advantages for delayed feedback. More broadly, these findings provide insights regarding the feasibility of conducting within-class randomized experiments across a range of naturally occurring learning environments.