Performance during training is a poor predictor of long-term retention. Worse yet, conditions of training that produce rapidly improving performance typically do not produce long-lasting, generalizable learning. As a result, learners and instructors alike can be misled into adopting training or educational experiences that are suboptimal for producing actual learning. Computer-based educational training platforms can counter this unfortunate tendency by providing only productive conditions of instruction—even if they are unintuitive (e.g., spacing instead of massing). The use of such platforms, however, introduces a different liability: being easy to interrupt. An assessment of this possible liability is needed given the enormous disruption to modern education brought about by COVID-19 and the subsequent widespread emergency adoption of computer-based remote instruction. The present study was therefore designed to (a) explore approaches for detecting interruptions that can be reasonably implemented by an instructor, (b) determine the frequency at which students are interrupted during a cognitive-science-based digital learning experience, and (c) establish the extent to which the pandemic and ensuing lockdowns affected students’ metacognitive ability to maintain engagement with their digital learning experiences. Outliers in time data were analyzed with increasing complexity and decreasing subjectivity to identify when learners were interrupted. Results indicated that only between 1.565% and 3.206% of online interactions show evidence of learner interruption. And although classroom learning was inarguably disrupted by the pandemic, learning in the present, evidence-based platform appeared to be immune.

There have been many tactics throughout the years aimed at reducing central line-associated bloodstream infections (CLABSI) in the healthcare setting. To reduce CLABSI rates at this facility, we employed cognitive science-based online training directed at nursing departments. Following implementation, we found significant reductions in CLABSI rates that were sustained for a minimum of 9 months. These results demonstrate that this learning methodology can be used to help decrease CLABSI and potentially other health care-associated infections.

Introduction: Confidence-weighted testing assesses learners’ beliefs about their knowledge and skills. As part of a hospital-wide quality improvement initiative to enhance care for pediatric patients with suspected sepsis, we developed a novel intervention using confidence-weighted testing to identify institutional areas of misinformation and knowledge gaps while also providing real-time feedback to individual learners. Methods: We developed pediatric sepsis eLearning modules incorporating confidence-weighted testing. We distributed them to nurses, advanced practitioners, and physicians in emergency departments and acute care/non-intensive care unit inpatient settings in our hospital system. We analyzed completion and response data over 2 years following module distribution. Our outcomes included completion, confidently held misinformation (CHM; when a learner answers a question confidently but incorrectly), struggle (when a learner repeatedly answers a question incorrectly or with low confidence), and mastery (when a learner initially answers a question correctly and confidently). Results: Eighty-three percent of assigned learners completed the modules (1,463/1,754). Although nurses had significantly more misinformation and struggled more than physicians and advanced practitioners, learners of all roles achieved 100% mastery as part of module completion. The greatest CHM and struggle were found in serum lactate interpretation’s nuances and the hemodynamic shock states commonly seen in sepsis. Conclusions: Our novel application of confidence-weighted testing enhanced learning by correcting learners’ misinformation. It also identified systems issues and institutional knowledge gaps as targets for future improvement.

Medical error is the third-leading cause of death in the United States, just behind heart disease and cancer. We describe a software platform used to train healthcare workers to prevent their errors. The platform (Amplifire) harnesses artificial intelligence and principles of cognitive psychology. Amplifire's AI continuously decides whether and when to require additional learning events, provide corrective and metacognitive feedback, and/or deliver self-regulatory guidance for the learner (e.g., "slow down"). Amplifire was deployed to several thousand nurses at a large healthcare system in attempts to reduce the rate of two types of hospital-acquired infections. The result was a 48% reduction in central-line-associated bloodstream infections (CLABSI) and a 32% reduction in catheter-associated urinary-tract infections (CAUTI). These findings demonstrate the effectiveness of using cognitive science along with AI in an e-learning platform.

We describe the Situated Pedagogical Authoring (SitPed) system that seeks to allow non-technical authors to create ITS content for soft-skills training, such as counseling skills. SitPed is built on the assertion that authoring tools should use the learner’s perspective to the greatest extent possible. SitPed provides tools for creating tasks lists, authoring assessment knowledge, and creating tutor messages. We present preliminary findings of a two-phase study comparing authoring in SitPed to an ablated version of the same system and a spreadsheet-based control. Findings suggest modest advantages for SitPed in terms of the quality of the authored content and student learning.

Serious games are generally designed with two goals in mind: promoting learning and creating compelling and engaging experiences (sometimes termed a sense of presence). Presence itself is believed to promote learning, but serious games often attempt to further increase pedagogical value. One way to do so is to use an intelligent tutoring system (ITS) to provide feedback during gameplay. Some researchers have expressed concern that, because feedback from an ITS is often extrinsic (i.e., it operates outside of the primary game mechanic), attending to it disrupts players’ sense of presence. As a result, learning may be unintentionally hindered by an ITS. However, the most beneficial conditions of instruction are often counterintuitive; in this paper, we challenge the assumption that feedback during learning hinders sense of presence. Across three experiments, we examined how an ITS that provided extrinsic feedback during a serious game affected presence. Across different modalities and conditions, we found that feedback and other ITS features do not always affect presence. Our results suggest that it is possible to provide extrinsic feedback in a serious game without detracting from the immersive power of the game itself.

Teachers and trainers often try to prevent learners from making errors, but recent findings (e.g., Kornell, Hays, & Bjork, 2009) have demonstrated that tests can potentiate subsequent learning even when the correct answer is difficult or impossible to generate (e.g., "What is Nate Kornell's middle name?"). In 3 experiments, we explored when and why a failed test enhances learning. We found that failed tests followed by immediate feedback produced greater retention than did a presentation-only condition. Failed tests followed by delayed feedback, by contrast, did not produce such a benefit-except when the direction of the final test was reversed (i.e., the participants were provided with the target and had to produce the original cue). Our findings suggest that generating an incorrect response to a cue both activates the semantic network associated with the cue and suppresses the correct response. These processes appear to have 2 consequences: If feedback is presented immediately, the semantic activation enhances the mapping of the cue to the correct response; if feedback is presented at a delay, the prior suppression boosts the learning of the suppressed response.

In the context of practicing intercultural communication skills, we investigated the role of fidelity in a game-based, virtual learning environment as well as the role of feedback delivered by an intelligent tutoring system. In 2 experiments, we compared variations on the game interface, use of the tutoring system, and the form of the feedback. Our findings suggest that for learning basic intercultural communicative skills, a 3-dimensional (3-D) interface with animation and sound produced equivalent learning to a more static 2-D interface. However, learners took significantly longer to analyze and respond to the actions of animated virtual humans, suggesting a deeper engagement. We found large gains in learning across conditions. There was no differential effect with the tutor engaged, but it was found to have a positive impact on learner success in a transfer task. This difference was most pronounced when the feedback was delivered in a more general form versus a concrete style.

Optimizing learning over multiple retrieval opportunities requires a joint consideration of both the probability and the mnemonic value of a successful retrieval. Previous research has addressed this trade-off by manipulating the schedule of practice trials, suggesting that a pattern of increasingly long lags-"expanding retrieval practice"-may keep retrievals successful while gradually increasing their mnemonic value (Landauer & Bjork, 1978). Here we explore the trade-off issue further using an analogous manipulation of cue informativeness. After being given an initial presentation of English-Iñupiaq word pairs, participants received practice trials across which letters of the target word were either accumulated (AC), diminished (DC), or always fully present. Diminishing cues yielded the highest performance on a final test of cued recall. Additional analyses suggest that AC practice promotes potent (effortful) retrieval at the cost of success, and DC practice promotes successful retrieval at the cost of potency. Experiment 2 revealed that the negative effects of AC practice can be partly ameliorated by providing feedback after each practice trial.

Improvised explosive devices (IEDs) cause more than half of Coalition casualties. In 2009, the Institute for Creative Technologies (ICT) developed a research prototype for a mobile counter-IED training system. This system—the Experiential Counter-IED Immersive Training Environment (ExCITE)—was described at I/ITSEC 2010. Since that time, researchers at the ICT have developed several ways of assessing learning from ExCITE. One measure is a paper-based pretest/posttest. Another is a system that tracks trainees’ responses to interactive training tools and their performance during a simulated vehicle patrol. This paper describes how these measures were created and refined. This paper also presents preliminary results from our empirical studies on the effectiveness of this counter-IED training.

In February 2009, the Institute for Creative Technologies (ICT) at the University of Southern California was contracted to develop a training system for Soldiers and Marines. The goal of the training system was to reduce the number of casualties caused by improvised explosive devices (IEDs). Over the next five months, a team at the ICT drew on subject-matter experts’ input, findings from cognitive psychology, and cinema-style script-writing to achieve this goal. The result is the Mobile Counter-IED Interactive Trainer (MCIT), a narrative-focused, mixedmedia training simulator that can be deployed anywhere in the world. This paper details the course of MCIT’s development, from initial concept to working prototype to finalized training system. We also discuss the technical challenges of developing in VBS2 and the changes driven by user feedback throughout the development process.

It seems uncontroversial that providing feedback after a test, in the form of the correct answer, enhances learning. In real-world educational situations, however, the time available for learning is often constrained-and feedback takes time. We report an experiment in which total time for learning was fixed, thereby creating a trade-off between spending time receiving feedback and spending time on other learning activities. Our results suggest that providing feedback is not universally beneficial. Indeed, under some circumstances, taking time to provide feedback can have a negative net effect on learning. We also found that learners appear to have some insight about the costs of feedback; when they were allowed to control feedback, they often skipped unnecessary feedback in favor of additional retrieval attempts, and they benefited from doing so. These results underscore the importance of considering the costs and benefits of interventions designed to enhance learning.

In ill-defined domains, properly assessing learning is, itself, an illdefined problem. Over the last several years, the domain of interest to us has been teaching Americans about Iraqi business culture via a serious-game-based practice environment. We describe this system and the various measures we used in a series of studies to assess its ability to teach. As subsequent studies identified the limits of each measure, we selected additional measures that would let us better understand what and how people were learning, using Bloom’s revised taxonomy as a guide. We relate these and other lessons we learned in the process of refining our solution to this ill-defined problem.

We investigate the role of presence in a serious game for intercultural communication and negotiation skills by comparing two interfaces: a 3D version with animated virtual humans and sound against a 2D version using text-only interactions with static images and no sound. Both versions provide identical communicative action choices and are driven by the same underlying simulation engine. In a study, the 3D interface led to a significantly greater self-reported sense of presence, but produced significant, but equivalent learning on immediate posttests for declarative and conceptual knowledge related to intercultural communication. Log data reveals that 3D learners needed fewer interactions with the system than those in the 2D environment, suggesting they benefited equally with less practice and may have treated the experience as more authentic.

Taking tests enhances learning. But what happens when one cannot answer a test question-does an unsuccessful retrieval attempt impede future learning or enhance it? The authors examined this question using materials that ensured that retrieval attempts would be unsuccessful. In Experiments 1 and 2, participants were asked fictional general-knowledge questions (e.g., "What peace treaty ended the Calumet War?"). In Experiments 3-6, participants were shown a cue word (e.g., whale) and were asked to guess a weak associate (e.g., mammal); the rare trials on which participants guessed the correct response were excluded from the analyses. In the test condition, participants attempted to answer the question before being shown the answer; in the read-only condition, the question and answer were presented together. Unsuccessful retrieval attempts enhanced learning with both types of materials. These results demonstrate that retrieval attempts enhance future learning; they also suggest that taking challenging tests-instead of avoiding errors-may be one key to effective learning.