Although the idea of learning engineering dates back to the 1960s, there has been an explosion of interest in the area in the last decade. This interest has been driven by an expansion in the computational methods available both for scaled data analysis and for much faster experimentation and iteration on student learning experiences. This article describes the findings of a virtual convening brought together to discuss the potential of learning engineering and the key opportunities available for learning engineering over the next decades. We focus the many possibilities into ten key opportunities for the field, which in turn group into three broad areas of opportunity. We discuss the state of the current art in these ten opportunities and key points of leverage. In these cases, a relatively modest shift in the field's priorities and work may have an outsized impact.

Ce chapitre examine comment les progrès récents de la technologie numérique pourraient conduire à une nouvelle génération d’évaluations éducatives par le jeu. Les systèmes d’éducation disposeraient alors d’évaluations capables de tester des compétences plus complexes que les tests standardisés classiques. Après avoir souligné certains des avantages des évaluations par le jeu par rapport aux autres tests, ce chapitre aborde la manière dont ces tests sont construits, comment ils fonctionnent, mais aussi certaines de leurs limites. Si les jeux présentent un grand potentiel pour améliorer la qualité des tests et étendre l’évaluation à des compétences complexes à l’avenir, ils viendront probablement compléter les tests classiques, qui ont aussi leurs avantages. Trois exemples d’évaluations par le jeu qui intègrent des technologies avancées illustrent cette perspective.

This chapter discusses how recent advancements in digital technology could lead to a new generation of game-based standardised assessments in education, providing education systems with assessments that can test more complex skills than traditional standardised tests can. After highlighting some of the advantages of game-based standardised assessment compared to traditional ones, this chapter discusses how these tests are built, how they work, but also some of their limitations. While games have strong potential to improve the quality of testing and expand assessment to complex skills in the future, they will likely supplement traditional tests, which also have their advantages. Three examples of game-based assessments integrating a range of advanced technologies illustrate this perspective.

The goal of this paper is to spur new conversations and deepen existing ones around how learning sciences (LS) professionals can build bridges through our work in academia and industry and expand the potential of our community. First, we briefly describe the evolution of LS education programs which includes the skill sets needed and the career roles and pathways graduates are prepared for. Next, we propose a role definition of an applied learning scientist (ALS); that is, a learning scientist working outside of academia. We also chronicle a partnership between four learning scientists working in academia and industry. Our paper concludes with reflections and implications for what academics and practitioners learn from one another when they collaborate and promote impactful and scalable environments for learning.

Imbellus is an assessment company that aims to test cognitive processes within the context of immersive simulation-based assessments. This paper explores our work with McKinsey & Company, a best-in-class management consulting firm, to build a simulation-based assessment that gauges applicants’ cognitive skills and abilities. Leveraging a cognitive task analysis grounded in theoretical work and practical observations of on the job activities, we defined key work activities and skills needed to complete them. We then developed scenarios that abstracted and generalized the most crucial skills. To make sense of significant telemetry data from users’ interactions with the assessment, we applied theoretically grounded expert models to guide our scoring algorithms. Our assessment draws inferences across seven major problem-solving constructs. We will present our initial findings and describe implications of our current work for the fields of artificial intelligence and assessment.

iSTART is an intelligent tutoring system designed to provide self-explanation instruction and practice to improve students' comprehension of complex, challenging text. This study examined the effects of extended game-based practice within the system as well as the effects of two metacognitive supports implemented within this practice. High school students (n = 234) were either assigned to an iSTART treatment condition or a control condition. Within the iSTART condition, students were assigned to a 2 × 2 design in which students provided self-assessments of their performance or were transferred to Coached Practice if their performance did not reach a certain performance threshold. Those receiving iSTART training produced higher self-explanation and inference-based comprehension scores. However, there were no direct effects of either metacognitive support on these learning outcomes.

The current study employs transitional probabilities as a way to classify and trace students’ interactions within an online learning system. Results revealed that students’ interaction patterns within the system varied in relation to their performances on embedded assessments. The results and methodologies presented here are designed to provide practitioners with a starting place for how to extract information concerning how and why their students interact within an online environment.

[Correction Notice: An Erratum for this article was reported in Vol 108(7) of Journal of Educational Psychology (see record 2016-27706-001). In the article, the author note should have included the following statement: The research reported here was supported by the Institute of Education Sciences, US Department of Education (IES R305A120707). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the Institute or the US Department of Education.] A commonly held belief among educators, researchers, and students is that high-quality texts are easier to read than low-quality texts, as they contain more engaging narrative and story-like elements. Interestingly, these assumptions have typically failed to be supported by the literature on writing. Previous research suggests that higher quality writing is typically associated with decreased levels of text narrativity and readability. In this study, the authors present the hypothesis that writing proficiency is associated with an individual’s flexible use of linguistic properties, rather than simply the consistent use of a particular set of linguistic properties. To test this hypothesis, the authors leveraged both natural language processing and dynamic methodologies to capture variability in students’ use of narrative style across multiple essay prompts. Forty-five high school students wrote 16 essays across 8 laboratory sessions. Natural language processing techniques were first used to calculate the narrativity of each essay. Random walk and Euclidian distance measures were then used to visualize and classify students’ flexibility in narrativity across essays. The results support the hypotheses that students who were flexible in their use of narrativity also wrote essays that were rated as having higher quality, whereas inflexible writers tended to write lower quality essays. Additionally, more flexible writers performed higher than the more inflexible writers on general assessments of literacy and prior knowledge. These results are important for researchers and educators, as they indicate that the link between textual properties and writing quality may fluctuate according to the context of a given writing assignment. (PsycINFO Database Record (c) 2016 APA, all rights reserved)

Game-based environments frequently afford students the opportunity to exert agency over their learning paths by making various choices within the environment. The combination of log data from these systems and dynamic methodologies may serve as a stealth means to assess how students behave (i.e., deterministic or random) within these learning environments. The current work captures variations in students' behavior patterns by employing two dynamic analyses to classify students' sequences of choices within an adaptive learning environment. Random Walk analyses and Hurst exponents were used to classify students' interaction patterns as random or deterministic. Forty high school students interacted with the game-based system, iSTART-ME, for 11-sessions (pretest, 8 training sessions, posttest, and a delayed retention test). Analyses revealed that students who interacted in a more deterministic manner also generated higher quality self-explanations during training sessions. The results point toward the potential for dynamic analyses such as random walk analyses and Hurst exponents to provide stealth assessments of students' learning behaviors while engaged within a game-based environment.

This study investigates a new approach to automatically assessing essay quality that combines traditional approaches based on assessing textual features with new approaches that assess individual differences in writers such as demographic information, standardized test scores, and survey results. The results demonstrate that combining text features and individual differences increases the accuracy of automatically assigned essay scores over using either individual differences or text features alone. The findings presented here have important implications for both educators and researchers because they reveal that essay scoring methods can benefit from the incorporation of features taken not only from the essay itself (e.g., features related to lexical and syntactic complexity), but also from the writer (e.g., vocabulary knowledge and writing attitudes). Such findings expand our knowledge of textual and non-textual features that are predictive of writing success.

This chapter provides an overview of the Interactive Strategy Tutor for Active Reading and Thinking-2 (iSTART-2). iSTART-2 is a game-based tutoring system designed to improve students' reading comprehension skills. It discusses why reading comprehension is a critical skill and how the iSTART-2 system addresses the development of this skill through the instruction of comprehension strategies. The chapter also provides an overview of the iSTART-2 system and its features. It also discusses the need for this reading comprehension technology in the classroom and provides a general overview of how iSTART-2 addresses those needs and the Common Core State Standards. The chapter describes previous findings from research using the various iterations of the iSTART-2 program. The iSTART-2 program provides students with instruction on comprehension strategies that help them to overcome gaps in their domain knowledge and achieve deep level comprehension of the material within the text. The chapter also describes how iSTART-2 can be accessed by teachers and used within classroom environments.

Game-based practice within Intelligent Tutoring Systems (ITSs) can be optimized by examining how properties of practice activities influence learning outcomes and motivation. In the current study, we manipulated when game-based practice was available to students. All students (n = 149) first completed lesson videos in iSTART-2, an ITS focusing on reading comprehension strategies. They then practiced with iSTART-2 for two 2-hour sessions. Students' first session was either in a game or nongame practice environment. In the second session, they either switched to the alternate environment or remained in the same environment. Students' comprehension was tested at pretest and posttest, and motivational measures were collected. Overall, students' comprehension increased from pretest to posttest. Effect sizes of the pretest to posttest gain suggested that switching from the game to nongame environment was least effective, while switching from a nongame to game environment or remaining in the game environment was more effective. However, these differences between the practice conditions were not statistically significant, either on comprehension or motivation measures, suggesting that for iSTART-2, the timing of game-based practice availability does not substantially impact students' experience in the system.

Writing researchers have suggested that students who are perceived as strong writers (i.e., those who generate texts that are rated as high quality) demonstrate flexibility in their writing style. While anecdotally this has been a commonly held belief among researchers, scientists, and educators, there is little empirical research to support this claim. This study further investigates this hypothesis by examining how students vary in their use of two linguistic features (i.e., narrativity and cohesion) across 16 prompt-based essays. Forty-five high school students wrote 16 essays across 8 sessions within an Automated Writing Evaluation (AWE) system. Natural language processing (NLP) techniques and Entropy analyses were used to calculate how rigid or flexible students were in their use of narrative and cohesive linguistic features over time and how this trait related to individual differences in literacy abilities (i.e., vocabulary knowledge and comprehension ability), prior world knowledge, and essay quality. For instance, through the unique combination of NLP and Entropy, we found that patterns of narrative flexibility (or rigidity) was significantly and reliably related to students’ prior reading comprehension ability after 2 sessions (4 essays). Conversely, students’ flexible (or rigid) use of cohesive features was reliably related to their prior reading comprehension ability after 5 sessions (10 essays). These exploratory methodologies are important for researchers and educators, as they indicate that writing flexibility is indeed a trait of strong writers and can be detected rather quickly using the combination of textual features and dynamic analyses.

Two embodied gears games were created. Better learners should use fewer gear switches to reflect their knowledge. Twenty-three 7th graders, playing as dyads, used gestures to manipulate virtual gears. The Kinect sensor tracked arm-spinning movements and switched gear diameters. Knowledge tests were administered. Statistically significant knowledge gains were seen. For Game 1 gear spun one direction, switching significantly predicted only pretest knowledge. For Game 2 gear spun two directions switching was also negatively correlated with both tests. For game 2, those who used fewer switches during gameplay understood the construct better scoring higher on both tests. Dyadic analyses revealed the winner used significantly fewer switches. In-process data can provide a window onto knowledge as it is being encoded. However, games should stay within the learner's ZPD, because if the game is too easy Game 1, meaningful data may be difficult to gather. The use of in ludo data from games with high sensitivity may attenuate the need for repetitive traditional, post-intervention tests.

Work in cognitive and educational psychology examines a variety of phenomena related to the learning and retrieval of information. Indeed, Alice Healy, our honoree, and her colleagues have conducted a large body of groundbreaking research on this topic. In this article we discuss how 3 learning principles (the generation effect, deliberate practice and feedback, and antidotes to disengagement) discussed in Healy, Schneider, and Bourne (2012) have influenced the design of 2 intelligent tutoring systems that attempt to incorporate principles of skill and knowledge acquisition. Specifically, this article describes iSTART-2 and the Writing Pal, which provide students with instruction and practice using comprehension and writing strategies. iSTART-2 provides students with training to use effective comprehension strategies while self-explaining complex text. The Writing Pal provides students with instruction and practice to use basic writing strategies when writing persuasive essays. Underlying these systems are the assumptions that students should be provided with initial instruction that breaks down the tasks into component skills and that deliberate practice should include active generation with meaningful feedback, all while remaining engaging. The implementation of these assumptions is complicated by the ill-defined natures of comprehension and writing and supported by the use of various natural language processing techniques. We argue that there is value in attempting to integrate empirically supported learning principles into educational activities, even when there is imperfect alignment between them. Examples from the design of iSTART-2 and Writing Pal guide this argument.

Metacognitive awareness has been shown to be a critical skill for academic success. However, students often struggle to regulate this ability during learning tasks. The current study investigates how features designed to promote metacognitive awareness can be built into the game-based intelligent tutoring system (ITS) iSTART-2. College students (n=28) interacted with iSTART-2 for one hour, completing lesson videos and practice activities. If students' performance fell below a minimum threshold during game-based practice, they received a pop-up that alerted them of their poor performance and were subsequently transitioned to a remedial activity. Results revealed that students' scores in the system improved after they were transitioned (even when they did not complete the remedial activity). This suggests that the pop-up feature in iSTART-2 may indirectly promote metacognitive awareness, thus leading to increased performance. These results provide insight into the potential benefits of real-time feedback designed to promote metacognitive awareness within a game-based learning environment.

There is increasing evidence that fine-grained aspects of student performance and interaction within educational software are predictive of long-term learning. Machine learning models have been used to provide assessments of affect, behavior, and cognition based on analyses of system log data, estimating the probability of a student’s particular affective state, behavior, and knowledge (cognition). These measures have (in aggregate) successfully predicted outcomes such as performance on standardized exams. In this paper, we employ a different approach of relating interaction patterns to learning outcomes, using dynamical methods that assess patterns of fine-grained measures of affect, behavior, and knowledge as they occur across time. We use Hurst exponents and Entropy scores computed from assessments of affect, behavior, performance, and knowledge acquired from 1,376 middle school students who used a math tutoring system (ASSISTments), and analyze the relations of these dynamical measures to the students’ end-of-year state test (MCAS) performance. Our results show that fine-grained changes in affect, behavior, and knowledge are significantly related to and predictive of their eventual MCAS performance, providing a new lens on the dynamic and nuanced nature of student interaction within online learning platforms and how it affects achievement.