Most educational technologies for teaching language skills give students the same learning experience based on the assumption that every student learns at the same rate. In order to provide more individualized instruction we need to track student knowledge at a fine-grained level. This research explores how to add the ability to track knowledge in an existing educational technology system for Japanese second language learners. We explore several potential skill models based on the features available in the system and then apply a Bayesian knowledge tracing algorithm. We also make a large dataset for Japanese language learning available with no previous applied knowledge tracing.

This study sought to characterize individual differences in pupillary dilations during a simple cognitive task. Eighty-four Navy and Marine Corps student pilots performed a digit memory recall test while their pupillary data were recorded. Results showed that peak pupil sizes significantly increased with difficulty of the memory task, however variability in pupillary dilations was substantial, with only 51% of individuals’ data corresponding with the aggregate results and dilations varying between participants by as much as 1 millimeter. The analyses presented in this paper illustrate the large individual variability that exists in pupil data, between individuals and even within individuals on a trial by trial basis. This work serves as a benchmark for understanding variability in pupillary dilations and encourages follow on work to explore casual mechanisms of differences in pupil dilations across individuals, especially before using pupil data for applied purposes.

This paper presents the simulated results of four different scenarios from an ABM to understand the effect of utilization of Alert and Recommender systems on graduation rate. Academic advising help students to enhance their academic performance, and it is also one of the critical aspects of judging institutional effectiveness. For evaluating the effectiveness, college graduation rates are calculated for every scenario, and then they are compared with the baseline scenario. This objective is achieved by using details of SAT scores and core courses of the Department of Physics and Astronomy at a large public university. The prime focus of the paper is to model the impact of the system, which, in reality, helps to save time and cost. The preliminary results show that the Alert and Recommender system could have a positive impact on graduation rates.

In this work we use prior to tutor-session data to generate an individualized student knowledge model. Intelligent learning environments use student models to individualize curriculum sequencing and help messages. Researchers decompose the learning tasks into sets of Knowledge Components (KCs) that represent individual units of knowledge; the student model estimates a parameters for each KC, but not for each student. Using existing performance data to adjust parameters for each individual student improves model fit, and leads to different practice recommendations. However, in order to be implemented in a live system we need to have a method to estimate the student parameters using only the student‘s prior activities. In this work, we use data collected from student reading, prior tutor lessons, to predict individualized difference weights for parameters of a Bayesian Knowledge Tracing (BKT) variant. We find that best-fitting student parameters trained on previous lessons do not directly transfer to new lessons; however, we can effectively predict the student parameters for the new lesson by using features derived from prior lessons, and prior to tutor text-reading transaction data.

As large-scale online classes become more prevalent there is great interest in finding ways to model students at scale in these classes in order to predict outcomes. Student models, if successful, would help determine strong predictors of student success, which would highlight potential causal factors for such success, allowing schools to focus on refinements and interventions that positively impact their student outcomes. In this research, TutorGen has partnered with Western Governors University (WGU), a large online university, and gathered data at scale in order to build exploratory models to predict student outcomes. This paper presents our results so far in successfully identifying students who will pass (or even take) the final exam. We have examined the order in which students take courses, as well as the timing of starting and completing work; our initial analysis reveals that these are strong predictors of course outcomes.

We demonstrate that, by using a small set of hand-graded student work, we can automatically generate rubric criteria with a high degree of validity, and that a predictive model incorporating these rubric criteria is more accurate than a previously reported model. We present this method as one approach to addressing the often challenging problem of grading assignments in programming environments. A classic solution is creating unit-tests that the student-generated program must pass, but the rigid, structured nature of unit-tests is suboptimal for assessing the more open-ended assignments students encounter in introductory programming environments like Alice. Furthermore, the creation of unit-tests requires predicting the various ways a student might correctly solve a problem - a challenging and time-intensive process. The current study proposes an alternative, semi-automated method for generating rubric criteria using low-level data from the Alice programming environment.

K-12 classrooms use block-based programming environments (BBPEs) for teaching computer science and computational thinking (CT). To support assessment of student learning in BBPEs, we propose a learning analytics framework that combines hypothesis- and data-driven approaches to discern students' programming strategies from BBPE log data. We use a principled approach to design assessment tasks to elicit evidence of specific CT skills. Piloting these tasks in high school classrooms enabled us to analyze student programs and video recordings of students as they built their programs. We discuss a priori patterns derived from this analysis to support data-driven analysis of log data in order to better assess understanding and use of CT in BBPEs.

....................................................................................................................................................... 1 The Interaction Networks Approach to Trace Data ...................................................................................... 1 Interaction Networks and Wheel Spinning ................................................................................................... 2 Interaction Networks and Evidence-Centered Design ................................................................................. 2 Difficulties of This Approach for Some Domains ..................................................................................... 4 Results ......................................................................................................................................................... 8 Results for BOTS ..................................................................................................................................... 8 Results for Quantum Spectre ................................................................................................................ 11 Discussion and Future Work ...................................................................................................................... 12 References ................................................................................................................................................ 13 Appendix: Design Pattern .......................................................................................................................... 15 Using Interaction Networks to Identify Unproductive Solution Steps in Multistep Problems 1

The number of students that can be helped in a given class period is limited by the time constraints of the class and the number of agents available for providing help. We use a classroom-replay of previously collected data to evaluate a data-driven method for increasing the number of students that can be helped. We use a machine learning model to identify students who need help in real-time, and an interaction network to group students who need similar help together using approach maps. By assigning these groups of struggling students to peer tutors (as well the instructor), we were able to more than double the number of students helped.

Many introductory programming environments generate a large amount of log data, but making insights from these data accessible to instructors remains a challenge. This research demonstrates that student outcomes can be accurately predicted from student program states at various time points throughout the course, and integrates the resulting predictive models into an instructor dashboard. The effectiveness of the dashboard is evaluated by measuring how well the dashboard analytics correctly suggest that the instructor help students classified as most in need. Finally, we describe a method of matching low-performing students with high-performing peer tutors, and show that the inclusion of peer tutors not only increases the amount of help given, but the consistency of help availability as well.

Building on the promise shown in game-based learning research, this paper explores methods for Game-Based Learning Assessments (GBLA) using a variety of educational data mining techniques (EDM). GBLA research examines patterns of behaviors evident in game data logs for the measurement of implicit learning—the development of unarticulated knowledge that is not yet expressible on a test or formal assessment. This paper reports on the study of two digital games showing how the combination of human coding with EDM has enabled researchers to measure implicit learning of Physics. In the game Impulse, researchers combined human coding of video with educational data mining to create a set of automated detectors of students' implicit understanding of Newtonian mechanics. For Quantum Spectre, an optics puzzle game, human coding of Interaction Networks was used to identify common student errors. Findings show that several of our measures of student implicit learning within these games were significantly correlated with improvements in external postassessments. Methods and detailed findings were different for each type of game. These results suggest GBLA shows promise for future work such as adaptive games and in-class, data-driven formative assessments, but design of the assessment mechanics must be carefully crafted for each game.

Systematic endeavors to take computer science (CS) and computational thinking (CT) to scale in middle and high school classrooms are underway with curricula that emphasize the enactment of authentic CT skills, especially in the context of programming in block-based programming environments. There is, therefore, a growing need to measure students’ learning of CT in the context of programming and also support all learners through this process of learning computational problem solving. The goal of this research is to explore hypothesis-driven approaches that can be combined with data-driven ones to better interpret student actions and processes in log data captured from block-based programming environments with the goal of measuring and assessing students’ CT skills. Informed by past literature and based on our empirical work examining a dataset from the use of the Fairy Assessment in the Alice programming environment in middle schools, we present a framework that formalizes a process where a hypothesis-driven approach informed by Evidence-Centered Design effectively complements data-driven learning analytics in interpreting students’ programming process and assessing CT in block-based programming environments. We apply the framework to the design of Alice tasks for high school CS to be used for measuring CT during programming.

Audience Participation Games challenge traditional assumptions about gameplay by blurring the line between audience and player, allowing audience members to impact gameplay in a meaningful way. Their recent rise in popularity has created new opportunities for game research and development. To better understand this design space, we developed several versions of two prototype games as design probes. We livestreamed them to an online audience in order to develop a framework for audience motivations and participation styles, to explore ways in which mechanics can affect audience members' sense of agency, and to identify promising design spaces. Our results show the breadth of opportunities and challenges that designers face in creating engaging Audience Participation Games.

In this age of fake news and alternative facts, the need for a citizenry capable of critical thinking has never been greater. While teaching critical thinking skills in the classroom remains an enduring challenge, research on an ill-defined domain like critical thinking in the educational technology space is even more scarce. We propose a difficulty factors assessment (DFA) to explore two factors that may make learning to identify fallacies more difficult: type of instruction and belief bias. This study will allow us to make two key contributions. First, we will better understand the relationship between sense-making and induction when learning to identify informal fallacies. Second, we will contribute to the limited work examining the impact of belief bias on informal (rather than formal) reasoning. The results of this DFA will also be used to improve the next iteration of our fallacy tutor, which may ultimately contribute to a computational model of informal fallacies.

Bayesian Knowledge Tracing (BKT) has been employed successfully in intelligent learning environments to individualize curriculum sequencing and help messages. Standard BKT employs four parameters, which are estimated separately for individual knowledge components, but not for individual students. Studies have shown that individualizing the parameter estimates for students based on existing data logs improves goodness of fit and leads to substantially different practice recommendations. This study investigates how well BKT parameters in a tutor lesson can be individualized ahead of time, based on learners’ prior activities, including reading text and completing prior tutor lessons. We find that directly applying best-fitting individualized parameter estimates from prior tutor lessons does not appreciably improve BKT goodness of fit for a later tutor lesson, but that individual differences in the later lesson can be effectively predicted from measures of learners’ behaviors in reading text and in completing the prior tutor lessons.

Our previous work has demonstrated that players who perceive a game as more challenging are likely to perceive greater learning from that game [8]. However, this may not be the case for all sources of challenge. In this study of a Science learning game called Quantum Spectre, we found that students' progress through the first zone of the game seemed to encounter a "roadblock" during gameplay, dropping out when they cannot (or do not want to) progress further. Previously we had identified two primary types of errors in the learning game, Quantum Spectre: Science Errors related to the game's core educational content; and Puzzle Errors related to rules of the game but not to science knowledge. Using this prior analysis, alongside Survival Analysis techniques for analyzing time-series data and drop-out rates, we explored players' gameplay patterns to help us understand player dropout in Quantum Spectre. These results demonstrate that modeling player behavior can be useful for both assessing learning and for designing complex problem solving content for learning environments.

This study examines how accurately individual student differences in learning can be predicted from prior student learning activities. Bayesian Knowledge Tracing (BKT) predicts learner performance well and has often been employed to implement cognitive mastery. Standard BKT individualizes parameter estimates for knowledge components, but not for learners. Studies have shown that individualizing parameters for learners improves the quality of BKT fits and can lead to very different (and potentially better) practice recommendations. These studies typically derive best-fitting individualized learner parameters from learner performance in existing data logs, making the methods difficult to deploy in actual tutor use. In this work, we examine how well BKT parameters in a tutor lesson can be individualized based on learners' prior performance in reading instructional text, taking a pretest, and completing an earlier tutor lesson. We find that best-fitting individual difference estimates do not directly transfer well from one tutor lesson to another, but that predictive models incorporating variables extracted from prior reading, pretest and tutor activities perform well, when compared to a standard BKT model and a model with best-fitting individualized parameter estimates.

Level creation is a creative game-play exercise that resembles problem-posing, and has shown to be engaging and helpful for players to learn about the game’s core mechanic. However, in user-authoring environments, users often create levels without considering the game’s objective, or with entirely different objectives in mind, resulting in levels which fail to afford the core gameplay mechanic. This poses a bigger threat to educational games, because the core gameplay is aligned with the learning objectives. Therefore, such levels fail to provide any opportunity for players to practice the skills the game is designed to teach. To address this problem, we designed and compared three versions of level creators in a programming game — BOTS. These level creators differ in their unit of construction — terrain-block, program and Pattern-block. We measured levels’ gameplay affordances as their relevance to the core game mechanic and learning objective: correct use of programming concepts such as loops, functions and subroutines, and used a zeroinflation model alongside expert tagging of created levels to analyze the differences between the three different editors. The results show that a simple-to-use building-block editor can guarantee levels that contain some affordances, but an editor designed to use the same core mechanic as gameplay results in the best-quality levels overall.

Past studies have shown that Bayesian Knowledge Tracing (BKT) can predict student performance and implement Cognitive Mastery successfully. Standard BKT individualizes parameter estimates for skills, also referred to as knowledge components (KCs), but not for students. Studies deriving individual student parameters from the data logs of student tutor performance have shown improvements to the standard BKT model fits, and result in different practice recommendations for students. This study investigates whether individual student parameters, specifically individual difference weights (IDWs) [1], can be derived from student activities prior to tutor use. We find that student performance measures in reading instructional text and in a conceptual knowledge pretest can be employed to predict IDWs. Further, we find that a model incorporating these predicted IDWs performs well, in terms of model fit and learning efficiency, when compared to a standard BKT model and a model with best-fitting IDWs derived from tutor performance.