One of the fundamental challenges towards building any intelligent tutoring system is its ability to automatically grade short student answers. A typical automatic short answer grading system (ASAG) grades student answers across multiple domains (or subjects). Grading student answers requires building a supervised machine learning model that evaluates the similarity of the student answer with the reference answer(s). We observe that unlike typical textual similarity or entailment tasks, the notion of similarity is not universal here. On one hand, para-phrasal constructs of the language can indicate similarity independent of the domain. On the other hand, two words, or phrases, that are not strict synonyms of each other, might mean the same in certain domains. Building on this observation, we propose JMD-ASAG, the first joint multidomain deep learning architecture for automatic short answer grading that performs domain adaptation by learning generic and domain-specific aspects from the limited domain-wise training data. JMD-ASAG not only learns the domain-specific characteristics but also overcomes the dependence on a large corpus by learning the generic characteristics from the task-specific data itself. On a large-scale industry dataset and a benchmarking dataset, we show that our model performs significantly better than existing techniques which either learn domain-specific models or adapt a generic similarity scoring model from a large corpus. Further, on the benchmarking dataset, we report state-of-the-art results against all existing non-neural and neural models.

Automatic short answer grading for Intelligent Tutoring Systems has attracted much attention of the researchers over the years. While the traditional techniques for short answer grading are rooted in statistical learning and hand-crafted features, recent research has explored sentence embedding based techniques. We observe that sentence embedding techniques, while being effective for grading in-domain student answers, may not be best suited for out-of-domain answers. Further, sentence embeddings can be affected by non-sentential answers (answers given in the context of the question). On the other hand, token level hand-crafted features can be fairly domain independent and are less affected by non-sentential forms. We propose a novel feature encoding based on partial similarities of tokens (Histogram of Partial Similarities or HoPS), its extension to part-of-speech tags (HoPSTags) and question type information. On combining the proposed features with sentence embedding based features, we are able to further improve the grading performance. Our final model achieves better or competitive results in experimental evaluation on multiple benchmarking datasets and a large scale industry dataset.

The ubiquity of toys viz-a-viz children and their close relationship have been leveraged by academics and industry alike to provide applications and services through these toys. Many domains such as healthcare, medical intervention, entertainment, education have all benefited from smart toys which could sense their context and/or perform actuation tasks such as conversations, support etc. However, state-of-the-art toy systems lack connected-yet-autonomous behavior and hence are unable to perform coordinated tasks which could facilitate complex interactions especially in education. Few such activities that require complex multi-toy coordination and interaction are teaching social skills such as social etiquettes, morals and group conversations; as well as cognitive skills such as counting, pattern identification, grouping etc. This paper is a first effort towards generating interest and discussion around this interesting area of using multiple connected toys for learning. In this paper, we propose Puppeteer, a decentralized orchestrator platform to enable coordinated yet autonomous multi-toy behavior which provides seamless coordination between toys, and a declarative mechanism to author and adapt learning scenarios for enactment. We evaluate this system through simulations and real-world prototype experiments, and report its performance.

Automatic short answer grading remains one of the key challenges of any dialog-based tutoring system due to the variability in the student answers. Typically, each question may have no or few expert authored exemplary answers which make it difficult to (1) generalize to all correct ways of answering the question, or (2) represent answers which are either partially correct or incorrect. In this paper, we propose an affinity propagation based clustering technique to obtain class-specific representative answers from the graded student answers. Our novelty lies in formulating the Scoring Rubric by incorporating class-specific representatives obtained after proposed clustering, selecting, and ranking of graded student answers. We experiment with baseline as well as stateof-the-art sentence-embedding based features to demonstrate the feature-agnostic utility of class-specific representative answers. Experimental evaluations on our large-scale industry dataset and a benchmarking dataset show that the Scoring Rubric significantly improves the classification performance of short answer grading.

Accurate classification of learner responses is a critical component of dialog based tutoring systems (DBT). Errors in identifying the intent and context of responses can have cascading effects on the ongoing interaction thereby affecting the learning experience and outcome. In this paper we attempt to quantify the impact of Tutor misclassifications on student behavior by analyzing differences across our hypothesized conditions namely, no-misclassification vs. misclassification using various dialog metrics. We find that not only are there significant changes in behavior across the two groups but that Tutor errors related to misunderstanding of Intent - although fewer in occurrence, appear to have a higher impact than a misclassification of a valid student answer. We also see some evidence of the effectiveness of scaffolds like FITBs in sustaining dialog thereby mitigating the effects of a Tutor error.

Instructional Systems Design is the practice of creating of instructional experiences that make the acquisition of knowledge and skill more efficient, effective, and appealing. Specifically in designing courses, an hour of training material can require between 30 to 500 hours of effort in sourcing and organizing reference data for use in just the preparation of course material. In this paper, we present the first system of its kind that helps reduce the effort associated with sourcing reference material and course creation. We present algorithms for document chunking and automatic generation of learning objectives from content, creating descriptive content metadata to improve content-discoverability. Unlike existing methods, the learning objectives generated by our system incorporate pedagogically motivated Bloom's verbs. We demonstrate the usefulness of our methods using real world data from the banking industry and through a live deployment at a large pharmaceutical company.

Intelligent tutoring systems (ITS) have been a topic of great interest for about five decades. Over the years, ITS research has leveraged AI advancements, and has also helped push the boundaries of AI capabilities with grounded usage scenarios. Using ITSs along with classroom instruction to augment traditional teaching is a canonical example of how humans and machines can work together to solve problems that are otherwise overwhelming and non-scalable individually. The experiences of personalized learning created by (1) seamless orchestration of human decision-making at few critical points with (2) scalability of cognitive capabilities using AI systems can drive increased student engagement leading to improved learning outcomes. By considering two particular use-cases of early childhood learning and higher education, we discuss the challenges involved in designing these complex human-centric systems. These systems integrate technologies involving interactivity, dialog, automated question generation, and learning analytics.

Accurately interpreting student responses is a critical requirement of dialog-based intelligent tutoring systems. The accuracy of supervised learning methods, used for interpreting or analyzing student responses, is strongly dependent on the availability of annotated training data. Collecting and grading student responses is tedious, time-consuming, and expensive. This work proposes an iterative data collection and grading approach. We show that data collection efforts can be significantly reduced by predicting question difficulty and by collecting answers from a focused set of students. Further, grading efforts can be reduced by filtering student answers that may not be helpful in training Student Response Analyzer (SRA). To ensure the quality of grades, we analyze the grader characteristics, and show improvement when a biased grader is removed. An experimental evaluation on a large scale dataset shows a reduction of up to 28% in the data collection cost, and up to 10% in grading cost while improving the response analysis macro-average F1.

Question answering forums in online learning environments provide a valuable opportunity to gain insights as to what students are asking. Understanding frequently asked questions and topics on which questions are asked can help instructors in focusing on specific areas in the course content and correct students’ confusions or misconceptions. An underlying task in inferring frequently asked questions is to identify similar questions based on their content. In this work, we use hierarchical agglomerative clustering that exploits similarities between words and their distributed representations, reflecting both lexical and semantic similarity of questions. We empirically evaluate our results on real world labeled dataset to demonstrate the effectiveness of the method. In addition, we report the results of inferring frequently asked questions from discussion forums of online learning environment providing lectures to middle school and high school students.

We report on a study of affective states of learners in a Massive Open Online Course (MOOC) and the interplay of Affect, Behavior and Cognition at various stages of the course. Affect is measured through a series of self-reports from learners at strategic time posts during the period of study. Behavior is characterized in terms of a learners' engagement, interactivity, impatience and reflectivity, which constitute a set of novel high-level features derived from the clickstream of learner interactions. Cognition is evaluated from the performance of learners on assessments that are part of the course. We discover that learners in the MOOC experience multiple as well as mixed emotions as they go through the course, which we handle using the psychological dimensions of arousal and valence. This results in a set of emotional quadrants, whose co-occurrence analysis reveals a strong association with cognition and specific behavioral characteristics demonstrated by the learner. These results advance our understanding of the experience of MOOC learners to a more holistic level across the key dimensions of affect, behavior and cognition. They also have important implications for the design of the next generation MOOCs that can potentially leverage affect and behavior-aware interventions to drive greater personalization and eventually, improved learning outcomes.

This paper discusses self-reported emotions experienced by students in a Massive Open Online Course (MOOC) learning context. Emotions have been previously shown to be related to learning in classrooms and laboratory studies and have even been leveraged to improve learning. In this study, frequently occurring discrete emotions as well as frequently, co-occurring pairs of emotions were analyzed during learning with a MOOC. Both discrete and co-occurring emotions were related to students dropping out of the course, illustrating the importance of student emotion in a MOOC context.

This paper presents affect data collected from periodic emotion detection surveys throughout an introductory Statistics MOOC called "I Heart Stats." This is the first MOOC, to our knowledge, to capture valuable student affect data through self-reported surveys. To collect student affect, we used two self-reporting methods: (1) The Self-Assessment Manikin and (2) A discrete emotion list. We found that the most common reported MOOC emotion was Hope followed by Enjoyment and Contentment. There were substantial shifts in affective states over the course, notably with Anxiety and Pride. The most valuable result of our study is a preliminary description of the methods for collecting self-reported student affect at scale in a MOOC setting.

Teaching in formal academic environments typically follows a curriculum that specifies learning objectives that need to be met at each phase of a student’s academic progression. In this paper, we address the novel task of identifying document segments in educational material that are relevant for differ-ent learning objectives. Using a dynamic programming algo-rithm based on a vector space representation of sentences in a document, we automatically segment and then label document segments with learning objectives. We demonstrate the effectiveness of our approach on a real-world education data set. We further demonstrate how our system is useful for related tasks of document passage retrieval and QA using a large publicly available dataset. To the best of our knowledge we are the first to attempt the task of segmenting and labeling education materials with academic learning objectives.

Learning standards (frequently referred to as academic standards, course curriculum etc.) define the specific structure of an educational program. Learning standards contain a list of instructions specifying various skills that students should learn at different points during their learning progression. For example,“calculate the area of a triangle” is one such instruction in a 6 grade geometry curriculum. Currently these instructions are imparted using prescribed textbooks or lesson plans which have been labeled with learning standard instructions. Teachers and students use this labeled learning content to identify relevant material for teaching and studying. However with an increasing amount of users as well as publisher generated content in recent days, teachers and students may want to refer to additional content apart from prescribed textbooks for their teaching/learning needs which is not labeled with learning standard instructions. Manually identifying the appropriate learning standard instruction for each learning content is time consuming and not scalable especially since learning standards frequently contain thousands of instructions, and subject to periodic revision. In this paper, we address the problem of automatically labeling digital learning content with the learning standards. Towards this goal, we first build semantic representations of the learning standard instructions using external knowledge sources such as Wikipedia and domain text books. These semantic representations are then used in a framework which utilizes structural constraints imposed by the hierarchy of the learning standards to assign labels to the learning materials. We demonstrate the usefulness of our approach on a collection of high school learning materials that were labeled by curriculum experts from a US school district according to a publicly available learning standard. The system developed has been deployed and is in use by the school district. To the best of our knowledge we are the first to attempt this novel task and develop such a system.

We present the motivation, design, and preliminary study of a mobile-enabled, blended learning technology called the Cognitive Learning Companion (CLC). The CLC concept emerged from field studies with teachers and students in Africa. These studies led to two key high-level requirements that shaped the design philosophy of CLC: 1) seamless support for different modes of learning and teaching in a blended scenario (where a student learns in part through face-to-face interactions with a teacher in a classroom, and in part through a combination of teacher and system supervision/direction outside of class) and 2) support for tracking student engagement and sentiment during this blended learning journey, and the interplay of these affective processes with concept and skill-building processes as part of learning. In this paper, we discuss findings from the field studies and outline our approach to address the requirements. We present the overall architecture and design of CLC. The first version supporting a core set of capabilities for blended learning has been implemented as mobile applications for teachers and students. We conducted a limited pilot to test the technology in an actual classroom setting. We also report on a usability study of CLC that demonstrates user awareness and support for data-driven cognitive decision-making in education.

Digitization of educational data and processes has enabled widespread development of technologies to support personalized learning. A key requirement in any personalized learning setup is to be able to accurately estimate students' weaknesses so they can be addressed appropriately during personalization. In this paper, we describe our work toward identifying K-12 students at risk of poor academic performance, with a special focus on 1) identifying specific components of varying granularity in the curriculum (such as subjects, topics, and concepts) that a student is finding difficult and 2) determining how early we can accurately estimate the risks. Such predictions could help teachers in planning effective personalized interventions for at-risk students and hence could help in achieving a long-term goal of minimal grade-level retentions and school dropouts. To predict performance risks, we use statistical models that utilize historical student data to learn patterns in their longitudinal journeys that correspond to performance risks. We describe in detail the risk prediction system we developed and its evaluation using data from one of the largest school districts in the United States. The experimental results demonstrate the ability of our system to make accurate risk predictions for subject-specific outcomes of varying granularity across different grade levels, early in a student's K-12 journey.

The education sector is experiencing an unprecedented shift in how students learn and progress through their education. With the large amounts of digital learning content available, teachers are increasingly turning to sources such as online tutorials and eBooks for their teaching needs. Schools often make use of a Learning Content Management System (LCMS) to store learning material, which is catalogued based on the curriculum followed in the school. However, the amount of content indexed in the LCMS is often limited by the ability to manually label and catalogue content. In this paper, we describe our LCMS called the Learning Content Hub (LCH), which not only offers the features of a traditional LCMS including document search and retrieval, document security, user role management, etc., but also provides the ability to automatically analyze and label documents. LCH provides a framework for easily extending the analytics support and exposes application programming interfaces that can be used to build custom education applications for which the content needs can be met using the LCH. We discuss experimental results of the analyzers in our system as well as our experience of deploying this system in a U.S. school district.

One-on-one instruction by teachers to their students improves learning outcomes; however, such kinds of instruction practices are not scalable even for smaller classrooms. In this paper, we present the Personalized Learning Pathways (PLP) application to enable personalized interventions for students by their teachers to improve student learning. The PLP application brings together learning content, learner skills, learning activities, and stakeholders (students, teachers, parents, etc.) to achieve specific learning objectives. We report the existing gaps with respect to intervention in learning systems and the requirements necessary to achieve personalized intervention. We also present the detailed architecture of the PLP, along with a preliminary study of PLP in Gwinnett County Public Schools for approximately 8 weeks. Field trial results demonstrate the effective use of PLP by the teachers for creating personalized learning plans for their students.

Several mobile-enabled solutions for education transformation have been deployed in Africa. Drawbacks of these systems include, a predominant focus on disseminating bulk learning content, student outcomes are measured solely on quantitative performance metrics, a lack of instrumentation to capture fine-grained user interaction data, and finally none of them seem to focus on capturing the contextual factors affecting learners. Hence, there are few chances to fully empower educators to create effective interventions for their students. In this paper, we present an adaptive event framework library that can be embedded within a blended learning environment. It enables the capturing of fine-grained learners activity stream data, including learners sentiment and context information. We conducted a limited controlled experiment to evaluate the effectiveness of the event framework. The real time visualizations provide useful insights to teachers in understanding their classroom and/or individual student engagement and progress. Finally, we outline challenges and preliminary solution how such system can be deployed in resource constrained environments.