Dialogue-based tutoring platforms have shown great promise in helping individual students improve mastery. Short answer grading is a crucial component of such platforms. However, generative short answer grading using the same platform for diverse disciplines and titles is a crucial challenge due to data distribution variations across domains and a frequent occurrence of non-sentential answers. Recent NLP research has introduced novel deep learning architectures such as the Transformer, which merely uses self-attention mechanisms. Pre-trained models based on the Transformer architecture have been used to produce impressive results across a range of NLP tasks. In this work, we experiment with fine-tuning a pre-trained self-attention language model, namely Bidirectional Encoder Representations from Transformers (BERT) applying it to short answer grading, and show that it produces superior results across multiple domains. On the benchmarking dataset of SemEval-2013, we report up to 10% absolute improvement in macro-average-F1 over state-of-the-art results. On our two psychology domain datasets, the fine-tuned model yields classification almost up to the human-agreement levels. Moreover, we study the effectiveness of fine-tuning as a function of the size of the task-specific labeled data, the number of training epochs, and its generalizability to cross-domain and join-domain scenarios.

We present a conversational ITS (intelligent tutoring system) called Watson Tutor, which interacts one-on-one with students via natural language chat. One-on-one tutoring is an effective way to learn deeper knowledge, and the exchanges between human tutors and students can improve learning gains significantly. Our Tutor leverages ITS research and advances in natural language processing to create an interactive dialogue that mimics the questions and feedback responses of a good human tutor.Students engage with Watson Tutor in a Socratic dialogue by answering or asking questions about a specific learning objective. Each student's understanding of the objective is tracked using a mastery score, which is updated when assessing their answers. The tutoring strategy is adaptive - students are guided through different concepts using questions, hints, pumps, etc., depending on their mastery. Since the Tutor is often anthropomorphized by students, we crafted its tone and personality to be genuine, engaging and nonjudgmental.The framework for Watson Tutor is designed to scale up efficiently to a large number of domains. Our first large-scale application augments online textbooks on the Pearson Revel platform, across the domains of sociology, US government, and public speaking. Thousands of undergraduate students have used Watson Tutor on Revel in the fall semester of 2018, and more than half of the students surveyed indicated that it helped them to understand the content more than reading alone.In this paper we introduce a new set of functionalities, which aim to improve learning interactions with Watson Tutor. These are based on transcripts and student surveys from pilot tests: 1) a feedback system for students to correct the Tutor when it makes a mistake, 2) interactive concept grouping activities to test student understanding, and 3) text linking of relevant paragraphs to assist unprepared students.We've developed a feedback system that allows students to vent their frustration in a positive way, when Watson Tutor makes a mistake interpreting or assessing student responses. Students can submit specific corrections by annotating the Tutor's utterances in the context of each dialogue turn. The corrections are reviewed by other students, and if approved, they are automatically used to retrain the Tutor's natural language understanding. In addition, the feedback system can also be used to flag rewrites of domain-specific content that causes issues for a significant number of students.Watson Tutor can auto-generate concept maps for each learning objective using relevant concepts extracted from domain models. Clusters of related concepts are also identified and used to create visual concept grouping activities, in which students can drag-and-drop concepts. These activities enable students to self-test their understanding, and to receive feedback from the Tutor. This leads to more effective learning of concepts compared to passive observation of a concept map.Text linking is a type of scaffolding that shows relevant paragraphs to students who are unprepared to answer a question. Watson Tutor displays these paragraphs embedded in the chat either after students give an incorrect answer, or if they request to see relevant text. Up to three paragraphs are extracted a priori for each learning objective from the textbook, using semantic search. We propose to validate the ranking of the paragraphs using student votes or subject matter expert reviews.

As artificially intelligent systems make their foray into the day-to-day educational experiences of students, we need to pay careful attention to the relationship between the system and the student. In this article, the authors discuss designing the personality of a virtual tutoring system called IBM Watson Tutor. The AI personality is key to the students' level of engagement and satisfaction with their experience. The designers have examined the qualities of effective human tutors, and used that insight as inspiration.

There are significant challenges involved in the design and implementation of a dialog-based tutoring system (DBT) ranging from domain engineering to natural language classification and eventually instantiating an adaptive, personalized dialog strategy. These issues are magnified when implementing such a system at scale and across domains. In this paper, we describe and reflect on the design, methods, decisions and assessments that led to the successful deployment of our AI driven DBT currently being used by several hundreds of college level students for practice and self-regulated study in diverse subjects like Sociology, Communications, and American Government.

IBM and Pearson have partnered to develop dialogue-based intelligent tutoring systems at an unprecedented scale. We leveraged the decades-long research in intelligent tutoring systems (specifically dialogue based tutoring systems) and advances in machine learning and natural language processing to create a Watson dialogue-based tutor (WDBT). WDBT is currently being used by hundreds of students across multiple institutions. This paper describes our plans for preliminary evaluations of WDBT. Our formal evaluations begin in Spring 2018 and we will present findings shortly thereafter.

In 2016, IBM and Pearson announced a partnership to deliver a next generation learning service in the form of a dialogue-based tutor. Dialogue-based tutoring systems have demonstrated efficacy, but they are difficult to design and scale across domains. We have developed a framework for enabling digital courseware with a dialogue-based tutoring experience that can be applied to new domains with additional domain-specific content, but without re-design of the conversation flow or use case. The framework uses a content model that’s consistent across domains, which enables a general dialogue-based tutoring strategy. We identify several challenges to this approach, as well as recommendations for future work. The Pearson-IBM partnership In 2016, IBM and Pearson announced a partnership to deliver next generation learning services in the form of a digital tutoring system. The aim of this partnership was to create learning experiences powered by Pearson’s highquality content and IBM’s Watson technologies. While there have been many successful intelligent tutoring systems (ITSs), our challenge was scaling the process across multiple disciplines and titles. This paper is organized into 4 parts. First, we briefly review the value of dialogue-based tutoring systems. Second, we describe what we call the Watson dialogue-based tutor. Third, we will describe some of our approaches to scaling and evaluating the Watson dialogue-based tutor. Finally, describe the limitations of our approach and recommendations for future work. Dialogue-based tutoring systems Dialogue-based tutoring systems (DBTs) are an approach to ITSs that create a learning experience driven by natural language dialogue and classification of student natural language responses (e.g., Graesser, 2011). DBT conversations can be described as Socratic because the tutor guides the student through concepts via dialogue moves, which can include questions, hints, and other prompts. DBTs have been claimed to support a variety of learning principles and strategies, like encouraging constructive behaviors and self-explanations (M. T. H. Chi, 2009), deep reasoning questions (e.g., Graesser & Person, 1994), and conceptual understanding through scaffolding (e.g., VanLehn, 2011). DBTs require students to construct natural language responses, which can have a positive impact on memory and comprehension of source text (e.g., McNamara, 1992). DBTs can also provide immediate feedback to facilitate learning (Shute, 2008). For example, AutoTutor is a DBT, i.e., an ITS that initiates discourse with a student. The discourse patterns of the earliest AutoTutor were inspired by analyses of approximately 100 hours of non-expert human tutoring interactions (Graesser, 2011), which showed that students in need of tutoring are not active, self-regulated learners, and are not aware of their knowledge deficits. This affects how students converse: they do not effectively take command of the tutorial agenda and typically ask only 6-8 genuine information-seeking questions per hour. In contrast, tutors set 100% of the agenda, introduced 93% of the topics, presented 82% of examples, and asked 80% of the questions. Tutors do this by invoking a curriculum script of topics, problems, questions, and examples to drive a Socratic tutoring dialogue with students. Based on this tutor analysis, AutoTutor was designed to control the conversation through an expectation-misconception discourse model of tutoring (Graesser, 2011). This consists of a set of anticipated correct ideal answers (expectations) and a set of invalid answers frequently expressed by students (misconceptions). AutoTutor follows this design in a five-step tutoring framework: (1) tutor poses a question/problem, (2) student attempts to answer, (3) tutor provides brief evaluation as feedback, (4) collaborative interaction to improve the answer, (5) tutor checks if student understands. Efficacy of DBTs Steenbergen-Hu and Cooper (2014) conducted a meta-analysis of 39 studies evaluating the use of ITSs (including DBTs) in higher education. The researchers found an overall, moderate, positive effect (g = .35) favoring the use of ITS over other instructional conditions. When compared specifically to alternatives that were either “self-reliant learning activities” or no-treatment conditions, the use of ITSs appeared to offer a large advantage (g =.86). AutoTutor in particular has shown significant learning gains over non-interactive learning materials in a variety of math and science domains: computer literacy, physics, biology, and critical thinking (Graesser, 2011). Typically, higher gains were found for more complex questions, such as “how” and “why” questions, versus shallow questions, such as “who” or “what” questions (Nye, Graesser and Hu, 2014). Scalability of DBTs While DBTs have demonstrated a wide range of possible behaviors and pedagogical strategies, building a DBT for a new domain, course, or textbook is a non-trivial task. Even when the use case and learning goals are clearly defined, creating the necessary domain models for these tutoring systems can be very challenging for domain experts. This is a general problem for ITSs, which is why the researchers behind the most widely adopted tutoring systems have also developed authoring tools (e.g., Aleven, McLaren, Sewall, & Koedinger, 2009). Watson DBT Watson dialogue-based tutor (WDBT) follows the design of AutoTutor in many respects, but with a content creation and iterative design cycle to support application to new domains. WDBT begins with deep reasoning questions and then provides hints to assist students to give a response that matches a set of assertions or knowledge components. An example transcript from an interaction with WDBT is shown in Table 1. WDBT is made up of 6 components to achieve this functionality: (1) Domain Model, (2) Dialogue Content, (3) Natural Language Response Classification, (4) Question Answering, (5) Learner Modeling, and (6) Dialogue Management. Domain Model The domain model defines the knowledge and skills we want students to learn. We create a domain model for a specific title (i.e. textbook) that breaks down the knowledge into educational objectives consisting of learning objectives and enabling objectives. Learning objectives are broad learning goal statements e.g., “Analyze physical changes that occur in middle adulthood.” Enabling objectives are more granular learning goal statements that support the learning objective e.g., “Identify the physical benchmarks of change in middle adulthood.” Educational objectives serve as the foundation for creating content and assessment in Pearson, so the same framework was used to enable WDBT. The domain model also contains misconception statements that are aligned to educational objectives. Domain experts create the domain models, choosing learning and enabling objectives that are particularly difficult for students or that would benefit from a conversational learning experience. For each title, domain models typically contain 24 learning objectives and 110 enabling objectives (about 5 enabling objectives per learning objective). Because the learning objectives follow the general organization of the title, the domain model acts as an extension of the core content in the title. Dialogue Content The dialogue content for WDBT consists of a mix of content created manually by domain experts, and automatically extracted content. Subject matter experts (SMEs) are responsible for authoring: • Main question and answer pairs aligned to both learning and enabling objectives. The questions are broad, high-level conceptual questions, to assess deep comprehension of source text. WDBT delivers these questions to students. The corresponding answers represent ideal responses and are used to evaluate student answer attempts. • Hint questions and assertions for every main question and answer pair. Assertions are independent knowledge components that make up a main answer. Hints are intended to elicit an assertion from a student. WDBT delivers these hint questions to students. • Misconception statements and corresponding feedback messages for every applicable learning and enabling objective. In the event that response classification detects that a student is expressing a misconception, WDBT delivers the corresponding feedback message. Other forms of dialogue content are automatically or semi-automatically extracted: • Question-answer pairs based on key terms and other sources within the textbook. • Concept groupings based on key concepts found in the unstructured text. These groupings are automatically extracted (1) but validated by a subject matter expert. • Fill in the blank prompts that are dynamically created within the dialogue using an assertion and response classification gap analysis. This approach to content organization is applied across titles to facilitate faster adaptation to new titles, but it is not without its limitations. There is still a considerable amount of human effort required for this process, although it is much less than what would be required if new conversational scripts (and associated content) were created for each domain. This points to the inherent trade-off between human effort savings and dialogue richness that can be addressed by better automatic content extraction techniques. Table 1: An excerpt of an interaction between a learner and Watson Dialogue-Based Tutor (WDBT). Dialogue moves and responses from WDBT are left aligned. Learner responses are right aligned. What is the nature of gender differences? its is common for gender to be defined by sex Looks like I have something different... How do sociologists define sex? biological differneces Well said. Sex refers to the biological differences between men and women. There are 2 key facts left in this review. What is gender? idk No problem. I’ll help you get this! Gender refers to socially created differences between men and women. How are gender diff

We're developing a feedback system that uses student annotations embedded in the dialogue to improve the natural language understanding and the content of dialogue-based tutors in multiple knowledge domains. The annotations are collected in the context of each dialogue turn, and are reviewed by other students and instructors. If approved by reviewers, the changes resulting from the feedback are automatically implemented by retraining natural language classifiers, or by updating domain-specific dialogue content. The number of students using our Watson dialogue-based tutors is projected to grow from hundreds in current pilot trials to tens of thousands in the commercialization phase, generating a large-scale feedback effect. We expect that the number of retraining events will peak initially, and then drop off as the tutor training becomes more comprehensive for each knowledge domain. This paper discusses our progress and identifies the challenges that we'll be addressing in future work.

We present a practical application of serverless technology to implement a key component of a personalized tutoring system. We employ OpenWhisk[2] to orchestrate a set of microservices tasked with different responsibilities in the larger system. The tutor, which is trained on a specific set of content, is embodied as a multimodal, stateful dialog system. This desired experience brings with it unique challenges, prime among them being real-time interaction and the inherently stateful nature. We discuss the drivers for selecting serverless technology, the specific design choices we made and the issues we have faced. This work is in progress, and we conclude with some notes on paths we plan to explore as we continue to evolve the system and exploit this exciting new computing paradigm to help us achieve our goals.

Recent advances in artificial intelligence and natural language processing greatly enhance the capabilities of intelligent tutoring systems. However, gathering a subject-appropriate corpus of training data remains challenging. In order to address this issue, we present a system based on a hybrid Wizard-of-Oz technique, which enables cognitive systems to work in tandem with a human operator (the "wizard"), to enhance collection of dialog variants.

Hybrid process models are considered an attractive approach for modeling knowledge-intensive processes. A hybrid process model combines both imperative and declarative modeling, which can handle both the structured and the flexible parts of a business process. However, it is difficult and time-consuming to create and refine a hybrid process model due to its structure complexity and case variability. This paper introduces the Case Analytics Workbench, an end-to-end system to accelerate hybrid process model creation and evolution by combining declarative and imperative process mining, event log clustering and human interaction in a cloud environment. We validated the effectiveness and applicability of our system by performing two case studies from insurance and health care industry respectively.

Provenance of scientific data will play an increasingly critical role as scientists are encouraged by funding agencies and grand challenge problems to share and preserve scientific data. But it is foolhardy to believe that all human processes, particularly as varied as the scientific discovery process, will be fully automated by a workflow system. Consequently, provenance capture has to be thought of as a problem applied to both human and automated processes. The unmanaged workflow is the full human-driven activity, encompassing tasks whose execution is automated by an orchestration tool, and tasks that are done outside an orchestration tool. In this chapter we discuss the implications of the unmanaged workflow as it affects provenance capture, representation, and use. Illustrations of capture include multiple experiences with unmanaged capture using the Karma tool. Illustrations of use include defining workflows by suggesting additions to workflow designs under construction, reconstructing process traces, and using analysis tools to assess provenance quality.

Semi-structured processes arise extensively in various industries such as government, insurance, banking and healthcare and are characterized by their flexibility and data-driven nature. Case management systems are evolving to handle the growing demand for providing operational support to knowledge workers involved in these processes. While enabling flexibility in human actions is an integral part of these systems, this capability increases the potential for errors in case handling. We propose a new paradigm of case health aimed at providing knowledge workers and supervisors a case health service to continuously and objectively assess both the current state of a case instance and provide a prognosis for the future state of the instance. The case health service is composed of constituent services each of which provide case health indicators and can be combined in flexible ways to determine the overall health of an instance. We have implemented case health with respect to a simulated credit card dispute scenario modeled in IBM Case Manager and evaluated the utility of computing the case health of running instances of the credit card dispute scenario. Results indicate the utility of incorporating case health in a case management system in terms of early detection of undesired outcomes and abnormal execution patterns in running case instances.

Most human centric business activities, like the handling of an insurance claim or the design of an IT solution, do not follow a formal process model word-by-word. While there are often underlying processes in place, the real business operation typically encompasses a wider scope of actions. It includes unstructured segments of human activities, additional documents and interactions, the integration of remote systems and services, and the reactions to exceptional situations. With the case management approach, the knowledge worker in the center takes control and acts as an orchestrator of available services rather than being a part of an assembly line. This paper presents how data-centric web services can be used to build a service-oriented case management application, and how advanced analytics can leverage historical information to improve the effectiveness of business processes execution.

Advanced case management gives enterprises the flexibility to organize their business operations in an efficient and transparent manner while providing knowledge workers the opportunity to proceed towards their goals and to react to exceptions as required in each individual situation. Powerful analytics based on pervasive observation of end-to-end case executions help to detect exceptional situations, determine possible derailment, and thus to prevent undesired outcomes. This paper demonstrates the application of proven technologies derived from real world examples to determine the health of a particular case.