Automated Essay Scoring (AES) can be used to automatically generate holistic scores with reliability comparable to human scoring. In addition, AES systems can provide formative feedback to learners, typically at the essay level. In contrast, we are interested in providing feedback specialized to the content of the essay, and specifically for the content areas required by the rubric. A key objective is that the feedback should be localized alongside the relevant essay text. An important step in this process is determining where in the essay the rubric designated points and topics are discussed. A natural approach to this task is to train a classifier using manually annotated data; however, collecting such data is extremely resource intensive. Instead, we propose a method to predict these annotation spans without requiring any labeled annotation data. Our approach is to consider AES as a Multiple Instance Learning (MIL) task. We show that such models can both predict content scores and localize content by leveraging their sentence-level score predictions. This capability arises despite never having access to annotation training data. Implications are discussed for improving formative feedback and explainable AES models.

Automated essay scoring (AES) allows writing to be assigned in large courses and can provide instant formative feedback to students. However, creating models for AES can be costly, requiring the collection and human scoring of hundreds of essays. We have developed and are piloting a web-based tool that allows instructors to incrementally score responses to enable AES scoring while minimizing the number of essays the instructors must score. Previous work has shown that techniques from the machine learning subfield of active learning can reduce the amount of training data required to create effective AES models. We extend those results to a less idealized scenario: one driven by the instructor's need to score sets of essays, in which the model is trained iteratively using batch mode active learning. We propose a novel approach inspired by a class of topological methods, but with reduced computational requirements, which we refer to as topological maxima. Using actual student data, we show that batch mode active learning is a practical approach to training AES models. Finally, we discuss implications of using this technology for automated customized scoring of writing across the curriculum.

The power and pervasiveness of conceptual metaphor can be harnessed to expand the class of textual entailments that can be performed in the Recognizing Textual Entailment (RTE) task and thus improve our ability to understand human language and make the kind of textual inferences that people do. RTE is a key component for question understanding and discourse understanding. Although extensive lexicons, such as WordNet, can capture some word senses of conventionalized metaphors, a more general capability is needed to handle the considerable richness of lexical meaning based on metaphoric extensions that is found in common news articles, where writers routinely employ and extend conventional metaphors. We propose adding to RTE systems an ability to recognize a library of common conceptual metaphors, along with scripts. The role of the scripts is to allow entailments from the source to the target domain in the metaphor by describing activities in the source domain that map onto elements of the target domain. An example is the progress of an activity, such as a career or relationship, as measured by the successful or unsuccessful activities in a journey towards its destination. In particular we look at two conceptual metaphors: IDEAS AS PHYSICAL OBJECTS, which is part of the Conduit Metaphor of Communication, and ABSTRACT ACTIVITIES AS JOURNEYS. The first allows inferences that apply to physical objects to (partially) apply to ideas and communication acts (e.g., "he lobbed jibes to the comedian"). The second allows the progress of an abstract activity to be assessed by comparing it to a journey (e.g., "his career was derailed"). We provide a proof of concept where axioms for actions on physical objects, and axioms for how physical objects behave compared to communication objects, are combined to make correct RTE inferences in Prover9 for example text-hypothesis pairs. Similarly, axioms describing different states in a journey are used to infer the current progress of an activity, such as whether it is succeeding (e.g., "steaming ahead"), in trouble (e.g., "off course"), recovering (e.g., "back on track"), or irrevocably failed (e.g., "hijacked").

We discuss extensions to a controlled natural language allowing spatiotemporal expressions to be interpreted as fuzzy logic functions. The extensions first required new sentence templates. Next, changes to a GPSG parser modified its lexicon, and then extended its parsing and logical form rules to allow user-defined spatial and temporal constraints to be extracted. The sentence templates ground user-defined culturally-specific times and places to boundaries surrounding prototypical ideals. Query points, defined by location and time, are compared to these definitions using Gaussians centered at prototypical 'ideal' times or places. The Gaussians provide soft fall-off at the boundaries. Fuzzy logic operators allow larger expressions to be interpreted, analogous to Boolean combinations of terms. The mathematically-interpreted spatiotemporal terms act as domain features for a machine learning algorithm. They allow easy specification (compared to programming) of basis functions for an inverse reinforcement learning algorithm that detects anomalous vehicle tracks or suspicious agent behavior.

We discuss extensions to the controlled natural language CPL to explicitly define and reason about temporal and spatial concepts. New sentence templates are added to define culturally-specific temporal and spatial names. Two kinds of queries are supported: (1) proximity queries, and (2) relational (e.g., overlap or containment) queries. Proximity queries require concepts grounded in terms of absolute time (UTC times) or location (latitude and longitude). Relational queries can be answered using Allen's Interval Algebra, for time; or Region Connection Calculus (RCC-8), for regions. Our motivation for handling proximity queries is to allow easy feature definition for a machine learning algorithm that detects anomalous vehicle travel. Our motivation for addressing more general, relational queries is to improve CPL's modeling of the temporal and spatial relationships of events and objects that occur in sentences, in order to further improve its semantic reasoning capabilities in applications such as Q&A.

Statistical Relational Learning (SRL) provides a common language to express diverse kinds of learner models for intelligent tutoring systems that are broadly applicable across different domains or applications. It provides new more expressive user modeling capabilities, such as the ability to express (1) probabilistic user models that model causal influence, with feedback loops allowed, (2) logical rules with exceptions, and (3) both hard and soft constraints in first-order logic. Practically, for example, SRL learner models can facilitate building team user models and user models for collaborative instruction by leveraging social network analysis. They can also facilitate building learner models for affective computing that simultaneously model inferences from affect to cognition and cognition to affect.

In this paper we describe two quite different philosophies used in developing controlled languages (CLs): A "naturalist" approach, in which CL interpretation is treated as a simpler form of full natural language processing; and a "formalist" approach, in which the CL interpretation is "deterministic" (context insensitive) and the CL is viewed more as an English-like formal specification language. Despite the philosophical and practical differences, we suggest that a synthesis can be made in which a deterministic core is embedded in a naturalist CL, and illustrate this with our own controlled language CPL. In the second part of this paper we present a fictitious debate between an ardent "naturalist" and an ardent "formalist", each arguing their respective positions, to illustrate the benefits and tradeoffs of these different philosophies in an accessible way.

One of the big challenges in understanding text, i.e., constructing an overall coherent representation of the text, is that much information needed in that representation is unstated (implicit). Thus, in order to "fill in the gaps" and create an overall representation, language processing systems need a large amount of world knowledge, and creating those knowledge resources remains a fundamental challenge. In our current work, we are seeking to augment WordNet as a knowledge resource for language understanding in several ways: adding in formal versions of its word sense definitions (glosses); classifying the morphosemantic links between nouns and verbs; encoding a small number of "core theories" about WordNet's most commonly used terms; and adding in simple representations of scripts. Although this is still work in progress, we describe our experiences so far with what we hope will be a significantly improved resource for the deep understanding of language.

Our primary interest in semantic knowledge discovery and use is in the context of natural language understanding. One of the big challenges for language understanding is that much information that text is intended to convey is not explicitly stated. Rather, the reader constructs a mental model of the scene described by the text, including many "obvious" features that were not explicitly mentioned, using large amounts of general knowledge about the world:

To score well in RTE3, and even more so to create good justifications for entailments, substantial lexical and world knowledge is needed. With this in mind, we present an analysis of a sample of the RTE3 positive entailment pairs, to identify where and what kinds of world knowledge are needed to fully identify and justify the entailment, and discuss several existing resources and their capacity for supplying that knowledge. We also briefly sketch the path we are following to build an RTE system (Our implementation is very preliminary, scoring 50.9% at the time of RTE). The contribution of this paper is thus a framework for discussing the knowledge requirements posed by RTE and some exploration of how these requirements can be met.

Game manuals and tutorial scenarios are insufficient for new players to learn games of deep complexity such as highly realistic tactical simulations of modern battlefields. Adding post-game after-action reviews improves the situation, but these typically do not provide guidance during the mission and tend to focus on quantitative feedback, rather than specifics about what the player did wrong and how to improve. Intelligent tutoring system (ITS) technology provides a higher level of interactivity and a more specific qualitative analysis to guide players during game play. This use of an AI technology is demonstrated with the integration of an ITS component with the tactical simulation Armored Task Force (ATF) resulting in a combined system called the the Virtual Combat Training Center (V-CTC). V-CTC simulates the Army's combat training center at Fort Irwin and its instructors, called observer / controllers. The ATF game itself was modified to send an event stream over TCP-IP sockets to the ITS component, which interprets the events and acts accordingly. V-CTC was originally intended for a military context: either classroom use, field instruction, or embedded deployment. However, in non-military games, tutors (or non-player characters acting in that role) may well enhance the gaming experience of players. Such players might otherwise become frustrated with learning very challenging games, or simply fail to appreciate the tactical possibilities and depth of strategy possible in a well-designed game.

The classic ITS conceptual diagram with boxes for domain, user, and pedagogical modeling encourages a monolithic architecture. Furthermore, it focuses on knowledge and simulation of tutoring capability while deemphasizing knowledge and simulation of the task environment. However, the actual task environment may require complex domain and graphical simulations whose software investment vastly exceeds the ITS, even though these simulations are commingled with other aspects of the ITS in the classic box-diagram view. A more useful architecture for combining intelligent tutoring capabilities with complex pre-existing simulations (e.g., tactical simulations) views the ITS system as a tutor-simulation hybrid. The ITS consists of two communicating components, a simulation agent and a tutor agent. The tutor agent models the pedagogical capabilities of an expert instructor. The simulation agent is the simulation wrapped to send an event stream to the tutor, and to send and receive requests. This hybrid architecture breaks up an ITS monolith into two reusable components. It allows the tutor agent to be reused in multiple teaching applications with different simulations and vice versa. The V-CTC system, which provides a reusable tutoring component to provide ITS capabilities for tactical simulations, illustrates this architecture.

Ontologies provide advantages of knowledge reusability, sharing, and greater robustness when used to build large knowledge-based applications. Unfortunately, translating between English statements and a specific ontology requires skill in knowledge engineering and all understanding of formal logic and the ontology itself. A knowledge engineer must be familiar with the concepts in the ontology, the fine distinctions between terms. and the specific way the ontology conceptualizes the world.We are developing a tool, CELT (Controlled English to Logic Translation), to enable non-programmers to add knowledge expressed in terms of an ontology. CELT is all automatic translation tool to convert controlled English to KIF formulas using ontologies built with the Suggested Upper Merged Ontology (SUMO). WordNet provides a base lexicon and a default preference for word senses. We do not expect CELT to obviate the need for knowledge engineers but to instead better leverage their time, as current machine translation tools assist professional human translators.CELT uses Discourse Representation Theory to handle the translation of multiple sentences, the use of logical quantifiers, and the resolution of anaphoric referents. Individual sentences are parsed using a Definite Clause Grammar augmentcd with feature grammar extensions.CELT is domain-independent but can be customized for particular domains by providing domain -specific ontologies and lexicons. The lexicons call specify both technical terms and domain-specific preferred word senses for common terms. CELT translates sentences to assertions and queries for a first-order logic theorem prover.

: The primary operational challenge facing the U.S. military for the foreseeable future is that military planners cannot know a priority specifically where our military will be deployed, who our adversaries will be, and what weapons and systems will be necessary for a given mission. There is an increased emphasis on both the broader spectrum of military missions, such as peacekeeping, and a broader range of potential adversaries, such as terrorists. Requirements for future education and training systems must take due account of this expanding range of operational conditions and needs. Our forces must be highly adaptive learning forces that organize to meet threats effectively and rapidly. In particular, commanders must be able to make and implement decisions in a timely, efficient, and effective manner, most often with very limited information in a constantly changing, complex, and dangerous environment. This capability is often referred to as cognitive readiness--ensuring that the warfighter is mentally prepared for accomplishing the mission and is performing at their optimal performance level.

Bayesian modeling techniques provide a rigorous formal approach to student modeling in contrast to earlier ad hoc or certainty-factor based approaches. Unfortunately, the application of Bayesian modeling techniques is limited due to computational complexity, conditional independence requirements of the model, and difficulties with knowledge acquisition. The approach presented here infers a student model from performance data using a Bayesian belief network. The belief network models the relationship between knowledge and performance for either test items or task actions. The measure of how well a student knows a skill is represented as a probability distribution over skill levels. Questions or expected actions are classified according to the same categories by the expected difficulty of answering them correctly or selecting the correct action. With this model only a small number of parameters are required: an expected probability distribution for the skill categories, and the expected conditional probabilities for slips and lucky guesses. By limiting the complexity of the user model in this way, and to a single level of propagation, updating can be performed in time linear to the number of test items and typically only about a half a dozen model parameters are required. Test items can be added or taken away without changing these parameters, provided only that their skill level is specified. We contrast this approach with other uses of Bayesian models in intelligent tutoring systems for diagnostic plan recognition or assessment. Other assessment approaches typically require 100's of conditional probabilities or an explicit authoring of the structure of the belief network; this approach requires neither.

: This intelligent tools and instructional simulations project was an investigation into the utility of a knowledge-based performance support system to support learning and on-task performance for using desktop applications. The Desktop Associate is a multiagent system that couples a pedagogical agent with a monitoring agent to help users with desktop applications such as spreadsheet programs, word processors, and Internet browsers. It is intented to carry out common business tasks such as evaluating loans, creating a corporate newsletter, or downloading Internet software. The Desktop Associate implementation described in this report is a proof-of-concept design and development effort. It has not been formally tested but encouraging results were received from an early evaluation of the design, development tasks, and its potential capabilities.