Research in the field of dialogue has been performed with the goal of making such more robust and easier to use in demanding situations. The term spoken dialogue systems signifies unified software containing speech-recognition, speech-synthesis, dialogue management, and ancillary components that enable human users to communicate, using natural language or nearly natural prescribed language, with other software that provide information and/or services.

The CALO Meeting Assistant (MA) provides for distributed meeting capture, annotation, automatic transcription and semantic analysis of multiparty meetings, and is part of the larger CALO personal assistant system. This paper presents the CALO-MA architecture and its speech recognition and understanding components, which include real-time and offline speech transcription, dialog act segmentation and tagging, topic identification and segmentation, question-answer pair identification, action item recognition, decision extraction, and summarization.

We use directed graphical models (DGMs) to automatically detect decision discussions in multi-party dialogue. Our approach distinguishes between different dialogue act (DA) types based on their role in the formulation of a decision. DGMs enable us to model dependencies, including sequential ones. We summarize decisions by extracting suitable phrases from DAs that concern the issue under discussion and its resolution. Here we use a semantic-similarity metric to improve results on both manual and ASR transcripts.

We address the problem of identifying words and phrases that accurately capture, or contribute to, the semantic gist of decisions made in multi-party human-human meetings. We first describe our approach to modelling decision discussions in spoken meetings and then compare two approaches to extracting information from these discussions. The first one uses an opendomain semantic parser that identifies candidate phrases for decision summaries and then employs machine learning techniques to select from those candidate phrases. The second one uses categorical and sequential classifiers that exploit simple syntactic and semantic features to identify words and phrases relevant for decision summarization.

The CALO Meeting Assistant provides for distributed meeting capture, annotation, automatic transcription and semantic analysis of multiparty meetings, and is part of the larger CALO personal assistant system. This paper summarizes the CALO-MA architecture and its speech recognition and understanding components, which include real-time and offline speech transcription, dialog act segmentation and tagging, question-answer pair identification, action item recognition, decision extraction, and summarization.

This paper addresses the problem of identifying action items discussed in open-domain conversational speech, and does so in two stages: firstly, detecting the subdialogues in which action items are proposed, discussed and committed to; and secondly, extracting the phrases that accurately capture or summarize the tasks they involve. While the detection problem is hard, we show that we can improve accuracy by taking account of dialogue structure. We then describe a semantic parser that identifies potential summarizing phrases, and show that for some task properties these can be more informative than plain utterance transcriptions.

This position paper describes an approach to building spoken dialogue systems for environments containing multiple human speakers and hearers, and multiple robotic speakers and hearers. We address the issue, for robotic hearers, of whether the speech they hear is intended for them, or more likely to be intended for some other hearer. We will describe data collected during a series of experiments involving teams of multiple human and robots (and other software participants), and some preliminary results for distinguishing robot-directed speech from human-directed speech. The domain of these experiments is Mars-analogue planetary exploration. These Mars-analogue field studies involve two subjects in simulated planetary space suits doing geological exploration with the help of 1-2 robots, supporting software agents, a habitat communicator and links to a remote science team. The two subjects are performing a task (geological exploration) which requires them to speak with each other while also speaking with their assistants. The technique used here is to use a probabilistic context-free grammar language model in the speech recognizer that is trained on prior robot-directed speech. Intuitively, the recognizer will give higher confidence to an utterance if it is similar to utterances that have been directed to the robot in the past.

NASA scientists in MDRS Crew 49 (April 23-May 7, 2006) field tested and significantly extended a prototype monitoring and advising system that integrates power system telemetry with a voice commanding interface. A distributed, wireless network of functionally specialized agents interacted with the crew to provide alerts (e.g., impending shut-down of inverter due to low battery voltage), access md interpret historical data, and display troubleshooting procedures. In practical application during two weeks, the system generated speech over loudspeakers and headsets lo alert the crew about the need to investigate power system problems. The prototype system adapts the Brahms/Mobile Agents toolkit to receive data from the OneMeter (Brand Electronics) electric metering system deployed by Crew 47. A computer on the upper deck was connected to loudspeakers, four others were paired with wireless (Bluetooth) headsets that enabled crew members to interact with their personal agents from anywhere in the hab. Voice commands and inquiries included: 1. What is the {battery | generator} {volts | amps | volts and amps}? 2. What is the status of the {generator | inverter | battery | solar panel}? 3. What is the hab{itat} {power usage | volts | voltage | amps | volts and amps}? 4. What was the average hab{itat} {amps | volts | voltage} since {AM | PM)? 5. When did the {generator | batteries} change status? 6. Tell {me I | everyone} when{ ever} the generator goes offline. 7. Tell {me | | everyone} when the hab{itat} {amps | volts | voltage} {exceeds | drops brelow} . 8. {Send | Take | Record} {a} voice note {(for | to} } {at }. This research demonstrates the principles of design in the context of use, investigating requirements through experimental use of prototype systems in an analog setting, and use of MDRS as a research facility for designing and implementing new systems.

We introduce a new interactive corpus exploration tool called InfoMagnets. InfoMagnets aims at making exploratory corpus analysis accessible to researchers who are not experts in text mining. As evidence of its usefulness and usability, it has been used successfully in a research context to uncover relationships between language and behavioral patterns in two distinct domains: tutorial dialogue (Kumar et al., submitted) and on-line communities (Arguello et al., 2006). As an educational tool, it has been used as part of a unit on protocol analysis in an Educational Research Methods course.

An ideal spoken dialogue system listens continually and determines which utterances were spoken to it, understands them and responds appropriately while ignoring the rest. This paper outlines a simple method for achieving this goal which involves trading a slightly higher false rejection rate of in domain utterances for a higher correct rejection rate of Out of Domain (OOD) utterances. The system recognizes semantic entities specified by a unification grammar which is specialized by Explanation Based Learning (EBL), so that it only uses rules which are seen in the training data. The resulting grammar has probabilities assigned to each construct so that overgeneralizations are not a problem. The resulting system only recognizes utterances which reduce to a valid logical form which has meaning for the system and rejects the rest. A class N-gram grammar has been trained on the same training data. This system gives good recognition performance and offers good out of domain discrimination when combined with the semantic analysis. The resulting systems were tested on a Space Station Robot Dialogue Speech Database and a subset of the OGI conversational speech database. Both systems run in real time on a PC laptop and the present performance allows continuous listening with an acceptably low false acceptance rate. This type of open microphone system has been used in the Clarissa procedure reading and navigation spoken dialogue system which is being tested on the International Space Station.

During the Mars Society's Mars Desert Research Station (MDRS) Rotation 49 (April 23May 7, 2006), we field tested and significantly extended a prototype monitoring and advising system for the habitat crew that integrates data from the MDRS power system. A distributed, wireless network of functionally specialized agents interact with the crew to provide alerts, access and interpret historical data (through voice command), and display troubleshooting procedures. In practical application during two weeks, the system generated speech over loudspeakers and headsets to alert the crew about the need to investigate generator-inverterbattery problems that had not yet come to their attention. This field test accomplished a milestone for the “CEV Agent-Mediated Situational Awareness” task in NASA’s Exploration Technology Development Program 12B, Human-Systems Interaction.

We present a system for extracting useful information from multi-party meetings and presenting the results to users via a browser. Users can view automatically extracted discussion topics and action items, initially seeing high-level descriptions, but with the ability to click through to meeting audio and video. Users can also add value: new topics can be defined and searched for, and action items can be edited or corrected, deleted or confirmed. These feedback actions are used as implicit supervision by the understanding agents, retraining classifier models for improved or user-tailored performance.

The Mars Society s Desert Research Station (MDRS) Rotation 38, April 3-17, 2005, was dedicated to field tests of NASA's Mobile Agents EVA communications system. MDRS provided an excellent, cost-effective venue for bringing together eighteen scientists and engineers from NASA Ames and Johnson Space Center, in an intensive two weeks of system integration and experiments. The Mobile Agents architecture and collaborative engineering methodology provides a flexible toolkit for configuring extravehicular activity (EVA) components, visualizing and formalizing EVA plans, and automating key supervisory functions.

NASA Ames’ Mobile Agents Architecture is a distributed agent-based architecture, which integrates diverse mobile entities in a wide-area wireless system for lunar and planetary surface operations. Software agents, implemented in the Brahms multiagent language, run in Brahms virtual machines onboard laptops for space suits, robots, and surface habitats. “Personal agents” support the habitat crew and surface astronauts, as well as the their robotic assistant. People communicate with their personal agents via a speech dialogue system and via a meeting-capture hyperlink database tool.

We have developed and tested an advanced EVA communications and computing system to increase astronaut self-reliance and safety, reducing dependence on continuous monitoring and advising from mission control on Earth. This system, called Mobile Agents (MA), is voice controlled and provides information verbally to the astronauts through programs called personal agents. The system partly automates the role of CapCom in Apollo-including monitoring and managing EVA navigation, scheduling, equipment deployment, telemetry, health tracking, and scientific data collection. EVA data are stored automatically in a shared database in the habitat/vehicle and mirrored to a site accessible by a remote science team. The program has been developed iteratively in the context of use, including six years of ethnographic observation of field geology. Our approach is to develop automation that supports the human work practices, allowing people to do what they do well, and to work in ways they are most familiar. Field experiments in Utah have enabled empirically discovering requirements and testing alternative technologies and protocols. This paper reports on the 2004 system configuration, experiments, and results, in which an EVA robotic assistant (ERA) followed geologists approximately 150 m through a winding, narrow canyon. On voice command, the ERA took photographs and panoramas and was directed to move and wait in various locations to serve as a relay on the wireless network. The MA system is applicable to many space work situations that involve creating and navigating from maps (including configuring equipment for local topology), interacting with piloted and unpiloted rovers, adapting to environmental conditions, and remote team collaboration involving people and robots.

In this paper, we present research toward ontology-based understanding of discourse in meetings and describe an ontology of multimodal discourse designed for this purpose. We investigate its application in an integrated but modular architecture which uses semantically annotated knowledge of communicative meeting activity as well as discourse subject matter. We highlight how this approach assists in improving system performance over time and supports understanding in a changing and persistent environment. We also describe current and future plans for ontology-driven robust naturallanguage understanding in the presence of the highly ambiguous and errorful input typical of the meeting domain.