A Proposal on the Exploitation of the Travel Information Dialogues Collected with the Evar System

In this report, an overview is rst presented of the various types of data collected over the 5-year period that the train information spoken dialogue system EVAR (Erkennen-Verstehen-Antworten-R uckfragen) has been operational. This data consists of system and user utterances, their orthographic and semantic representation, and protocols related to system performance and eeciency measures. The only extensive study of this data in order to evaluate the status of EVAR has been to date the one presented in Eckert1995]. A number of criteria were used therein which established, at the very least, the feasibility of the longer-term goals in developing EVAR. In addition, a number of its current shortcomings were identiied, as were directions for future research. On the basis of the observations in Eckert1995] and those collected a year later in the context of the EC Human Capital Mobility project MSDoS (Modelling Spontaneous Dialogues of Speech), a number of modiications were eeected on the various modules of EVAR in order to enhance the robustness and the functionality of the system. More speciically, the Word Recogniser, the Parser, and the Dialogue Manager of EVAR were adapted to treat the problem of words encountered in the user utterance that have not been included in the system's lexicon (OOV words). Moreover, the system was extended to cover 3 languages simultaneously, German, Slovak, and Slovenian in the context of the EC project SQEL (Spoken Queries in European Languages). Finally, EVAR was ported to a new, albeit related domain, that of Underground enquiries for the city of N urnberg. An initial evaluation of the system with the new OOV functionality was carried out in relation with the EUROSPEECH'97 Spoken Dialogue Systems Olympics. This was based on a number of questionnaires lled in by the attendees who tested the system over the phone from Rhodes. Although EVAR attained a medium score on average, it was the only participating system that could deal with unknown words, thereby rendering an absolute comparison with them impossible. The data collected will be exploited further for a more formal and rigourous evaluation based on the PARADISE scheme Walker et al.1997b]. This evaluation scheme could prove very useful both for assessing individual versions of EVAR, e.g. in diierent data collection phases using diierent dialogue strategies, and for estimating their functionality in comparison to one another. Finally, alternative sources of information are enumerated which can be considered in the future in studying the …