This paper presents our achievements after 18 months of the ALFFA project dealing with African languages technologies. We focus on a multilingual calculator (Android app) that will be demonstrated during the Show and Tell session.

Motivation -- To reduce user linguistic variability in human-system interaction. Research approach -- An experiment was conducted in which 72 participants interacted over the phone with a simulated natural language dialogue system. The main manipulation concerned the lexical content and the structure of the message prompts. Findings/Design -- The results confirm that users align with the system on the lexical and structural levels in human-system dialogue. However, the strength of the syntactic alignment depends on the content of the prime. Research limitations/Implications -- This experiment should be replicated user a greater number of different prime system messages. Originality/Value -- By manipulating prime content, this study allows investigating alignment strength as a function of word frequency and user preferences. Take-away message -- Lexical and syntactic priming can be used to reduce user linguistic variability in human-system interaction, but the strength of these phenomena depends on the content of the prime.

This document is a short report to accompany the Prototype Deliverable D1. 1.3, due at month 36 of the CLASSIC project. This prototype is an enhancement of the industrial Dialogue Manager with another module called the Learning Manager that provides the Spoken Dialogue System with adaptive online reinforcement learning capabilities. This document reviews the foundations of the reinforcement learning theory developed in the deliverable D1. 1.1 and it gives an overview of the API between those two modules.

This paper shows how the convergence between design and monitoring tools, and the integration of a dedicated reinforcement learning can be complementary and offer a new design experience for Spoken Dialogue System (SDS) developers. Most industrial SDS developers use a graphical tool to implement the dialogue strategies. First, this article proposes to integrate dialogue logs into this design tool, so that it constitutes a monitoring tool as well, by revealing call flows and their associated Key Performance Indicators (KPI). Second, the SDS developer is opened the possibility of designing several alternatives and of visually comparing his design choice performances. Third, reinforcement learning technique is integrated to automatically optimise the SDS choices. The design/monitoring tool helps the SDS developers to understand and analyse the user behaviour, with the assistance of the learning algorithm. The SDS developers can then confront the different KPI and control the further SDS choices by removing or adding alternatives. Index Terms : Dialogue Design, Online Learning, Spoken Dialogue Systems, Monitoring Tools

With a view to rationalise the evaluation process within the Orange Labs spoken dialogue system projects, a field audit has been realised among the various related professionals. The article presents the main conclusions of the study and draws work perspectives to enhance the evaluation process in such a complex organisation. We first present the typical spoken dialogue system project lifecycle and the involved communities of stakeholders. We then sketch a map of indicators used across the teams. It shows that each professional category designs its evaluation metrics according to a case-by-case strategy, each one targeting different goals and methodologies. And last, we identify weaknesses in the evaluation process is handled by the various teams. Among others, we mention: the dependency on the design and exploitation tools that may not be suitable for an adequate collection of relevant indicators, the need to refine some indicators' definition and analysis to obtain valuable information for system enhancement, the sharing issue that advocates for a common definition of indicators across the teams and, as a consequence, the need for shared applications that support and encourage such a rationalisation.

Building an industrial spoken dialogue system (SDS) requires several iterations of design, deployment, test, and evaluation phases. Most industrial SDS developers use a graphical tool to design dialogue strategies. They are critical to get good system performances, but their evaluation is not part of the design phase. We propose integrating dialogue logs into the design tool so that developers can jointly monitor call flows and their associated Key Performance Indicators (KPI). It drastically shortens the complete development cycle, and offers a new design experience. Orange Dialogue Design Studio (ODDS), our design tool, allows developers to design several alternatives and compare their relative performances. It helps the SDS developers to understand and analyse the user behaviour, with the assistance of a reinforcement learning algorithm. The SDS developers can thus confront the different KPI and control the further SDS choices by updating the call flow alternatives.

In the Spoken Dialogue System literature, all studies consider the dialogue move as the unquestionable unit for reinforcement learning. Rather than learning at the dialogue move level, we apply the learning at the design level for three reasons : 1/ to alleviate the high-skill prerequisite for developers, 2/ to reduce the learning complexity by taking into account just the relevant subset of the context and 3/ to have interpretable learning results that carry a reusable usage feedback. Unfortunately, tackling the problem at the design level breaks the Markovian assumptions that are required in most Reinforcement Learning techniques. Consequently, we decided to use a recent non-Markovian algorithm called Compliance Based Reinforcement Learning. This paper presents the first experimentation on online optimisation in dialogue systems. It reveals a fast and significant improvement of the system performance with by average one system misunderstanding less per dialogue. Index Terms : Spoken Dialogue Systems, Reinforcement Learning, Online Learning, Hybrid System

This document is a short report to accompany the Prototype Deliverable 5.3. 2, due at month 27 of the CLASSIC project. It describes the adaptations made to the industrial platform to conform to the specified CLASSIC architecture. The result is known as System 3. The main evolutions made are the introduction of system-based decisions in the dialogue automata design, an exhaustive logging, and the ability to handle use feedback and rewards.

This paper shows how the convergence between design and monitoring tools, and the integration of a dedicated reinforcement learning can be complementary and offer a new design experience for Spoken Dialogue System (SDS) developers. Most industrial SDS developers use a graphical tool to implement the dialogue strategies. First, this article proposes to integrate dialogue logs into this design tool, so that it constitutes a monitoring tool as well, by revealing call flows and their associated Key Performance Indicators (KPI). Second, the SDS developer is opened the possibility of designing several alternatives and of visually comparing his de-sign choice performances. Third, reinforcement learning technique is integrated to automatically optimise the SDS choices. The design/monitoring tool helps the SDS developers to understand and analyse the user behaviour, with the assistance of the learning algorithm. The SDS developers can then confront the different KPI and control the further SDS choices by removing or adding alternatives.

We present our Multi Point Of vieW Evaluation Refinement Studio (MPOWERS), an application framework for Spoken Dialogue System evaluation that implements design conventions in a user-friendly interface. It ensures that all evaluator-users manipulate a unique shared corpus of data with a shared set of parameters to design and retrieve their evaluations. It therefore answers both the need for convergence among the evaluation practices and the consideration of several analytical points of view addressed by the evaluators involved in Spoken Dialogue System projects. After introducing the system architecture, we argue the solution's added value in supporting a both data-driven and goal-driven process. We conclude with future works and perspectives of improvement upheld by human processes.

This document is the deliverable 6.1. 3, due at month 24 of the CLASSIC project. It describes the data collection process implemented during the specification, the preproduction and the production of the appointment scheduling 1013+ commercial service. At the time of writing, approximately 400 dialogues per day are being collected. 100% of calls from French customers are routed to the CLASSIC system.

This document presents the initial evaluations of the TownInfo (System 1) and Self-Help (System 3) systems after the first year of the CLASSiC project, as well as an evaluation of the DIPPER-POMDP system, also in the TownInfo domain. System 1 was evaluated at Cambridge, System 2 at France Telecom/Orange Labs, and DIPPER-POMDP at Edinburgh University. It presents the methodologies followed to evaluate each system, the quantitative results obtained, and an analysis with perspectives for future system developments.

This paper presents a new methodology for the control and design of distributed service architectures in an open environment. A domotic service illustrates and introduces the modelling keypoints. In particular, an explicit modelling of the attentional mechanism is used to overcome the lack of a global state in distributed systems and the relative impossibility to explicitly model all external events.

This paper addresses the problem of introducing learning capabilities in industrial handcrafted automata-based Spoken Dialogue Systems, in order to help the developer to cope with his dialogue strategies design tasks. While classical reinforcement learning algorithms position their learning at the dialogue move level, the fundamental idea behind our approach is to learn at a finer internal decision level (which question, which words, which prosody,...). These internal decisions are made on the basis of different (distinct or overlapping) knowledge. This paper proposes a novel reinforcement learning algorithm that can be used to make a data-driven optimisation of such handcrafted systems. An experiment shows that the convergence can be up to 20 times faster than with Q-Learning.

This document is a short report to accompany the Prototype Deliverable D1. 1.2, due at month 12 of the CLASSIC project. This prototype is an enhancement of the industrial Dialogue Manager with another module called the Learning Manager that provides the Spoken Dialogue System with adaptive online reinforcement learning capabilities. This document reviews the foundations of the reinforcement learning theory developed in the deliverable D1. 1.1 and it gives an overview of the API between those two modules.

This document is a short report to accompany the Prototype Deliverable 5.3. 1, due at month 12 of the CLASSIC project. It describes the adaptations made to the industrial platform to conform to the specified CLASSIC architecture. The result is known as System 3. The main evolutions made are the introduction of system-based decisions in the dialogue automata design, an exhaustive logging, and the ability to handle use feedback and rewards. Finally, nexts steps will be presented.

Evaluation of a human-machine dialogue system is a difficult problem for which neither the objectives nor the proposed solutions gather a unanimous support. Traditio- nal approaches in the ergonomics field evaluate the system by describing how it fits the user in the user referential of practices. However, the user referential is even more complicated to formalise, and one cannot ground a common use context to enable the comparison of two sys- tems, even if they are merely an evolution of the same service. We propose to shift the point of view on the evaluation problem : instead of evaluating the system in interaction with the user in the user's referential, we will now measure the user's adequacy to the system in the system referential. This is our Copernician revolution : for the evaluation purpose, our system is no longer user-centric, because the user referential is not properly objectifiable, while the system referential is completely known by design. Mots-clés : Évaluation, Dialogue.

L’evaluation des systemes de dialogue homme-machine est un probleme difficile et pour lequel ni les objectifs ni les solutions proposees ne font aujourd’hui l’unanimite. Les approches ergonomiques traditionnelles soumettent le systeme de dialogue au regard critique de l’utilisateur et tente d’en capter l’expression, mais l’absence d’un cadre objectivable des usages de ces utilisateurs empeche une comparaison entre systemes differents, ou entre evolutions d’un meme systeme. Nous proposons d’inverser cette vision et de mesurer le comportement de l’utilisateur au regard du systeme de dialogue. Aussi, au lieu d’evaluer l’adequation du systeme a ses utilisateurs, nous mesurons l’adequation des utilisateurs au systeme. Ce changement de paradigme permet un changement de referentiel qui n’est plus les usages des utilisateurs mais le cadre du systeme. Puisque le systeme est completement defini, ce paradigme permet des approches quantitatives et donc des evaluations comparatives de systemes.