Teachers in challenging conflict situations often experience shame and self-blame, which relate to the feeling of incompetence but may externalise as anger. Sensing mixed signals fails the contingency rule for developing affect regulation and may result in confusion for students about their own emotions and hinder their emotion regulation. Therefore, being able to constructively regulate emotions not only benefits individual experience of emotions but also fosters effective interpersonal emotion regulation and influences how a situation is managed. MITHOS is a system aimed at training teachers' conflict resolution skills through realistic situative learning opportunities during classroom conflicts. In four stages, MITHOS supports teachers' socio-emotional self-awareness, perspective-taking and positive regard. It provides: a) a safe virtual environment to train free social interaction and receive natural social feedback from reciprocal student-agent reactions, b) spatial situational perspective taking through an avatar, c) individual virtual reflection guidance on emotional experiences through co-regulation processes, and d) expert feedback on professional behavioural strategies. This chapter presents the four stages and their implementation in a semi-automatic Wizard-of-Oz (WoZ) System. The WoZ system affords collecting data that are used for developing the fully automated hybrid (machine learning and model-based) system, and to validate the underlying psychological and conflict resolution models. We present results validating the approach in terms of scenario realism, as well as a systematic testing of the effects of external avatar similarity on antecedents of self-awareness with behavior similarity. The chapter contributes to a common methodology of conducting interdisciplinary research for human-centered and generalisable XR and presents a system designed to support it.

Human interaction partners can deal with interruptions and then resume the interaction. This ability should be emulated by social agents. How fast interruptions are handled might influence the overall impression of an agent. In this paper, we present the results of a user study on how a human dialog partner perceives the be- havior of a virtual agent handling verbal user interruptions with different reaction times. The study goes beyond typical perception experiments by preserving the real-time interaction experience. For the evaluation, we rely on a parametrizable parallelized computa- tional model that represents dialog flow, overlap detection, conflict recognition, and conflict handling in real-time. The evaluation re- sults show that the timing of the agent's interruption handling in interactive human-agent dialogues is related to different interper- sonal attitudes.

In this paper, we focus on experience-based role play with virtual agents to provide young adults at the risk of exclusion with social skill training. We present a scenario-based serious game simulation platform. It comes with a social signal interpretation component, a scripted and autonomous agent dialog and social interaction behavior model, and an engine for 3-D rendering of lifelike virtual social agents in a virtual environment. We show how two training systems developed on the basis of this simulation platform can be used to educate people in showing appropriate socioemotive reactions in job interviews. Furthermore, we give an overview of four conducted studies investigating the effect of the agents' portrayed personality and the appearance of the environment on the players' perception of the characters and the learning experience.

Grounding is an important process that underlies all human interaction. Hence, it is also crucial for social companions to interact naturally. Maintaining the common ground requires domain knowledge but has also numerous social aspects, such as attention, engagement and empathy. Integrating these aspects and their interplay with the dialog management in a computational interaction model is a complex task. We present a modeling approach overcoming this challenge and illustrate it based on some social companion applications.

We present work in progress on an intelligent embodied conversation agent in the basic care and healthcare domain. In contrast to most of the existing agents, the presented agent is aimed to have linguistic cultural, social and emotional competence needed to interact with elderly and migrants. It is composed of an ontology-based and reasoning-driven dialogue manager, multimodal communication analysis and generation modules and a search engine for the retrieval of multimedia background content from the web needed for conducting a conversation on a given topic.

The outcome of interpersonal interactions depends not only on the contents that we communicate verbally, but also on nonverbal social signals. Because a lack of social skills is a common problem for a significant number of people, serious games and other training environments have recently become the focus of research. In this work, we present NovA (Nonverbal behavior Analyzer), a system that analyzes and facilitates the interpretation of social signals automatically in a bidirectional interaction with a conversational agent. It records data of interactions, detects relevant social cues, and creates descriptive statistics for the recorded data with respect to the agent's behavior and the context of the situation. This enhances the possibilities for researchers to automatically label corpora of human--agent interactions and to give users feedback on strengths and weaknesses of their social behavior.

Technology-enhanced learning environments are designed to help users practise social skills. In this paper, we present and evaluate a virtual job interview training game which has been adapted to the special requirements of young people with low chances on the job market. The evaluation spanned three days, during which we compared the technology-enhanced training with a traditional learning method usually practised in schools, i.e. reading a job interview guide. The results are promising as professional career counsellors rated the pupils who trained with the system significantly better than those who learned with the traditional method.

Through interaction with the virtual environment and virtual characters, users are able to influence the storyline of many games. The design choice for the style of interactivity can thereby have a crucial influence on the user's experience. However, only a few approaches evaluate different interaction modalities for one system to investigate the impact of design choice on the users' experience. In this paper, we present an experimental approach in which we first reflect on design alternatives concerning a specific element of interactive narratives-user-character dialog-and then investigate user responses to different design options (round-based dialog versus continuous dialog). Results of an experimental evaluation study show that users tend to prefer continuous interaction in a soap-opera-like game environment using typed text input to communicate with virtual characters that act and react using speech output, although the recognition rate of user utterances of the continuous version was slightly worse compared to the round-based version.

Job interviews come with a number of challenges, especially for young people who are out of employment, education, or training (NEETs). This paper presents an approach to a job training simulation environment that employs two virtual characters and social cue recognition techniques to create an immersive interactive job interview. The two virtual characters are created with different social behavior profiles, understanding and demanding, which consequently influ- ences the level of difficulty of the simulation as well as the impact on the user. In this context we present a user study which investigates the feasibility of the proposed approach by measuring the effect the different virtual characters have on users.

Grounding is an important process that underlies all human interaction. Hence, it is crucial for building social robots that are expected to collaborate effectively with humans. Gaze behavior plays versatile roles in establishing, maintaining and repairing the common ground. Integrating all these roles in a computational dialog model is a complex task since gaze is generally combined with multiple parallel information modalities and involved in multiple processes for the generation and recognition of behavior. Going beyond related work, we present a modeling approach focusing on these multi-modal, parallel and bi-directional aspects of gaze that need to be considered for grounding and their interleaving with the dialog and task management. We illustrate and discuss the different roles of gaze as well as advantages and drawbacks of our modeling approach based on a first user study with a technically sophisticated shared workspace application with a social humanoid robot.

Grounding is essential in human interaction and crucial for social robots collaborating with humans. Gaze plays versatile roles for establishing, maintaining and repairing the common ground. It is combined with parallel modalities and involved in several processes for behavior generation and recognition. We present a uniform modeling approach focusing on the multi-modal, parallel and bidirectional aspects of gaze and their interleaving with the dialog logic.

Motivation is a critical requirement for successful learning. Previous research has identified that animated pedagogical agents can increase motivation. Following these results, we present the cast of pedagogical agents in the DynaLearn Intelligent Learning Environment. Each of these agents is associated with one of the different support types available in the environment, giving each agent a clearly defined role. We describe the different character roles, how their knowledge is generated and related to the pedagogical purpose at hand, how they interact with the learners and finally how this interaction helps increasing the learners' motivation. To assess this, we conducted a preliminary evaluation with three of the characters and report our findings.

In this paper we present a novel approach to the combined modeling of multimodal fusion and interaction management. The approach is based on a declarative multimodal event logic that allows the integration of inputs distributed over multiple modalities in accordance to spatial, temporal and semantic constraints. In conjunction with a visual state chart language, our approach supports the incremental parsing and fusion of inputs and a tight coupling with interaction management. The incremental and parallel parsing approach allows us to cope with concurrent continuous and discrete interactions and fusion on different levels of abstraction. The high-level visual and declarative modeling methods support rapid prototyping and iterative development of multimodal systems.

Creating interactive applications with multiple virtual characters comes along with many challenges that are related to different areas of expertise. The definition of context-sensitive interactive behavior requires expert programmers and often results in hard-to-maintain code. To tackle these challenges, we suggest a visual authoring approach for virtual character applications and present a revised version of our SceneMaker tool. In SceneMaker a separation of content and logic is enforced. In the revised version, the Visual SceneMaker, we introduce concurrency and specific history structures as key concepts to facilitate (1) clearly structured interactive behavior definition, (2) multiple character modeling, and (3) extensions to existing applications. The new integrated developer environment allows sceneflow visualization and runtime modifications to support the development of interactive character applications in a rapid prototyping style. Finally, we present the result of a user study, which evaluates usability and the key concepts of the authoring tool.

The integration of culture into the behavioral models of virtual characters requires knowledge from very different disciplines such as sociology and computer science. If culture-related behavioral differences are integrated into a virtual character system, users do not necessarily understand the intent of such a system. In this paper, we present a prototype that tries to integrate the masculinity dimension of culture with prototypical differences in verbal and nonverbal behavior. In a preliminary evaluation study, we investigated how these differences are judged by human observers with different cultural backgrounds.

How human users perceive and interact with interactive story-telling applications has not been widely researched so far. In this paper, we present an experimental approach in which we investigate the impact of different dialog-based interaction styles on human users. To this end, an interactive demonstrator has been evaluated in two different versions: one providing a continuous interaction style where interaction is possible at any time, and another providing system-initiated interaction where the user can only interact at certain prompts.

In this paper, we present a modeling approach for the management of highly interactive, multithreaded and multimodal dialogues. Our approach enforces the separation of dialogue content and dialogue structure and is based on a statechart language enfolding concepts for hierarchy, concurrency, variable scoping and a detailed runtime history. These concepts facilitate the modeling of interactive dialogues with multiple virtual characters, autonomous and parallel behaviors, flexible interruption policies, context-sensitive interpretation of the user's discourse acts and coherent resumptions of dialogues. An interpreter allows the realtime visualization and modification of the model to allow a rapid prototyping and easy debugging. Our approach has successfully been used in applications and research projects as well as evaluated in field tests with non-expert authors. We present a demonstrator illustrating our concepts in a social game scenario.

In this paper we present the cast of pedagogical agents in the DynaLearn Intelligent Learning Environment. We describe the different character roles and how they interact with the learners. Our aim in using these characters is to increase the learners’ motivation.

This document presents the progress and effort made in the design of a dialog system for the virtual characters in DynaLearn. The main purpose of this system is to provide means by which the virtual characters can present relevant system knowledge to the learners in a pedagogically sound manner. After providing an overview of the role of dialogs in interactive learning environments we present the architecture of our approach and describe in detail the functionality of its three main components: Dialog Management, Verbalization and User Modeling.