Adapting users’ emotional states to virtual health assistance systems can enhance interaction quality and overall user experience. Virtual characters or avatars, can improve the usability of healthcare applications by making more engaging and personalized re- and interactions. Their appearance, visual characteristics, and voice influence how users perceive both the avatar and the system as a whole. By incorporating facial expressions, gestures, and verbal communication, avatars can take on human-like roles, such as providing counseling, to improve user satisfaction. This study presents the development of photorealistic virtual assistants using Unreal Engine’s Metahuman Creator. These avatars adapt their speech and facial expressions based on the user’s emotional states. The system synchronizes Metahuman avatars with an AI-powered chatbot, enabling the display of emotions such as happiness, excitement, boredom, and sadness. The avatars displayed on a monitor in landscape mode respond with appropriate expressions and voice tones to create a more natural and engaging conversation. To evaluate the system, we conducted a user study with 21 participants, measuring usability, social presence, emotional response, and trust. Our results in the ATI and UEQ questionnaires indicate that emotion-adaptive avatars significantly enhance experience quality and emotional engagement compared to standard virtual assistants. Future work will focus on automated facial expression detection in real-time and assessing the impact of the system on user engagement, well-being, and adherence to health recommendations.

Patient-reported outcome measure (PROM) completion is hindered by patient-level barriers—including motor, sensory, cognitive, and motivational constraints—that risk insufficient participation and non-response bias. While technology-enabled approaches such as multimodal speech assistance hold promise for reducing these barriers, assistance is a complex interaction: it can both alleviate and introduce barriers depending on how well it aligns with patients’ routines and needs. This qualitative study explores how patients perceive the advantages and disadvantages of AI-based speech assistance for PROM collection, focusing on how assistance functionalities interact with individual barriers and completion practices. We conducted semi-structured qualitative interviews with 96 psychosomatic and neurological rehabilitation outpatients, embedded in a pragmatic cross-randomised controlled trial. Participants completed PROMs with and without an AI-based speech assistance system offering speech output, speech input, and guidance by a socially interactive agent (SIA) that was physically, virtually, or voice-only embodied. The system was iteratively refined during data collection to address usability and performance issues. We included a broad sample to reflect real-world care settings, including patients without reported barriers. Using inductive content analysis (61 codes, grouped into 4 overarching and 9 subthemes), we examined perceived advantages and disadvantages of the three main assistance functionalities and multimodal interaction. Reporting followed the COREQ guideline. The speech output function emerged as the most widely valued assistance feature, with many patients reporting improved concentration, question comprehension, and deeper engagement with item content. The social agent was described as making the interaction more engaging and less monotonous, by at the same time not evoking social pressure. Speech input was perceived as helpful by some, especially for those with motor impairments or a preference for verbal expression. However, each function also introduced challenges: speech output disrupted reading routines for some, the social agent was perceived as distracting or unnecessary by others, and speech input was criticised for recognition errors, inefficiency, and privacy concerns. AI-based speech assistance for PROM collection offers significant potential to reduce barriers and enhance patient engagement, but its effectiveness depends on alignment with individual needs, preferences and routines. While speech output proved broadly beneficial, speech input and socially interactive agents require careful design to avoid introducing new barriers, particularly for marginalised groups. Configurable, modular assistance systems that adapt to diverse user preferences and impairments are essential for equitable implementation. Future research should focus on inclusive co-design and longitudinal studies to refine these technologies for real-world clinical use. German Clinical Trail Register-ID: DRKS00035213

The collection of patient-reported outcome measures (PROMs) is a key measurement tool for patient-centred care. At the same time, collecting these measures poses obstacles for many patients, leading to these groups being underrepresented in the data. We have therefore developed a multimodal, AI-driven assistance system to support patients in collecting these data. The interface of the system comprised a digital tablet containing the PROM questionnaire items and the assistant in three forms of embodiment: A virtual avatar, a physical avatar, and a voice-only agent. To evaluate the users' experience and ratings of the system, two separate studies were implemented in two rehabilitation centers with 195 patients. A mixed within-between RCT was conducted at an outpatient clinic, where patients completed PROMs both with and without an assistant, and a between-subject design at an inpatient clinic comparing routine PC-based care with avatar- and robot-assisted PROM administration. Our results suggest a preference for the non-assisted tablet-only condition in Clinic A, whereas, in Clinic B, both agent conditions were preferred over routine care. We have further analyzed aspects such as trust and social presence in this study to gain a more thorough understanding of the users' experience. Our analysis shows a higher trust rating for the voice-only assistant, whereas the robot, virtual avatar, and the voice-only conditions were perceived as more socially present. The impact of demographic factors and affinity for technology on the user ratings was also thoroughly studied. Our findings shed light on the role of agent embodiment in PROM assistance and contribute to the future design and evaluation of effective, engaging, and trustworthy systems for data collection in healthcare settings.

In this study, we explore the application of Large Language Models (LLMs) for generating synthetic users and simulating user conversations with a task-oriented dialogue system and present detailed results and their analysis. We propose a comprehensive novel approach to user simulation technique that uses LLMs to create diverse user profiles, set goals, engage in multi-turn dialogues, and evaluate the conversation success. We employ two proprietary LLMs, namely GPT-4o and GPT-o1 (Achiam et al., 2023), to generate a heterogeneous base of user profiles, characterized by varied demographics, multiple user goals, different conversational styles, initial knowledge levels, interests, and conversational objectives. We perform a detailed analysis of the user profiles generated by LLMs to assess the diversity, consistency, and potential biases inherent in these LLM-generated user simulations. We find that GPT-o1 generates more heterogeneous user distribution across most user attributes, while GPT-4o generates more skewed user attributes. The generated set of user profiles are then utilized to simulate dialogue sessions by interacting with a task-oriented dialogue system.

In this paper, we compare the intent classification performance of several small-scale ( ∼ 7B parameters range) open-source Large Language Models (LLMs), namely Mistral-7B-Instruct-v0.1, Mistral-7B-Instruct-v0.2 [12] and Meta-Llama-3-8B-Instruct [3] and fine-tune with a variety of training sample sizes (20, 100, 1k, 10k, and 50k samples) for multi-intent classification tasks using LoRA [11]. We also compare their results with two relatively smaller models, namely Phi-3-mini-4k-instruct ( ∼ 3.8B parameters) [1] and bert-base-uncased ( ∼ 110M parameters) [8], also fine-tuned on the same training chunks. We use the MultiWOZ 2.1 dataset [9], a widely used dataset for task-oriented dialogue systems that has 17 different intent classes. We get similar performance across these models while the training sample size is larger (50k samples) with Mistral-7B-Instruct-v0.2 having the best results with a micro-average f1-score of ∼ 91 ∼ 85

Large Language Models (LLMs) hold promise for addressing the shortage of qualified therapists in mental health care. While chatbot-based Cognitive Behavioral Therapy (CBT) tools exist, their efficacy in sensitive contexts remains underexplored. This study examines the potential of LLMs to support therapy sessions aimed at reducing Child Sexual Abuse Material (CSAM) consumption. We propose a Retrieval-Augmented Generation (RAG) framework that leverages a finetuned BERT-based retriever to guide LLM-generated responses, better capturing the multiturn, context-specific dynamics of therapy. Four LLMs-Qwen2-7B-Instruct, Mistral-7B-Instruct-v0.3, Orca-2-13B, and Zephyr-7B-Alpha-were evaluated in a small-scale study with 14 domain-expert psychotherapists. Our comparative analysis reveals that, in certain scenarios, LLMs like Mistral-7B-Instruct-v0.3 and Orca-2-13B were preferred over human therapist responses. While limited by sample size, these findings suggest that LLMs can perform at a level comparable to or even exceeding that of human therapists, especially in therapy focused on reducing CSAM consumption. Our code is available online. (1)

The integration of emotional intelligence into virtual health assistance systems has the potential to revolutionize the way users interact with healthcare applications. By adapting to users’ emotional states, virtual characters or avatars can provide more personalized and engaging interactions, leading to improved user satisfaction and overall well-being. This study presents the development of photorealistic virtual assistants using Unreal Engine’s Metahuman Creator, which can adapt their speech and facial expressions to match the user’s emotional state. The system, which synchronizes Metahuman avatars with an AI-powered chatbot, enables the display of emotions such as happiness, excitement, boredom, and sadness, creating a more natural and engaging conversation. A user study with 21 participants evaluated the system’s effectiveness, revealing that emotion-adaptive avatars notably enhance experience quality and emotional engagement compared to neutral virtual assistants. The findings of this study have significant implications for the development of more effective and user-friendly healthcare applications and highlight the need for further research into the impact of emotion-adaptive virtual assistants on user engagement, well-being, and adherence to health recommendations.

In this paper, we provide an extensive analysis of multi-label intent classification using Large Language Models (LLMs) that are open-source, publicly available, and can be run in consumer hardware. We use the MultiWOZ 2.1 dataset, a benchmark in the dialogue system domain, to investigate the efficacy of three popular open-source pre-trained LLMs, namely LLama2-7B-hf, Mistral-7B-v0.1, and Yi-6B. We perform the classification task in a few-shot setup, giving 20 examples in the prompt with some instructions. Our approach focuses on the differences in performance of these models across several performance metrics by methodically assessing these models on multi-label intent classification tasks. Additionally, we compare the performance of the instruction-based fine-tuning approach with supervised learning using the smaller transformer model BertForSequenceClassification as a baseline. To evaluate the performance of the models, we use evaluation metrics like accuracy, precision, and recall as well as micro, macro, and weighted F1 score. We also report the inference time, VRAM requirements, etc. The Mistral-7B-v0.1 outperforms two other generative models on 11 intent classes out of 14 in terms of F-Score, with a weighted average of 0.50. It also has relatively lower Humming Loss and higher Jaccard Similarity, making it the winning model in the few-shot setting. We find BERT based supervised classifier having superior performance compared to the best performing few-shot generative LLM. The study provides a framework for small open-source LLMs in detecting complex multi-intent dialogues, enhancing the Natural Language Understanding aspect of task-oriented chatbots.

In recent times, discussions on social media platforms have increasingly come under scrutiny due to the proliferation of science denial and fake news. Traditional solutions, such as regulatory actions, have been implemented to mitigate the spread of misinformation; however, these measures alone are not sufficient. To complement these efforts, educational approaches are becoming essential in empowering users to critically engage with misinformation. Conversation training, through serious games or personalized methods, has emerged as a promising strategy to help users handle science denial and toxic conversation tactics. This paper suggests an interdisciplinary seminar to explore the suitability of Large Language Models (LLMs) acting as a persona of a science denier to support people in identifying misinformation and improving resilience against toxic interactions. In the seminar, groups of four to five students will develop an AI-based chatbot that enables realistic interactions with science-denial argumentation structures. The task involves planning the setting, integrating a Large Language Model to facilitate natural dialogues, implementing the chatbot using the RASA framework, and evaluating the outcomes in a user study. It is crucial that users understand what they need to do during the interaction, how to conclude it, and how the relevant information is conveyed. The seminar does not aim to develop chatbots for practicing debunking but serves to teach AI technologies and test the feasibility of this idea for future applications. The chatbot seminar is conducted as a hybrid, parallel master's module at the participating educational institutions.

This paper presents a study aimed at enhancing the classification accuracy of patients' spoken answer selections to German medical Patient-Reported Outcome Measures (PROM) questionnaires within a multimodal dialog system. We collected 1,737 speech data samples for training and evaluation through a lab experiment, employing textual priming as opposed to the visual priming utilized in prior research. For classification, we compare results from utilizing sentence embeddings against results from prompting various Large Language Models. We conduct a comparative analysis of approaches in terms of prediction performance, efficiency, hardware restraints, budget, inference time, and data privacy. Further, we investigate if adding the survey item text as context improves the classification. Results show the highest accuracy for gpt-4 prompting, and indicate that including the questionnaire item text alongside user utterances is beneficial for LLM prompting. Additionally, we find significant positive correlations between accuracy and certain prompt characteristics.

A central requirement for the use of artificial intelligence (AI) in medicine is its explainability, i. e., the provision of addressee-oriented information about its functioning. This leads to the question of how socially adequate explainability can be designed. To identify evaluation factors, we interviewed healthcare stakeholders about two scenarios: diagnostics and documentation. The scenarios vary the influence that an AI system has on decision-making through the interaction design and the amount of data processed. We present key evaluation factors for explainability at the interactional and procedural levels. Explainability must not interfere situationally in the doctor-patient conversation and question the professional role. At the same time, explainability functionally legitimizes an AI system as a second opinion and is central to building trust. A virtual embodiment of the AI system is advantageous for language-based explanations.

In the last decade, crowdsourcing has become a popular method for conducting quantitative empirical studies in human-machine interaction. The remote work on a given task in crowdworking settings suits the character of typical speech/language-based interactive systems for instance with regard to argumentative conversations and information retrieval. Thus, crowdworking promises a valuable opportunity to study and evaluate the usability and user experience of real humans in interactions with such interactive systems. In contrast to physical attendance in laboratory studies, crowdsourcing studies offer much more flexible and easier access to large numbers of heterogeneous participants with a specific background, e.g., native speakers or domain expertise. On the other hand, the experimental and environmental conditions as well as the participant's compliance and reliability (at least better monitoring of the latter) are much better controllable in a laboratory. This paper seeks to present a (self-)critical examination of crowdsourcing-based studies in the context of complex (spoken) dialogue systems. It describes and discusses observed issues in crowdsourcing studies involving complex tasks and suggests solutions to improve and ensure the quality of the study results. Thereby, our work contributes to a better understanding and what needs to be considered when designing and evaluating studies with crowdworkers for complex dialogue systems.

A central requirement for the use of artificial intelligence (AI) in medicine is its explainability, i. e., the provision of addressee-oriented information about its functioning. This leads to the question of how socially adequate explainability can be designed. To identify evaluation factors, we interviewed healthcare stakeholders about two scenarios: diagnostics and documentation. The scenarios vary the influence that an AI system has on decision-making through the interaction design and the amount of data processed. We present key evaluation factors for explainability at the interactional and procedural levels. Explainability must not interfere situationally in the doctor-patient conversation and question the professional role. At the same time, explainability functionally legitimizes an AI system as a second opinion and is central to building trust. A virtual embodiment of the AI system is advantageous for language-based explanations

In modular dialog systems, a dialog system consists of multiple conversational agents. The task “module selection” selects the appropriate sub-dialog system for an incoming user utterance. Current models for module selection use features derived from the current user turn only, such as the utterances text or confidence values of the natural language understanding systems of the individual conversational agents, or they perform text classification on the user utterance. However, dialogs often span multiple turns, and turns are embedded into a context. Therefore, looking at the current user turn only is a source of error in certain situations. This work proposes four models for module selection that include the dialog history and the current user turn into module selection. We show that these models surpass the current state of the art in module selection.

A natural way for humans to build an opinion on a topic is through the gathering and exchange of new arguments. Speech interfaces for argumentative dialogue systems (ADS) are rather scarce and quite complex. To provide a more natural and intuitive interface, we include an adaption of a recently introduced natural language understanding (NLU) framework tailored to argumentative tasks into a complete end-to-end ADS. Within this paper we investigate the influence of two different input/output modalities (speech/speech and drop-down menu/text) and discuss issues and problems we encountered in a user study with 202 participants using our ADS.

Nowadays speech-driven interfaces such as mobile digital assistants and chatbots can support collaborative information seeking and are becoming increasingly commonplace. Especially, mobile dialogue assistants offer innovative approaches to deliver and access information and thus, display a promising approach to assist humans in their opinion building process. Still, due to the complexity of argumentative tasks mobile argumentative speech interfaces are still very scarce. Hence, the effect of such interfaces on a user's opinion building process is quite unexplored. In this paper, we investigate the influence of such interfaces on the interest and opinion building process of users. Both categories Therefore we introduce two (I/O) modalities (menu/speech) of the argumentative dialog system (ADS) BEA ("Building Engaging Argumentation" [2]) which enables the user to scrutinize arguments on both sides of a controversial topic. In particular, we reflect on the influence and advantages of a spoken hands-free versus a clickable drop-down menu-based ADS with regard to "mobile" dialog systems use cases. Therefore the users' expectations and experiences in a self-assessment questionnaire are evaluated and discussed in comparison to our user interest and opinion model.

EDITORIAL article Front. Comput. Sci., 30 June 2022Sec. Human-Media Interaction https://doi.org/10.3389/fcomp.2022.966163