Many laypeople are motivated to improve the health behavior of their family or friends but do not know where to start, especially if the health behavior is potentially stigmatizing or controversial. We present an approach that uses virtual agents to coach community-based volunteers in health counseling techniques, such as motivational interviewing, and allows them to practice these skills in role-playing scenarios. We use this approach in a virtual agent-based system to increase COVID-19 vaccination by empowering users to influence their social network. In a between-subjects comparative design study, we test the effects of agent system interactivity and role-playing functionality on counseling outcomes, with participants evaluated by standardized patients and objective judges. We find that all versions are effective at producing peer counselors who score adequately on a standardized measure of counseling competence, and that participants were significantly more satisfied with interactive virtual agents compared to passive viewing of the training material. We discuss design implications for interpersonal skills training systems based on our findings.

Public speaking experts intentionally take their audience on an emotional roller coaster, staying attuned to their audience's collective emotional feedback. In this research, we explore how bidirectional sharing of heart rates between a speaker and their audience facilitates this emotional exchange, through empathy, emotional awareness, and engagement. Firstly, in two design studies $(N=25)$, we evaluated this concept and identified design elements for a heart rate sharing interface. Subsequently, we developed Feeling-the-Beat, a system for sharing heart rate between speakers and audiences in real time. Finally, in a randomized, counter-balanced, within-subjects study $(N=36)$, we compared our system to one sharing fabricated heart rates and a baseline system without heart rate sharing. Feeling-the-Beat significantly increased audience empathy towards the speaker, audience engagement during moments of heightened speaker heart rate, and social presence.

Games have been successfully used to provide engaging health interventions for adolescents. However, translating health education goals into a playable game has historically taken many person-months of effort, involving game designers, scriptwriters, and artists. This work presents an exploratory study into rapidly developing physician-validated health education games for adolescents using virtual agents and LLMs. We evaluated this approach in an intervention to promote Human Papillomavirus (HPV) vaccination among adolescents, as lack of knowledge and vaccine hesitancy contribute to suboptimal HPV vaccination rates. We conducted a between-subjects randomized study comparing a fantasy narrative game to a non-gamified pedagogical virtual agent, with both interventions conveying the same HPV information. Among our study’s 9-12-year-old adolescent participants, our findings demonstrate large pre-to-post improvements in HPV knowledge for both conditions. The gamified intervention showed higher engagement and entertainment than the pedagogical agent based on participant interviews, demonstrating that gamification enriched the educational experience for adolescents.

Speaker diarization is a key component of systems that support multiparty interactions of co-located users, such as meeting facilitation robots. The goal is to identify who spoke what, often to provide feedback, moderate participation, and personalize responses by the robot. Current systems use a combination of acoustic (e.g. pitch differences) and visual features (e.g. gaze) to perform diarization, but involve the use of additional sensors or require overhead signal processing efforts. Alternatively, automatic speech recognition (ASR) is a necessary step in the diarization pipeline, and utilizing the transcribed text to directly identify speaker labels in the conversation can eliminate such challenges. With that motivation, we leverage large language models (LLMs) to identify speaker labels from transcribed text and observe an exact match of 77% and a word level accuracy of 90%. We discuss our findings and the potential use of LLMs as a diarization tool for future systems.

Parental permission is required for medical care for children, and decisions may be made without incorporating children's views, even for adolescents. We explore the impact of including adolescents in virtual agent-based multiparty health counseling to promote Human Papillomavirus (HPV) vaccination. The agent is designed to encourage HPV vaccination for children aged 9-12 by engaging co-present parent/adolescent dyads in an online interaction. Several techniques are incorporated, including HPV education, motivational interviewing, persuasion, modeling, and enablement, to address parents' intent to vaccinate their children. We conduct a between-subjects randomized study comparing a version of the agent exclusively for the parent, to one that includes the adolescent in the conversation and incorporates the child's views in counseling strategies. We measure pre- and post-intervention changes in intent to vaccinate, vaccination hesitancy, and knowledge in both the parent and the adolescent, hypothesizing greater improvements in these measures when the adolescent is included in the conversation. We also examine the satisfaction, engagement, and comfort of the parent/child interactions with the virtual agent. We found significant pre-post increases in parent intent to vaccinate their adolescent for both versions of the agent. Our work provides insights into the effectiveness of the virtual agent in promoting HPV vaccination, the impact of child participation on healthcare decision-making, and the user experience of multi-party interaction with the virtual agent.

Persuasion is a primary goal of public speaking, and eliciting audience empathy increases persuasion. In this research, we explore sharing a speaker’s heart rate as a social cue, to elicit empathy and increase persuasion in the audience. In particular, we developed two interfaces embedding the speaker’s heart rate over a recorded presentation video - as an animated line graph (raw representation) and as a color coded channel (abstract representation). In a randomized, counter-balanced, within subjects study (n = 18), we evaluated the concept using the two interfaces along with a baseline no heart rate condition. We observed that heart rate sharing significantly increased persuasion for participants with normal baseline empathy levels and increased empathic accuracy for all participants. Our qualitative analysis showed that heart rate was a useful cue in highlighting the emotions of the speaker, making the participants empathize with the speaker and pay more attention to the talk during those times. Our findings lead to a discussion of using heart rate as a social signal in a persuasive context, with implications for future research.

Narrative accounts are the ultimate authoritative source for pain assessment, and face-to-face encounters provide a rich context in which nonverbal conversational behavior can be used to enrich the detail in these descriptions. Embodied Conversational Agents—animated characters that simulate face-to-face conversation—can provide a medium for automated pain assessment in which multimodal pain narratives are elicited, clarified, and grounded. These agents can also use facial displays and nonverbal behavior to provide simulated empathic responses to help patients feel understood. We describe work towards a conversational agent that elicits various aspects of a pain experience, followed by an empathic summary. Our preliminary findings indicate that individuals are comfortable reporting their pain experiences to the agent, and are largely satisfied with this assessment methodology. We also find evidence that patients prefer the conversational empathic summary to standard self-report measures.

Public speaking is important in the sciences, but poor quality presentations are common, as are high rates of public speaking anxiety. In this work we explore the use of a mobile humanoid robot as a co-presenter that can share the stage with a scientist giving an oral presentation. We conducted a within-subjects experiment comparing presentations given with and without the robot and impacts on public speaking anxiety and speaker confidence. We found that participants reported significantly greater confidence and lower public speaking anxiety when co-presenting with the robot, compared to when they presented on their own, without the robot. Audiences accepted scientific presentations given with the robot, rating these presen-tations significantly greater than neutral on presentation quality.

Designers of virtual agents have a combinatorically large space of choices for the look and behavior of their characters. We conducted two between-subjects studies to explore the systematic manipulation of animation quality, speech quality, rendering style, and simulated empathy, and its impact on perceptions of virtual agents in terms of naturalness, engagement, trust, credibility, and persuasion within a health counseling domain. In the first study, animation was varied between manually created, procedural, or no animations; voice quality was varied between recorded audio and synthetic speech; and rendering style was varied between realistic and toon-shaded. In the second study, simulated empathy of the agent was varied between no empathy, verbal-only empathic responses, and full empathy involving verbal, facial, and immediacy feedback. Results show that natural animations and recorded voice are more appropriate for the agent’s general acceptance, trust, credibility, and appropriateness for the task. However, for a brief health counseling task, animation might actually be distracting from the persuasive message, with the highest levels of persuasion found when the amount of agent animation is minimized. Further, consistent and high levels of empathy improve agent perception but may interfere with forming a trusting bond with the agent.

Training laypeople to promote vaccination among their friends and family may be an effective way to boost the reach of vaccination interventions. We describe a virtual agent system that teaches laypeople communication and counseling skills using a combination of a pedagogical agent as well as a role-playing agent that takes on the persona of someone resistant to vaccination. We conducted a preliminary evaluation of the prototype, in which trainees first interacted with the prototype and then had a recorded conversation with a second person who was unvaccinated. Firstly, we found that trainees were mostly adherent to the skills taught by the agent. Secondly, there was a positive correlation of change in unvaccinated individuals' intent to get vaccinated with objective scores of display of empathy by the trainee during their conversation. Thirdly, unvaccinated partners rated the trainees high on relationship quality and use of empathic listening skills.

Healthcare and wellbeing are two main interconnected application areas of conversational agents (CAs). There is a significant increase in research, development, and commercial implementations in this area. In parallel to the increasing interest, new challenges in designing and evaluating CAs have emerged. This study aims to identify key design, development, and evaluation challenges of CAs in healthcare and wellbeing research. The focus is on the very recent projects with their emerging challenges. A review study was conducted with 17 invited studies, most of which were presented at the ACM CHI2020 conference workshop on CAs for health and wellbeing. Eligibility criteria required the studies to involve a CA applied to a health or wellbeing project in an ongoing or recently finished project. The participating studies were asked to report on their projects' design and evaluation challenges. We used thematic analysis to review the studies. The findings include a range of topics from primary care to caring for older adults to health coaching. We identified four major themes: i) domain information and integration, ii) user-system interaction and partnership, iii) evaluation, and iv) conversational competence. While some challenges are shared with other CA application areas, safety and privacy remain the major challenges in the healthcare and wellbeing domains. An increased level of collaboration across different institutions and entities may be a promising direction to address some of the major challenges which otherwise would be too complex to be addressed by the projects with their limited scope and budget.

Service-Learning is a pedagogical model that integrates classroom learning objectives with community needs and service goals. In this work, we present a case study implementation of Service-Learning within a graduate visualization curriculum, utilizing the design study“lite” methodology to meet the project’s goals within three months. In the case study, we worked with the Chester Square Neighborhood Association in Boston, MA, USA to help them improve their resident urban park by leveraging existing data, identifying current issues debilitating the park, and providing visualizations for insight. This case study demonstrates the “lite” methodology as a potential solution for a short-term design study approach as well as a viable approachin conjunction with Service-Learning, to facilitate visualization for social good.

Background Health care and well-being are 2 main interconnected application areas of conversational agents (CAs). There is a significant increase in research, development, and commercial implementations in this area. In parallel to the increasing interest, new challenges in designing and evaluating CAs have emerged. Objective This study aims to identify key design, development, and evaluation challenges of CAs in health care and well-being research. The focus is on the very recent projects with their emerging challenges. Methods A review study was conducted with 17 invited studies, most of which were presented at the ACM (Association for Computing Machinery) CHI 2020 conference workshop on CAs for health and well-being. Eligibility criteria required the studies to involve a CA applied to a health or well-being project (ongoing or recently finished). The participating studies were asked to report on their projects’ design and evaluation challenges. We used thematic analysis to review the studies. Results The findings include a range of topics from primary care to caring for older adults to health coaching. We identified 4 major themes: (1) Domain Information and Integration, (2) User-System Interaction and Partnership, (3) Evaluation, and (4) Conversational Competence. Conclusions CAs proved their worth during the pandemic as health screening tools, and are expected to stay to further support various health care domains, especially personal health care. Growth in investment in CAs also shows the value as a personal assistant. Our study shows that while some challenges are shared with other CA application areas, safety and privacy remain the major challenges in the health care and well-being domains. An increased level of collaboration across different institutions and entities may be a promising direction to address some of the major challenges that otherwise would be too complex to be addressed by the projects with their limited scope and budget.

We present Friendly Face - a virtual agent designed to reduce public speaking anxiety by standing within an audience. The agent senses the speaker's behavior during an oral presentation and provides emotional and instrumental support. The system unobtrusively tracks the motion, speech, and prosody of the presenter and provides an intuitive interface to give supportive feedback whenever the presenter looks at the agent, attentive listening behavior through agent gaze and backchannel listening behavior, and time and topic cueing based on real-time analysis of speech content compared to presentation slide contents. An evaluation of Friendly Face agent with a functionally equivalent control system demonstrated that the agent system led to significant reductions in public speaking anxiety compared to a control condition, assessed both objectively with physiological measures and validated self-report instruments.

People use their hands in a variety of ways to communicate information along with speech during face-to-face conversation. Humanoid robots designed to converse with people need to be able to use their hands in similar ways, both to increase the naturalness of the interaction and to communicate additional information in the same way people do. However, there are few studies of the particular meanings that people derive from robot hand gestures, particularly for more abstract gestures such as so-called metaphoric gestures that may be used to communicate quantitative or affective information. We conducted an exhaustive study of the 51 hand gestures built into a commercial humanoid robot to determine the quantitative and affective meaning that people derive from observing them without accompanying speech. We find that hypotheses relating gesture envelope parameters (e.g., height, distance from body) to metaphorically corresponding quantitative and affective concepts are largely supported.

The pandemic has caused a significant increase in the use of videoconferencing for oral presentations. Prior work demonstrated that an embodied conversational agent that co-delivers an oral presentation could be used in face-to-face presentations to reduce public speaking anxiety and increase presentation quality. In this work, we evaluate the use of a co-presenter agent in the delivery of virtual presentations given over a videoconferencing system, comparing them to presentations given without the agent. We found that participants were satisfied with the co-presenter agent, and those who liked the agent (scoring above the mean on a composite self-report measure of satisfaction) rated the presentations they gave with the agent as having significantly higher quality compared to those given without the agent. There was evidence the agent helped participants feel less nervous about their talks. Interviews confirmed these findings, and identified additional advantages and disadvantages of using co-presenter agents in virtual presentations.

The ability to monitor audience reactions is critical when delivering presentations. However, current videoconferencing platforms offer limited solutions to support this. This work leverages recent advances in affect sensing to capture and facilitate communication of relevant audience signals. Using an exploratory survey (N=175), we assessed the most relevant audience responses such as confusion, engagement, and head-nods. We then implemented AffectiveSpotlight, a Microsoft Teams bot that analyzes facial responses and head gestures of audience members and dynamically spotlights the most expressive ones. In a within-subjects study with 14 groups (N=117), we observed that the system made presenters significantly more aware of their audience, speak for a longer period of time, and self-assess the quality of their talk more similarly to the audience members, compared to two control conditions (randomly-selected spotlight and default platform UI). We provide design recommendations for future affective interfaces for online presentations based on feedback from the study.

The quality of scientific oral presentations is often poor, owing to a number of factors, including public speaking anxiety. We present DynamicDuo, a system that uses an automated, life-sized, animated agent to help inexperienced scientists deliver their presentations in front of an audience. The design of the system was informed by an analysis of TED talks given by pairs of human presenters to identify the most common dual-presentation formats and transition behaviors used. We explore the usability and acceptability of DynamicDuo in both controlled laboratory-based studies and real-world environments, and its ability to decrease public speaking anxiety and improve presentation quality. In a within-subjects study (N = 12) comparing co-presenting with DynamicDuo against solo-presenting with conventional presentation software, we demonstrated that our system led to significant improvements in public speaking anxiety and speaking confidence for non-native English speakers. Judges who viewed videotapes of these presentations rated those with DynamicDuo significantly higher on speech quality and overall presentation quality for all presenters. We also explore the affordances of the virtual co-presenter through empirical evaluation of novel roles the agent can play in scientific presentations and novel ways it can interact with the speaker in front of the audience.

We describe the design and evaluation of a humanoid robot that explains inherited breast cancer genetics, and motivates women to obtain cancer genetic testing. The counseling dialogue is modeled after a human cancer genetic counselor, extended with data visualizations and nonverbal behavior. In a quasi-experimental pilot study, we demonstrated that interaction with the robot leads to significant increases in cancer genetics knowledge.

Design studies are frequently used to conduct problem-driven visualization research by working with real-world domain experts. In visualization pedagogy, design studies are often introduced but rarely practiced due to their large time requirements. This limits students to a classroom curriculum, often involving projects that may not have implications beyond the classroom. Thus we present the Design Study "Lite" Methodology, a novel framework for implementing design studies with novice students in 14 weeks. We utilized the Design Study "Lite" Methodology in conjunction with Service-Learning to teach five Data Visualization courses and demonstrate that it benefits not only the students but also the community through service to non-profit partners. In this paper, we provide a detailed breakdown of the methodology and how Service-Learning can be incorporated with it. We also include an extensive reflection on the methodology and provide recommendations for future applications of the framework for teaching visualization courses and research.