As robot programming becomes increasingly important in enabling non-expert users to customize autonomous systems across domains such as manufacturing, healthcare, and service industries, there is a growing demand for methods that are both intuitive and support users’ perceived trust. While natural language programming (NLP) lowers the barrier to entry by allowing users to describe robot behavior in everyday language, it often lacks structural transparency—particularly when paired with opaque deep learning models. To address this challenge, we present LLiBTree, a novel robot programming framework that integrates Large Language Models (LLMs) directly into Behavior Trees (BTs) as executable nodes. This approach allows users to iteratively define and refine robot behavior via natural language while preserving the modularity and clarity of visual programming. By embedding the LLM within the BT, users can generate and replace sub-trees in response to natural language instructions, enabling a seamless blend of language-based flexibility and structured visual logic. We evaluate LLiBTree through a controlled user study, comparing it with both natural language-only and conventional visual programming methods. The results show that the PreGeneratedLLiBTree condition yields significantly higher perceived trust, while LLiBTree conditions achieve usability that is non-inferior to natural language-only programming and improve users’ sense of agency, suggesting that LLiBTree offers a practical approach for end-user robot programming.
Recent research has demonstrated the potential for representing intelligent guidance using multi-modal cues, yet few guidelines or processes exist to guide the design of such a system. In this article, we seek to address this gap by investigating the design of multi-modal assistant systems for setting the optimal parameters in industrial plants. We present the results of two user studies conducted with a total of 37 participants to evaluate the effectiveness and experience of different combinations of visual ( Highlights and Ambient lights) and haptic ( Clicks and Vibration ) modalities for providing intelligent dynamic guidance. Our findings demonstrate that providing the intelligent guidance with the multi-modality of Highlights+Ambient resulted in shorter task duration and higher practicality than Ambient lights alone. Regarding task accuracy, the Highlights and Vibration outside-zone guidance resulted in no failed attempts. Moreover, Highlights+Ambient+Vibration guidance was rated as having lower usability than Highlights+Ambient , and higher mental demand than merely Highlights .
Human-in-the-loop methods leverage human feedback to enhance machine learning and AI. Manual review of outputs can correct errors, identify model weaknesses, or expand labels to broaden model capabilities. Feedback collection methods range from simple flagging of outputs as correct or incorrect to more complex feature-level adjustments or natural language interpretations. This article presents a user study evaluating changes in user performance over time and explores the tradeoff between feedback quality and human effort. We compare four interactive input methods for reviewing and correcting outcomes in object detection and activity recognition in videos. Our findings indicate that while some complex input methods, such as free-text, require more time, the quality and impact of their feedback on model accuracy often surpass those of simpler methods that require less effort. However, more effort does not always lead to better-quality feedback, especially when aiming to improve the model. Our VLM experiments show that the most accurate models were trained using detailed natural language feedback or precise word-level corrections, while simple yes/no judgments also led to solid performance at a much lower annotation cost.
Inclusive usability testing, such as the GenderMag method, wants to identify gender-related usability problems in digital interfaces. Large Language Models (LLMs) have been used by usability engineers in usability evaluations but their contribution is still underexplored, especially regarding inclusive usability testing. Research has shown that GenderMag workshops can produce valuable insights but are resource intense and might show effects from the evaluators’ ability to embody personas with a different cognitive style. Therefore, we need to assess if LLM-agent based testing can aid human-based evaluations. This study evaluates an LLM-agent system for GenderMag persona-based usability testing, and compares its performance to traditional human-led evaluations. The agent system integrates GenderMag persona facets into three LLM-agents, which analyze usability issues of four web interfaces, three generic and one intentionally flawed interface with gender-related usability issues. We quantitatively and qualitatively compare the types, severity, and relevance of usability problems LLM-agents identified to those produced in three GenderMag workshops involving nine participants. Findings show a broad overlap in detecting usability issues of humans and LLM-agents that are not gender specific via the generic interfaces. Agents thereby consistently assign significantly higher severity and relevance ratings. For the intentionally flawed interface, humans and LLM-agents assign similar ratings but the overlap between human and agent gender-related usability issues was low, with each missing issues the other caught. The agent system is an efficient tool for gender-related usability evaluation that humans may overlook, thereby expanding the coverage of evaluation.
Conversational agents (CAs) are increasingly integrated into augmented reality (AR), yet evidence on how conversational capabilities are combined with AR contexts remains dispersed across application domains and device types. We conducted a systematic review of 44 peer-reviewed studies (2007–2025) using predefined search and screening procedures. Across the included systems, speech is the predominant input modality (95%), and head-mounted displays (HMDs) are most frequently used (66%). Many prototypes couple speech with contextual information such as location or object recognition (41%) to adapt conversational behaviour. We also observed a strong trend in dialogue management toward agent-based architectures powered by Large Language Models, with 12 of the 14 agent-based systems being published after 2023. Our analysis resulted in the development of a framework for AR-CA interaction, which positions systems along two dimensions: Presence Form and Context Awareness. The framework reveals an emerging trend toward dynamic, context-aware interaction. User experience reports are broadly positive, but limitations include challenges with long-term engagement, privacy risks, and hardware constraints. Our review yields four key design considerations: match an avatar’s embodiment with equally capable LLM dialogue; enrich interactions through haptics and spatial audio; preserve physical and social plausibility; and adopt user-centred, privacy-aware hardware choices. Together, these principles chart a path toward robust, contextually intelligent, and ethically responsible AR conversational agents.
We introduce PromptCanvas, a UI concept that transforms prompting into a composable, widget-based experience on an infinite canvas. Users can generate, customize, and arrange interactive widgets that represent various facets of their text, offering greater control over AI-generated content. PromptCanvas allows to create widgets through system suggestions, user prompts, or manual input, providing a flexible environment tailored to individual needs. This enables deeper engagement with the creative process. In two lab studies, PromptCanvas outperformed both the conversational UI (lab study 1) and the structured baseline, Wordcraft (lab study 2) on the Creativity Support Index. Participants found that it reduced cognitive load, while at the same time providing better performance. Qualitative feedback revealed that the visual organization of thoughts and easy iteration encouraged new perspectives and ideas. The field study ( \(N=10\) ) also confirmed these results, showcasing the potential of dynamic, customizable interfaces to improve collaborative writing with AI.
The rapid advancement of the automotive industry toward automated and semi-automated vehicles has rendered traditional methods of vehicle interaction, such as touch-based and voice command systems, inadequate for a widening range of non-driving related tasks, such as referencing objects outside of the vehicle. Consequently, research has shifted toward gestural input (e.g., hand, gaze, and head pose gestures) as a more suitable mode of interaction during driving. However, due to the dynamic nature of driving and individual variation, there are significant differences in drivers’ gestural input performance. While, in theory, this inherent variability could be moderated by substantial data-driven machine learning models, prevalent methodologies lean toward constrained, single-instance trained models for object referencing. These models show a limited capacity to continuously adapt to the divergent behaviors of individual drivers and the variety of driving scenarios. To address this, we proposed, in our previous work, an incremental learning approach, adapting to changing user behavior. Although this method proved superior over state-of-the-art approaches, it still lacked in performance and had some computational resource limitations, mainly the small-sized memory needed to store a subset of old data for incremental learning. Therefore, in this work, we propose an alternative solution using reinforcement learning that can be used to overcome memory storage limitations, achieving a zero data storage approach for incremental learning without performance loss. Our newly enhanced solution has been added to our previous framework at https://github.com/amrgomaaelhady/IcRegress .
Personalised AI coaching systems are increasingly used across healthcare, education, workplace training, and sport to support behaviour change and skill development through adaptive interventions. Despite this growth, the field remains fragmented, with limited systematic understanding of how such systems are designed, implemented, and evaluated. This systematic mapping review examines how personalised AI coaching systems are conceptualised and developed, focusing on application domains, target users, AI techniques, personalisation strategies, and interface modalities. Peer-reviewed journal and conference papers in English were included if they described a coaching technology, employed AI-driven methods to select or generate coaching interventions, and implemented automatic or semi-automatic personalisation. Following PRISMA-ScR guidelines, a comprehensive search across ACM Digital Library, IEEE Xplore, PubMed, and Web of Science retrieved 2072 records, of which 42 met the inclusion criteria. Data were systematically extracted on domains, user populations, coaching scope, AI techniques for intervention selection and personalisation, user modelling, theoretical grounding, interface modalities, and evaluation methods. Quantitative charting and qualitative synthesis were used to map trends and identify gaps. Results indicate that most systems targeted lay users, predominantly in healthcare, with fewer studies in education and minimal presence in professional or workplace settings. The recent uptake of deep learning introduced greater methodological diversity and enabled expansion across application domains. Personalisation largely focused on individual-level adaptation, was seldom grounded in theory or expert input, and was less frequently evaluated than coaching outcomes, often without independent assessment. Most systems relied on graphical user interfaces, while robotic, virtual reality, and game-based interfaces remained rare. Future research should extend beyond health to underrepresented domains and user groups, advance hybrid AI approaches that balance interpretability and performance, and strengthen theory-grounded and ethically informed design. Improved reporting transparency, more rigorous evaluation of personalisation effects, and clearer pathways from research prototypes to real-world deployment are also needed.
Computer Vision (CV) models can become outdated after deployment as real-world data evolves, requiring intensive attention from AI engineers to address degraded performance through tasks like data relabeling to update models with new human perceptions. Interactive human-in-the-loop systems have considerable potential to enhance model-steering practices. However, such workflows reveal two challenges: (1) scalability, where labor demands increase with data size, and (2) actionability, where human insights do not readily transform into model revisions. Based on our formative study (S1) on the current challenges faced by CV professionals, we developed De-Decay, an end-to-end Human-Data Alignment system offering scalable label-less assessment and actionable insight transformation. This enables engineers to investigate degradation and auto-retrain models with AI support, such as image clustering and regeneration. Our summative study (S2) showed that De-Decay helped engineers effectively identify and address CV degradation. We discuss how future research can enhance scalability and actionability in AI evaluation systems for aligning AI behaviors with human mental models.
The adaptability and intuitiveness of Human-Computer-Interaction systems are enhanced by emotion recognition capabilities, whose rapid advancement asks for updated and more complete surveys. In this comprehensive work, papers using at least one of three peripheral physiological signals (galvanic skin response, heart rate and respiration signals) were identified, resulting in 386 papers and 448 studies that were reviewed according to the entire emotion recognition pipeline, and not just based on types of signals and recognition methods as done in related work. Accordingly, this review identifies trends, challenges and opportunities across different aspects of the emotion recognition literature. Our investigation showed that multimodal approaches, benefitting from complementary physiological information, dominate the literature. Emotion-inducing methods tend to be dynamic and to progress towards real-life applications. To facilitate such applications, building novel datasets should be considered. For instance, there is room for novel continuously annotated datasets to facilitate the development of dynamic emotion models—which is also crucial for reliable real-life applications. At the same time, to guarantee a reliable continuous annotation, the combination of stimuli and assessment/report method should not be too overwhelming for the studies’ participants. Our results showed that support vector machines remain prevalent among traditional machine learning methods, but the growth of deep learning methods used either for feature extraction or end-to-end recognition is evident—both in number of studies and advanced developed techniques. Although a balance between algorithms’ performance and interpretability is essential in emotion recognition, there is a noticeable gap in integrating emotion theory into algorithms, which would improve such balance. Besides bringing to light a broad panorama of the literature, this work offers a digital table with the analysis of all studies and a filter possibility, allowing researchers to take advantage of it to accelerate and/or get inspiration for their own work.
The field of Human-Centred Explainable AI (HCXAI) has been rapidly expanding. In turn, there has been an increase in the number of papers suggesting design principles for HCXAI. However, it is unclear the extent to which design requirements overlap between papers, and in turn what the field overall considers to be HCXAI design requirements. To overcome this, this study analysed the state of the field via a scoping review of papers suggesting HCXAI design requirements, and a Content Analysis of the extracted principles. A total of 330 design principles were identified from 35 papers, which were subsequently categorised into 43 codes and grouped into 4 main areas of focus. Based on these findings, we propose a definition of HCXAI which identifies HCXAI as a design process rather than an XAI technique. Finally, an overview of the current state of HCXAI is presented, as well as areas where further research is required.
As AI systems increasingly shape decision-making in creative design contexts, understanding how humans engage with these tools has become a critical challenge for interactive intelligent systems research. This article contributes a challenge to rethink how to evaluate human-AI collaborative systems, advocating for a more nuanced and multidimensional approach. Findings from one of the largest field studies to date (n = 808) of a human-AI co-creative system, The Genetic Car Designer, complemented by a controlled lab study (n = 12), are presented. The system is based on an interactive evolutionary algorithm where participants were tasked with designing a simple 2D representation of a car. Participants were exposed to galleries of design suggestions generated by an intelligent system, MAP-Elites, and a random control. Results indicate that exposure to galleries generated by MAP-Elites significantly enhanced both cognitive and behavioral engagement, leading to higher-quality design outcomes. Crucially for the wider community, the analysis reveals that conventional evaluation methods, which often focus on solely behavioral and design quality metrics, fail to capture the full spectrum of user engagement. By considering the human-AI design process as a changing emotional, behavioral, and cognitive state of the designer, we propose evaluating human-AI systems holistically and considering intelligent systems as a core part of the user experience-not simply a back-end tool.
Nonspeaking autistic individuals (“nonspeakers”) represent about one third of the autistic population, yet most lack access to an effective alternative to speech. This lack of effective communication significantly limits their access to educational, social, and employment opportunities. Some nonspeakers have learned to spell words and sentences by pointing to letters on a physical letterboard held in their field of view by a trained human assistant. While effective, this method relies on the assistant for positioning the letterboard, limiting user autonomy and privacy. We report here a system we developed that uses Behavioral Cloning (BC) to automatically and adaptively position a virtual letterboard in Augmented Reality (AR). By observing finger, palm, head, and physical letterboard poses during real-life interactions between a nonspeaker and their assistant, we train a BC Machine Learning (ML) model that can adapt the placement of a virtual letterboard for that user. Results from 11 experiments (3 emulated scenarios and 8 nonspeaking autistic participants) show that our approach can accurately replicate the actions of the human assistant of any given user, outperforming a non-ML baseline personalized placement policy in both positional and rotational accuracies. Further, our novel BC formulation overcomes traditional data-efficiency limitations, allowing us to achieve high accuracy with a modest training effort. This work represents a foundational step toward enabling more autonomous and private communication for nonspeakers.
The adoption of Recommender Systems (RSs) in various domains has become increasingly popular, but concerns have been raised about their lack of transparency and interpretability. While significant advancements have been made in creating explainable RSs, there is still a shortage of automated approaches that can deliver meaningful and contextual human-centered explanations. Numerous studies have evaluated explanations based on human-generated recommendations and explanations to address this gap. However, such approaches do not scale for real-world systems. Building on recent research that exploits Large Language Models (LLMs) for RSs, we propose leveraging the conversational capabilities of ChatGPT to provide users with personalized, human-like, and meaningful explanations for recommended items. Our article presents a user study with 94 participants that measures users’ perceptions of ChatGPT-generated explanations when acting as a recommender system. Regarding recommendations, we assess whether users prefer ChatGPT over random (but popular) recommendations. Concerning explanations, we assess users’ perceptions of personalization, effectiveness, and persuasiveness. We also break down the explanations in its constituent arguments and investigate the differences in argument types between generic and user-specific explanations. Our results show that participants rated ChatGPT’s recommendations significantly higher than random ones ( \(\beta=-.53\) , \({\textrm{p}} < .001\) ). Surprisingly, user-based explanations that explicitly referenced participants’ preferences were not perceived as more personalized or persuasive than generic ones, except when the recommended movie was unfamiliar, where user-based explanations became more effective ( \(\beta=0.35\) , \({\textrm{p}} < .05\) ). Overall, our findings highlight both the promise and the limitations of using LLMs for generating personalized explanations, suggesting that personalization is most impactful when users lack prior knowledge of the item.
Recommender systems are increasingly used in mobile health interventions, such as managing Chronic Musculoskeletal Pain (CMP). While researchers have highlighted the importance of explaining health-related recommendations to lay users, with benefits such as increased trust and a higher tendency to follow up on these recommendations, how to design explanations for lay users in critical contexts such as health remains largely unexplored. To address this gap, we develop a mobile health application to support users with CMP through coaching and personalised health recommendations delivered via a conversational rule-based recommender system. This article describes the three-phase iterative development of the RS, involving health experts and end users. In the first iteration, we conduct a preliminary validation study with N = 282 participants to ensure the app's validity and improve the initial set of health recommendations. Next, two user studies are conducted centred around designing effective and understandable explanations for these recommendations. First, we design six explanation modalities tailored towards lay users, and through a qualitative study (N = 11), extract initial design guidelines for explaining health recommendations, finding a strong preference towards feature importance explanations and identifying issues with modalities that highlight negative emotions. Given these results, we explore whether extending feature importance explanations with textual information into a 'hybrid' explanation could benefit end users, and whether these benefits depend on a user's personal characteristics (need for cognition and ease-of-satisfaction). Through a mixed-methods study with N = 262 participants, we find that the hybrid modality significantly increased user trust, transparency, persuasiveness, usefulness and satisfaction compared to unimodal explanations. However, users with a higher need for cognition rate unimodal explanations more positively than hybrid ones.
Modelling human gaze behaviour on 360 degrees images is important for various human-computer interaction applications. However, existing methods are limited to predicting discrete fixation sequences or aggregated saliency maps, thereby neglecting fine-grained gaze behaviour such as saccadic eye movements that can be captured by commercial eye-trackers. We introduce a more challenging task-fine-grained gaze sequence generation. This task aims to generate eye-tracker-like gaze data for given stimuli. We propose DiffGaze, a diffusion-based method for generating realistic and diverse fine-grained human gaze sequences conditioned on 360 degrees images. We evaluate Diff Gaze on two 360 degrees image benchmarks for fine-grained gaze sequence generation as well as two downstream tasks, scanpath prediction and saliency prediction. Our evaluations show that Diff Gaze outperforms the fine-grained gaze generation baselines in all tasks on both benchmarks. We also report a 21-participant survey study showing that our method generates gaze sequences that are indistinguishable from real human sequences. Taken together, our evaluations not only demonstrate the effectiveness of Diff Gaze but also point towards a new generation of methods that faithfully model the rich spatial and temporal nature of natural human gaze behaviour.
The proliferation of misinformation in journalism, often stemming from flawed reasoning and logical fallacies, poses significant challenges to public understanding and trust in news media. Traditional fact-checking methods, while valuable, are insufficient for detecting the subtle logical inconsistencies that can mislead readers within seemingly factual content. To address this gap, we introduce Skeptik, a hybrid framework that integrates Large Language Models (LLMs) with heuristic approaches to analyze and annotate potential logical fallacies and reasoning errors in online news articles. Operating as a web browser extension, Skeptik automatically highlights sentences that may contain logical fallacies, provides detailed explanations, and offers multi-layered interventions to help readers critically assess the information presented. The system is designed to be extensible, accommodating a wide range of fallacy types and adapting to evolving misinformation tactics. Through comprehensive case studies, quantitative analyses, usability experiments, and expert evaluations, we demonstrate the effectiveness of Skeptik in enhancing readers' critical examination of news content and promoting media literacy. Our contributions include the development of an expandable classification system for logical fallacies, the innovative integration of LLMs for real-time analysis and annotation, and the creation of an interactive user interface that fosters user engagement and close reading. By emphasizing the logical integrity of textual content rather than relying solely on factual accuracy, Skeptik offers a comprehensive solution to combat potential misinformation in journalism. Ultimately, our framework aims to improve critical reading and protect the public from deceptive information online and enhance the overall credibility of news media.
Can AI be cognitively biased in automated information judgment tasks? Despite recent progresses in measuring and mitigating social and algorithmic biases in AI and large language models (LLMs), it is not clear to what extent LLMs behave "rationally", or if they are also vulnerable to human cognitive bias triggers. To address this open problem, our study, consisting of a crowdsourcing user experiment and a LLM-enabled simulation experiment, compared the credibility assessments by LLM and human judges under potential decoy effects in an information retrieval (IR) setting, and empirically examined the extent to which LLMs are cognitively biased in COVID-19 medical (mis)information assessment tasks compared to traditional human assessors as a baseline. The results, collected from a between-subject user experiment and a LLM-enabled replicate experiment, demonstrate that 1) Larger and more recent LLMs tend to show a higher level of consistency and accuracy in distinguishing credible information from misinformation. However, they are more likely to give higher ratings for misinformation due to the presence of a more salient, decoy misinformation result; 2) While decoy effect occurred in both human and LLM assessments, the effect is more prevalent across different conditions and topics in LLM judgments compared to human credibility ratings. In contrast to the generally assumed "rationality" of AI tools, our study empirically confirms the cognitive bias risks embedded in LLM agents, evaluates the decoy impact on LLMs against human credibility assessments, and thereby highlights the complexity and importance of debiasing AI agents and developing psychology-informed AI audit techniques and policies for automated judgment tasks and beyond.
The need to generate a spider to provoke a desired anxiety response arises in the context of personalized virtual reality exposure therapy (VRET), a treatment approach for arachnophobia. This treatment involves patients observing virtual spiders in order to become desensitized and decrease their phobia, which requires that the spiders elicit specific anxiety responses. However, VRET approaches tend to require therapists to hand-select the appropriate spider for each patient, which is a time-consuming process and takes significant technical knowledge and patient insight. While automated methods exist, they tend to employ rules-based approaches with minimal ability to adapt to specific users. To address these challenges, we present a framework for VRET utilizing procedural content generation (PCG) and reinforcement learning (RL), which automatically adapts a spider to elicit a desired anxiety response. We demonstrate the superior performance of this system compared to a more common rules-based VRET method.
