Artificial Intelligence (AI) and digital devices are becoming more pervasive in the monitoring of individuals’ health as device functionalities increase and their presence in daily life continues to grow. Subsequently, these tools can provide transformative healthcare practice by enabling more personalized, proactive, and engaging interventions. This work presents the design and development of an interactive physical therapy system for adolescents that integrates a wearable EMG leg sensor array with a computer-based video game. The system captures EMG signals during therapeutic exercises and uses them to drive in-game mechanics, providing real-time feedback while also aiming to improve therapeutic engagement. A key feature of the system is its ability to adapt based on individual performance and clinician input, allowing personalized therapy progression and reporting. Using this approach, the system supports continuous personalized assessment and customized intervention for children while promoting motivation through dynamic gameplay. The developed system illustrates a promising direction for designing interactive, AI-augmented health technologies that can enhance pediatric rehabilitation outcomes through accessible, game-based experiences. Further, this work contributes to ongoing research in human-computer interaction, AI, and behavioral health by exploring multisensory interactions, social AI elements, and methods for evaluating user engagement.
Decentralized study settings have been proven to increase efficiency and diversity in clinical trials and research studies. However, one of the biggest challenges of decentralized settings is high dropout rates and low adherence to the study protocol. Gamification has shown to be a promising tool to increase motivation and engagement in different settings, but very few tools exist implementing personalization, which is crucial to address the needs of diverse user contexts. In this paper, we present our vision of using gamification to improve adherence and retention in digital, decentralized research studies and clinical trials. We aim to design, implement and evaluate a modular, AI-based gamification engine designed to personalize motivational strategies based on user profiles and context. Its modular, test-agnostic design approach enables the integration across platforms and study types, offering the potential of a reusable infrastructure for adherence support in digital health research.
The growing global burden of chronic conditions like diabetes and osteoarthritis necessitates innovative, scalable rehabilitation strategies. This paper proposes a conversational AI system designed to deliver dynamic, personalized physical rehabilitation plans. The system leverages large language models (LLMs) and physiotherapist supervision to adapt recommendations based on user feedback and physiological data. A key novelty of this work is its avatar-based AI agent, designed with the potential for integration into extended reality (XR) environments, thus aligning with emerging AR glass technologies. We present an early-stage prototype supported by a modular architecture and propose a pathway toward implementation and evaluation to enhance patient adherence, safety, and engagement in long-term rehabilitation.
Wayfinding is an important skill in the context of everyday life. Especially in unknown environments, spatial orientation in real life relies heavily on landmarks. The present study investigated the impact of outdoor and indoor landmarks in a wayfinding task performed in unknown complex office buildings in Virtual Reality. In order to investigate the performance measures of orientation, twenty-two participants had to find the conference room in the office buildings. The office buildings were constructed in the manner of mazes, incorporating dead ends and loops, and were constructed as two-story buildings. These buildings had no landmarks, indoor landmarks, outdoor landmarks or both types of landmarks. At the end, participants had to draw a digital sketch map to show the way to the conference room. The presence of landmarks led to more accurate sketch maps, in comparison to conditions without landmarks. While a discrepancy in accuracy was observed between the outdoor and the indoor landmark conditions in the data, this discrepancy did not attain statistical significance.
Recent technological advances enable seamlessly changing between different degrees of reality and virtuality using consumer hardware. This so-called cross-reality brings new chances to solve tasks that augmented and virtual reality alone were not fit for, but it also brings new challenges, for example when users need a mental representation of the content that is currently not visible and when choosing and executing transitions between realities. I propose to use context awareness to lower the mental load in such situations and to improve overall user-friendliness. My thesis begins with a literature research to gain an overview of the specific challenges and existing approaches in cross-reality context awareness. From the results, first, a taxonomy will be developed. There will also be a solution in the form of a framework supporting the inclusion of context awareness into cross-reality experiences. The taxonomy will additionally be the base for cross-reality building blocks that can be used in the framework. The results will be evaluated in developer and end-user studies to ensure that the framework is usable and solves the found challenges.
Artificial Intelligence (AI) is increasingly being integrated into a wide array of Extended Reality (XR) applications, including sophisticated navigation systems, immersive training simulations for educational use-cases and data analysis in medicine, e.g. for MRI scans. Consequently, ensuring the transparency and interpretability of these AI-driven applications has become a major challenge. This paper examines the growing importance of Explainable AI (XAI) in Extended Reality environments and identifies key challenges in developing effective explanation systems. We analyze how these AI-powered XR applications particularly benefit from transparent explanations that build trust, enhance user understanding and improve overall adoption. After summarizing the general challenges in the field of XAI, we investigate how these challenges manifest in the specific context of XR. By synthesizing current research and identifying critical open questions, this work aims to guide future XAI development towards more transparent, trustworthy systems that prioritize human needs across XR applications and beyond.
Formal verification has the potential to play a central role in the development of interactive safety-critical systems by providing rigorous guarantees about system behavior. However, the effective use of verification results remains a challenge in practice, particularly when these results must be interpreted by designers and domain experts who will not be formal methods specialists. This paper explores the use of Large Language Models (LLMs) to generate natural language explanations of counterexamples produced by model checking. More specifically, we present a study evaluating how different LLMs handle counterexamples produced from a range of formal models. Our focus is on the potential of LLMs to serve as mediators between formal verification tools and the multidisciplinary teams that design, develop, and validate interactive systems. The goal is to bridge the gap between formal outputs and human understanding. By examining the limitations and opportunities of the use of LLMs, we contribute to a broader discussion on integrating AI technologies into the engineering of trustworthy interactive systems.
As Active Assisted Living (AAL) systems evolve, a key challenge is to provide timely assistance without overwhelming users with constant interaction. We introduce the concept of Silent Interfaces—interaction paradigms that remain unobtrusive in everyday life yet engage precisely when necessary. Building on principles from Calm Technology and Situation Awareness, we outline a human-centered design framework and a technical architecture based on semantic rule sets, sensor fusion, and AI-driven event interpretation. Our approach, implemented in the uCORE platform, focuses on multimodal, context-triggered cues that range from subtle visual or haptic signals to autonomous emergency calls. Field deployments in assisted living facilities indicate increased user acceptance, reduced alert fatigue, and improved safety, without compromising privacy or autonomy. While these insights stem from practical operation rather than controlled studies, they highlight the potential of Silent Interfaces to extend human perception and enable proactive care. The concept offers guidance for designing unobtrusive, trustworthy, and scalable AAL solutions.
Emotion regulation (ER) is an essential skill that significantly impacts children’s social and emotional development. While prior research has shown that parents play a critical role in shaping their children’s ER abilities, the challenges and complexity in parental-involved ER interactions call for technology that delivers context-aware and personalized social support. Although social robotics and Large Language Models (LLMs) both show promise for ER facilitation, few systems integrate language-based reasoning with embodied actions to address mental health needs effectively. To expand the potential applications, we developed an LLM-powered robotic system to facilitate ER in parent-child dyads. We adopt a supervised autonomy approach to integrate natural language dialogues with physical robotic behaviors for multimodal interactions. We detail the technical implementation and interaction design of the system, along with preliminary user tests involving six parent–child dyads. The findings highlight the positive user engagement and trust in interactions with the LLM-powered social robot. Accordingly, we discuss design insights and implications in developing LLM-powered multimodal and autonomous social robot systems for family-centered mental health applications.
Air pollution is a significant public health concern in many urban environments. A small number of fixed-location monitoring stations provide data that when aggregated and averaged can present a useful picture of an urban environment’s pollution levels. However, this is often insufficient for individuals with acute respiratory health conditions. Wearable air quality monitors offer a promising alternative by enabling localized, real-time pollutant sensing. In this research, such monitors are defined as devices that detect airborne pollutants and contextualise these readings against established health standards provided by governmental or international agencies, thereby supporting informed personal decision-making. The development of wearable AQMs faces key challenges in sensor accuracy, wearability, usability, and public trust. This doctoral research addresses these challenges through the development of an interdisciplinary, user-centred design framework that integrates principles from human-computer interaction (HCI), fashion technology, and environmental sensing. The aim is to bridge the gap between technical feasibility and lived user experience, supporting the development of inclusive, acceptable, and practically deployable solutions for everyday air quality monitoring.
As AI systems increasingly permeate design education, the dominant interaction paradigm (text-based chat) risks constraining cognitive engagement in complex, iterative design tasks. This work explores whether and how multimodal interaction methods (e.g., visual, auditory, haptic) enhance cognitive performance compared to traditional chat-based AI interfaces within human-centered design (HCD) education. Based on an initial targeted literature analysis, multimodal interfaces have demonstrated benefits such as reduced cognitive load, increased user engagement, improved learning outcomes, and enhanced collaborative processes. The current work is situated within a broader doctoral research project investigating how generative AI reshapes cognitive design processes in novice designers. Building on an earlier case study that demonstrated that chat interfaces often lead to superficial understanding and linear thinking, this article urges a rethinking of AI-assisted design education as a multimodal, situational, and didactically coordinated experience. The current workshop provides a platform to exchange frameworks and strategies for engineering multimodal, cross-device AI experiences that better serve cognitive growth and design literacy in the generative age.
Critical interactive systems operate in high-consequence environments where automated decisions have immediate and significant impact. Artificial Intelligence (AI) offers substantial potential to support human operators in these settings, yet its adoption introduces challenges related to transparency, interpretability, and trust. This paper provides a systematic analysis of the opportunities and challenges of embedding Explainable AI (XAI) in such systems, with attention to core properties including usability, dependability, safety, privacy, and confidentiality. We illustrate these considerations through a case study on an AI-driven early-warning system for atmospheric turbulence in commercial aviation, emphasizing the interactions between algorithmic explanations, human cognition, operational constraints, and regulatory requirements. Based on this analysis, we propose research directions that address methodological, human-centered, and regulatory challenges for the practical integration of XAI in safety-critical interactive systems.
Structured input tasks, such as form filling and data entry, are common across domains but can be time-consuming and cognitively demanding, especially when users must convert free-form descriptions into rigid formats. This paper explores using Large Language Model (LLM) agents to support transforming unstructured natural language and document-based inputs into structured outputs aligned with predefined schemas. We describe a modular pipeline that takes as input a natural language description, optional supporting documents (e.g., PDFs, spreadsheets), and a form schema. The LLM processes these inputs to generate structured key-value pairs with confidence scores, which can be used to populate forms automatically while allowing users to review and adjust the output. The system follows a mixed-initiative approach, emphasizing human oversight and editable results. Principles from human-centered AI and adaptive interface engineering guide our design. Rather than presenting a full empirical evaluation, this work contributes a modular architecture and design perspective on embedding LLM agents into structured input workflows, highlighting integration challenges and early feasibility observations in real-world contexts such as the MICS project. The approach is domain-agnostic and compatible with existing infrastructures through the Model-Context Protocol (MCP).
In context-aware systems, user interface adaptation often occurs automatically, offering limited transparency and minimal user control. To overcome this limitation, this research elaborates on the concept of the Extra-User Interface, an additional user interface layer designed to empower users by enabling real-time observation, customization, and steering of adaptation behaviours. Positioned above the primary user interface, an Extra-User Interface extends the meta-user interface concept by providing structured services, such as inspecting and modifying underlying user interface artefacts. To assess the impact of Extra-User Interfaces on user experience, we adopt a Goal-Questions-Metrics approach, focusing on user satisfaction, comprehension, and performance. The research follows a multi-layered method, combining conceptual development, model-based design, and empirical validation. As a proof of concept, a prototype was developed and tested in a mid-air gesture-controlled environment, showing how an Extra-User Interface can enhance user autonomy and efficiency in controlling adaptation. Moreover, the research explores the applicability of extra-user Interfaces in high-stakes domains through the development of a second prototype for a data visualization interface designed to support critical decision tasks. Contributions of this work are threefold: (1) a theoretical framework defining Extra-User Interface concepts and primitives; (2) a model-based approach for designing graphical user interfaces whose adaptation is controlled by an extra-user interfaces; and (3) the implementation and evaluation of Extra-User Interface environments in realistic settings. Preliminary results validate the feasibility and usefulness of Extra-User Interfaces as a user-centric solution for controllable user interface adaptation in ambient intelligence and ubiquitous computing.
Tangible interactions have been widely studied, involving various objects equipped with different sensors in different application domains. Among these, tangible cube interfaces represent a particularly promising modality for these domains, including educational contexts of use, as they promise to improve learning. However, gestural interaction with tangible cubic objects remains mostly task- or domain-specific, limiting their reusability and generalizability. This paper investigates gestural interaction with tangible cubic objects, focusing on AudioCubes, through a mixed-method approach that integrates a systematic literature review, gesture elicitation studies, and usability evaluations. The findings of this study will contribute to the following: (i) a classification of reusable gestures for tangible cubes, (ii) some insights into common barriers for diverse user groups, including children and users with motor or cognitive differences, and (iii) some design implications for more adaptable and generalizable gestural interaction with tangible cubes.
While the increased integration of AI technologies into interactive systems enables them to solve an increasing number of tasks, the black-box problem of AI models continues to spread throughout the interactive system as a whole. Explainable AI (XAI) techniques can make AI models more accessible by employing post-hoc methods or transitioning to inherently interpretable models. While this makes individual AI models clearer, the overarching system architecture remains opaque. This challenge not only pertains to standard XAI techniques but also to human examination and conversational XAI approaches that need access to model internals to interpret them correctly and completely. To this end, we propose conceptually representing such interactive systems as sequences of structural building blocks. These include the AI models themselves, as well as control mechanisms grounded in literature. The structural building blocks can then be explained through complementary explanatory building blocks, such as established XAI techniques like LIME and SHAP. The flow and APIs of the structural building blocks form an unambiguous overview of the underlying system, serving as a communication basis for both human and automated agents, thus aligning human and machine interpretability of the embedded AI models. In this paper, we present our flow-based approach and a selection of building blocks as MATCH: a framework for engineering Multi-Agent Transparent and Controllable Human-centered systems. This research contributes to the field of (conversational) XAI by facilitating the integration of interpretability into existing interactive systems.
AutoML systems targeting novices often prioritize algorithmic automation over usability, leaving gaps in users' understanding, trust, and end-to-end workflow support. To address these issues, we propose an abstract pipeline that covers data intake, guided configuration, training, evaluation, and inference. To examine the abstract pipeline, we report a user study where we assess trust, understandability, and UX of a prototype implementation. In a 24-participant study, all participants successfully built their own models, UEQ ratings were positive, yet experienced users reported higher trust and understanding than novices. Based on this study, we propose four design principles to improve the design of AutoML systems targeting novices: (P1) support first-model success to enhance user self-efficacy, (P2) provide explanations to help users form correct mental models and develop appropriate levels of reliance, (P3) provide abstractions and context-aware assistance to keep users in their zone of proximal development, and (P4) ensure predictability and safeguards to strengthen users' sense of control.
Prototyping multimodal interaction in XR (Augmented/Virtual Reality) is challenging for non-technical designers. Moreover, existing XR prototyping tools do not support concurrent multimodal interaction and lack an overall view of the interaction design of the experience. We created Mucho , a novel no-coding immersive tool for prototyping interactive multimodal XR experiences. Mucho supports three modes: In recording, designers can demonstrate input examples to populate a timeline representation with events. In playback, designers can create a video prototype based on the timeline while adding system actions at the corresponding timeframes. In live mode, Mucho creates a state machine to test the interactive experience in runtime. We present an evaluation by demonstration to illustrate how Mucho can prototype a diverse set of examples including modalities such as hand gestures, proximity, speech, and gaze. Finally, we discuss the limitations of our approach and outline future directions to support immersive prototyping of multimodal XR experiences.
Internet of Things-enabled home automation is starting to significantly transform our daily lives. Users must be able to configure and coordinate the connected objects in their dwellings to personalise and fully benefit from their potentialities. Trigger-action programming is one relevant approach to enable users to create useful automations. However, current approaches mainly based on visual Web or mobile interfaces have limitations, such as the difficulty in finding and selecting the correct object and associated services. Mobile augmented reality is an interaction modality that can support more direct and usable automation control. In this perspective, the support of a recommendation system can facilitate users in creating personalised automations. This paper presents a system for generating personalised daily automations recommendations and presenting them in a mobile augmented reality solution in order to facilitate their monitoring and creation. Seven classification approaches were assessed on different datasets to determine their ability to provide personalised and context-aware recommendations. We also report a user study (N = 16) showing that the personalised recommendation system improves user performance when creating automations in a trigger action format with a mobile augmented reality editing environment.
The growing capabilities of microcontrollers, sensors, and actuators, coupled with decreasing costs, have led to a proliferation of embedded interactive systems. Prototyping such electronic systems has become democratized across a broad audience, including students, hobbyists, professional engineers, and programmers. Central to this evolution is the ease of software development, and in particular, the availability of low-level drivers and programming libraries which have significantly lowered the barriers to programming these systems. However, this ecosystem often presents challenges due to the tight coupling between programming libraries, drivers, and the underlying sensors and actuators. This frequently leads to compatibility issues. This paper introduces LogicGlue, which addresses these challenges by providing a platform-independent driver specification format. LogicGlue driver specifications allow hardware-independent application logic to be written, facilitating the process of interchanging components with minimal-to-no code adjustments. Unlike existing solutions, LogicGlue supports efficient interfacing via native communication protocols. This approach not only simplifies electronics prototyping but also ensures compatibility between various types of electronic components from different vendors. By reducing the complexity of hardware integration, LogicGlue enables a more seamless exploration of novel interactive behaviours and interfaces, forming a new tool for engineering interactive computing systems.
