Postsecondary STEM courses include substantial mathematics and algorithms. Students need motivation to dig deep into the topic. Multisensory modeling allows users to explore objects with multiple senses, i.e., audio, visual, and touch. If augmented with virtual reality (VR), the overall experience could be more immersive and enjoyable. This research investigated students’ experience with unimodal versus multimodal visualization and exploration of wave functions with and without VR integration. Students interactively explored wave functions, i.e., sine, cosine, sawtooth, square, and triangular waves, with 3D models to concretize understanding of wave parameters, i.e., frequency, amplitude, phase, and vertical shifts. Initial findings showed that students preferred the audio-visual exploration of wave functions over the visual-only version in VR-enhanced and non-VR platforms. Overall, VR enriched their experience while interacting with the data.
Autism Spectrum Disorder (ASD) is a developmental disability often characterized by sensory processing difficulties that can lead to anxiety, particularly in children and adolescents. Previous research on virtual reality-based anxiety intervention tools focuses on using social skills training, exposure therapy, and meditative coaching to mitigate social and phobia related anxiety. However, minimal work has specifically evaluated the effects of virtual multi-sensory environments for people with ASD, often only testing feasibility. This pilot study aims to build on previous work by investigating how various auditory, visual, and interactive components contribute to user satisfaction and sensory-related anxiety reduction. The objective is to gain a better understanding of what features are significant towards developing a successful virtual anxiety intervention tool. Results suggest using interactive activities that promote fine motor skills can provide a healthy outlet for self-mediated stress relief. Future development aims to incorporate task-based activities, and enhance audio, visual, and lighting displays. The deployment of a full-scale study with a larger sample size and target participant pool is warranted to substantiate these initial findings.
The design of the lighting environment is important for determining room specifications. Recently, systems that can change the lighting environment in a room by controlling the direction and intensity of light using a computer have been proposed. However, with such a method of controlling the light, it is easy to specify the size and position of the brightly illuminated area, but not the darkened area. In this study, we controlled shadows and proposed a method for controlling dark areas. We construct Spot Shadow, a system that can generate shadows of arbitrary sizes and shapes at positions specified by users. The prototypes of a tabletop system and a large system can create shadow generation areas of 1 x 0.9m(2) and 7 x 4.3m2, respectively. This study proposes a method for manipulating shadows and demonstrates new possibilities for creating interactive spaces using shadows.
Polarization has been considered to be a reference instead of near-infrared light sources in eye tracking because the light emitted from a liquid-crystal display is typically polarized. However, the degree of polarization depends on the display content. Thus, devising a novel method is crucial for stably extracting the display reflection from the corneal surface. Therefore, we propose an eye-tracking method that inserts a white background between the display contents using a high-speed display to extract the screen reflection on the cornea. A high-speed camera and polarization modulator are integrated to extract the polarized light emitted from a high-speed display, and then the point-of-gaze is estimated. We evaluated the accuracy of the estimated point-of-gaze under several conditions to compare the proposed method with conventional approaches. The results revealed that the proposed method improved eye gaze estimation.
Designing accessible locomotion methods for individuals who are blind or have low vision (BLV) is a complex challenge, particularly in mobile VR environments with limited interface options. In this paper, we propose a novel locomotion technique on mobile VR that enables users to control a virtual character’s movement while staying stationary or within a small physical area. The technique utilizes the phone’s gyroscope for movement control, while providing spatial audio and vibration feedback to enhance virtual exploration for BLV individuals. Our study examines how BLV individuals acquire spatial knowledge in mobile VR environments. A user study is conducted to assess the effectiveness of the proposed approach.
Visual methods have become increasingly vital in Human Computer Interaction (HCI) research, particularly as we analyze and interpret the complex visual data that emerges from various interaction modalities. However, the methodologies for analyzing this visual data remain underdeveloped compared to textual data analysis. This workshop seeks to unite HCI researchers who work with visual data—such as hand sketches, photographs, physical artifacts, UI screenshots, videos, and information visualizations—to identify, name, and categorize methods for analyzing visual data in HCI.
The persistent gender gap in computer science, especially among women from low-income backgrounds, continues to limit diversity and innovation within the technological sector. This underrepresentation also restricts access to career paths that can enhance social mobility, particularly for women in developing countries. Physical computing offers a hands-on approach that can improve programming skills and computational thinking through interaction with tangible hardware. This research focuses on developing a short physical computing workshop tailored to young girls from low-income communities. The study combines education, tangible interfaces, and coding. Through a series of classroom-based studies and laboratory experiments, this PhD work will assess the impact of the workshop on self-efficacy and learning in programming and computational thinking. The anticipated contributions of this research include insights into the effectiveness of tangible, user-friendly physical computing workshops in increasing engagement among underrepresented groups in computer science.
Interfaces and visualizations often challenge comprehension, especially as they grow in complexity. Traditional methods—relying on standard inputs like touch, mouse, and keyboard—fall short in addressing the nuanced demands for explainability in complex systems. This workshop explores innovative interaction strategies to enhance self-enabled comprehension, focusing on the development and refinement of new devices and input modalities. We aim to gather researchers, designers, and practitioners to exchange ideas and explore interaction techniques that promote clearer understanding and transparency across diverse applications. This collaborative effort seeks to advance interactive systems that improve explanation and user comprehension in digital environments.
This PhD project introduces a mixed reality passive stylus system designed for smartphones enabling digital ink creation on a surface. Traditional passive stylus systems face challenges in usability and accessibility. This research aims to overcome these limitations through a dual approach that integrates interactive design and robust machine learning, trained on extensive datasets. The research methodology is divided into phases of data collection, design, implementation and evaluation. The project develops an application to deliver an accessible digital tool viable for widespread use, particularly in developing regions.
The increasing global prevalence of obesity and related health issues underscores the need for innovative dietary interventions. This paper explores the potential of combining gamification and Virtual Reality (VR) to promote healthier eating habits among young adults. By creating an interactive VR food environment with engaging game elements, we aim to assess the impact of gamified VR intervention on nutritional knowledge and attitudes. Preliminary results show an increase in nutritional understanding and awareness, though further research is necessary for statistical validation. This study suggests that VR-based gamified interventions could be a promising tool for nutrition education, behavior modification, and virtual food selection.
Many children with cerebral palsy (CP) should do various exercises to restore motor control for specific functions such as hand grasping and gripping. During daily exercises, they need intensive support from either therapists or caregivers in setting tasks and providing feedback, which creates a heavy workload. Thus, we introduce a two-handed ellipsoidal device to control computer games for interactive grasping and gripping rehabilitation training. The ellipsoidal device is designed to house an ESP32 microcontroller, a Wheeltec N100 IMU and an SF15 flexible thin-film pressure sensor so as to monitor children’s grip strength and wrist rotation. The sensing data can be used to control the characterizer motion in computer games. Preliminary user trials supported the implementation of such devices in hospitals for the hand grasping and gripping exercise and the cognition and coordination exercise between eyes, ears and hands.
Immersive data storytelling is an emerging field that combines narrative visualisation and immersive analytics to engage an audience. While there are existing design spaces for narrative visualisation on 2D displays, there are no guidelines for creating immersive data stories, making it difficult for practitioners and researchers to explore this space. In this paper, we present a preliminary design space for immersive data storytelling that is informed by current practices and multi-disciplinary views. We interviewed multi-disciplinary experts, including museum designers, architects, and game designers, to understand how they communicate stories in physical spaces and immersive mediums. We applied inductive thematic analysis to the interview responses to inform the dimensions of the design space and analysed a systematic selection of publicly available immersive stories. In the end, we had 13 dimensions in 7 categories.
The initial phase of innovative product design is marked by uncertainty and complexity. This paper examines the use of participatory workshops to navigate this phase within the ToCaro research project. The project aims to develop tactile and multisensory interfaces for remote communication to mitigate feelings of loneliness by promoting a sense of physical proximity. Fourteen co-design workshops were conducted with senior participants (age >= 65) to examine their communication behaviors, identify latent needs and evaluate physical sensations elicited by various materials and forms of interaction. The workshops included semi-structured interviews, sensory perception tests, interaction concept evaluations, and "quick-and-dirty" prototyping. This paper outlines the facilitators' experiences, the challenges, and learnings. Results indicate that while participants exhibited varied levels of engagement, those with a perceived need for new communication devices contributed effectively to the creative process.
In balance training, such as ballet, observing one's posture in a mirror makes it easier to maintain balance. By using projectors, it is possible to show the user's posture from different angles, magnify specific body parts, and display the Center of Pressure (COP) trajectory in real time. This gives the user more visual feedback information than a mirror can provide, helping to improve balance. However, the appropriate projection position and size to enhance the effect of such visual feedback remains unclear. This study focuses on releve in ballet and examines the effects of different types and positions of visual feedback on balance improvement. We conducted a user study and calculated balance metrics from COP data obtained from a balance board. The results indicate that only visual feedback projected directly at eye level in front of the user during releve contributes to balance improvement. In contrast, visual feedback projected above eye level to the right did not show a clear effect on balance improvement.
Self-guided tutorials are popular resources for learning new tasks, but they lack important aspects of in-person guidance like feedback or personalized explanations. Adaptive guidance systems aim to overcome this challenge by reacting to users’ performance and expertise and adapting instructions accordingly. We aim to understand the users’ preferred balance of automation and control, what representation of instructions they prefer, and how human experts give instructions to match users’ needs. We contribute an experiment where users perform different virtual tasks, guided by instructions that are controlled by experts using a wizard-of-oz paradigm. We employ different levels of automation to control instructions and alter their level of detail and step granularity to match the user’s needs. Results indicate that while users preferred automated systems for convenience and instant feedback, they appreciated a degree of manual control since they felt less rushed. Experts relied on factors such as expected expertise, hesitation, errors, and their understanding of the current task state as main triggers to adapt instructions.
Cerebral Palsy (CP) affects motor coordination, resulting in slow walking and irregular step and stride lengths. Effective rehabilitation exercises are essential for strengthening leg muscles and enhancing mobility in children with CP. However, traditional hospital rehabilitation programs often lack engagement, making it challenging for children to maintain consistent participation. Additionally, many advanced lower extremity rehabilitation systems remain largely inaccessible. This study introduces an interactive gaming carpet combined with intelligent gait pattern analysis to enhance rehabilitation efforts. The gaming carpet employs visual and auditory cues to train leg coordination and correct stepping patterns, making the exercises more engaging for children. Meanwhile, the intelligent gait analysis system provides therapists with objective data to assess conditions and develop personalized exercise plans. Initial tests indicate that this system effectively engages children and improves adherence to rehabilitation exercises, while also providing accurate progress monitoring. This innovative approach demonstrates significant potential for integrating game-based interventions and data analysis into CP rehabilitation, offering practical solutions for both clinical and home-based settings.
Spatially-stable touch menus, like FastTap, leverage users’ spatial memory to enable rapid command selection on tablets. Although these spatial tablet interfaces can aid in developing spatial memory of commands having a small command set, it is, however, unknown whether spatial memory remains beneficial when the number of commands grows. Therefore, we carried out a study to investigate spatial learning in four different sizes of single-tab FastTap Menus: Small, Medium, Large, and Extra-Large, with 16, 30, 42, and 56 items, respectively. Results indicated that people do develop spatial memory in all menus; however, there is a negative correlation between command capacity and spatial memory development in tablets. We contribute new knowledge on spatial memory development in touch tablets that can enhance the design of future spatial memory-based tablet interfaces.
Mixed reality enables users to immerse themselves in high-workload interaction spaces like office work scenarios. We envision physiologically adaptive systems that can move users into different mixed reality manifestations, to improve their focus on the primary task. However, it is unclear which manifestation is most conducive for high productivity and engagement. In this work, we evaluate whether physiological indicators for engagement can be discriminated for different manifestations. For this, we engaged participants in a typing task in three different mixed reality manifestations (augmented reality, augmented virtuality, virtual reality) and monitored physiological correlates (EEG, ECG, and eye tracking) of users' engagement and workload. We found that users achieved best typing performances in augmented reality and augmented virtuality. At the same time, physiological engagement peaked in augmented virtuality, while workload decreased. We conclude that augmented virtuality strikes a good balance between the different manifestations, as it facilitates displaying the physical keyboard for improved typing performance and, at the same time, allows one to block out the real world, removing many real-world distractors.
The Dual Gaussian Distribution Model can be utilized for predicting the success rates of tapping targets. However, previous studies have shown that the prediction error increases to as much as 10 points, where "points" represent the percentage difference between the observed and predicted values of the tap success rate, particularly for a small target width W such as 2 mm. We hypothesize that this could be due to the experimental designs with sparse W levels performed by few participants, rather than the model itself. Our experiment involving horizontal and vertical bar targets with W = 2-8 mm (step: 0.2 mm) performed by more than 180 participants showed that the maximum prediction errors were relatively small: 2.769 and 3.185 points, respectively. Furthermore, the correlation between W and the prediction error was statistically small (Pearson's |r| < 0.2), and W was not a significant contributor to changing prediction errors (p>0.05). As these results do not support the concerns that the Dual Gaussian Distribution Model has an issue when used with small targets, the development of applications and refined models is encouraged to continue.
While augmented reality (AR) headsets provide entirely new ways of seeing and interacting with data, traditional computing devices can play a symbiotic role when used in conjunction with AR as a hybrid user interface. A promising use case for this setup is situated analytics. AR can provide embedded views that are integrated with their physical referents, and a separate device such as a tablet can provide a familiar situated overview of the entire dataset being examined. While prior work has explored similar setups, we sought to understand how people perceive and make use of visualizations presented on both embedded visualizations (in AR) and situated visualizations (on a tablet) to achieve their own goals. To this end, we conducted an exploratory study using a scenario and task familiar to most: adjusting light levels in a smart home based on personal preference and energy usage. In a prototype that simulates AR in virtual reality, embedded visualizations are positioned next to lights distributed across an apartment, and situated visualizations are provided on a handheld tablet. We observed and interviewed 19 participants using the prototype. Participants were easily able to perform the task, though the extent the visualizations were used during the task varied, with some making decisions based on the data and others only on their own preferences. Our findings also suggest the two distinct roles that situated and embedded visualizations can have, and how this clear separation might improve user satisfaction and minimize attention-switching overheads in this hybrid user interface setup. We conclude by discussing the importance of considering the user's needs, goals, and the physical environment for designing and evaluating effective situated analytics applications.
