This study addresses a critical challenge in Virtual Reality (VR): the detection of visually induced motion sickness (VIMS), a factor that significantly limits prolonged VR use. By integrating head, eye, and mouth movement data, we propose a novel multi-modality-based approach for early detection of VIMS. Our Attention-VR model is designed to dynamically assign weights to the more important modalities among the three (head, eye, and mouth) for identifying the early signs of VIMS. This paper reports on two studies. In Study 1, we developed our predictive models by analyzing multi-modal data from 38 participants experiencing a customized VR roller coaster environment. In Study 2, we validated these models using data from commercial VR games. By integrating diverse physical movement signals, Attention-VR significantly improves the precision and interpretability of VIMS predictions, providing new insights into the relative importance of each modality. We discuss the limitations of the current approach and suggest directions for future research.

We propose a novel multi-modalities based approach that predicts visually induced motion sickness (VIMS) by integrating head, eye, and mouth movement data. Our Attention-VR model dynamically assigns higher weights to the most informative of the three modalities for identifying early signs of VIMS. Multi-modal data from 38 participants in a customized VR rollercoaster environment were collected to develop our predictive model. By combining diverse physical movement signals, Attention-VR significantly improves prediction accuracy and interpretability, clarifying the relative importance of each modality.

Advancements in wearable technologies have made the use of physiological signals, such as Electrodermal Activity (EDA) and Heart Rate Variability (HRV), more prevalent for detecting changes in the autonomic nervous system within virtual reality (VR). However, the challenge lies in utilizing these signals to objectively detect presence in VR, which typically relies on self-reports that can be inherently biased. This paper addresses this issue and presents a study (N=26) that investigates the effect that different levels of presence has on physiological responses in VR. A neutral VR environment was created that incorporated three levels of presence (high, medium and low) that were invoked by tuning different parameters. Participants wore a wrist-worn wearable device that captured their physiological signals whilst they experienced each of these environments. Results indicated that tonic and phasic components of the EDA signal were significant in differentiating between the levels. Two novel features, constructed using both the phasic and tonic components of EDA, successfully differentiated between presence levels. Analysis of the HRV data illustrated a significant difference between the low and medium levels using the ratio between low frequency to high frequency.

AbstractIn Virtual Reality (VR), a higher level of presence positively influences the experience and engagement of a user. There are several parameters that are responsible for generating different levels of presence in VR, including but not limited to, graphical fidelity, multi-sensory stimuli, and embodiment. However, standard methods of measuring presence, including self-reported questionnaires, are biased. This research focuses on developing a robust model, via machine learning, to detect different levels of presence in VR using multimodal neurological and physiological signals, including electroencephalography and electrodermal activity. An experiment has been undertaken whereby participants (N = 22) were each exposed to three different levels of presence (high, medium, and low) in a random order in VR. Four parameters within each level, including graphics fidelity, audio cues, latency, and embodiment with haptic feedback, were systematically manipulated to differentiate the levels. A number of multi-class classifiers were evaluated within a three-class classification problem, using a One-vs-Rest approach, including Support Vector Machine, k-Nearest Neighbour, Extra Gradient Boosting, Random Forest, Logistic Regression, and Multiple Layer Perceptron. Results demonstrated that the Multiple Layer Perceptron model obtained the highest macro average accuracy of $$93\pm 0.03\%$$ 93 ± 0.03 % . Posthoc analysis revealed that relative band power, which is expressed as the ratio of power in a specific frequency band to the total baseline power, in both the frontal and parietal regions, including beta over theta and alpha ratio, and differential entropy were most significant in detecting different levels of presence.

Recently, we saw a trend toward using physiological signals in interactive systems. These signals, offering deep insights into users' internal states and health, herald a new era for HCI. However, as this is an interdisciplinary approach, many challenges arise for HCI researchers, such as merging diverse disciplines, from understanding physiological functions to design expertise. Also, isolated research endeavors limit the scope and reach of findings. This workshop aims to bridge these gaps, fostering cross-disciplinary discussions on usability, open science, and ethics tied to physiological data in HCI. In this workshop, we will discuss best practices for embedding physiological signals in interactive systems. Through collective efforts, we seek to craft a guiding document for best practices in physiological HCI research, ensuring that it remains grounded in shared principles and methodologies as the field advances.

Researchers have used machine learning approaches to identify motion sickness in VR experience. These approaches would certainly benefit from an accurately labeled, real-world, diverse dataset that enables the development of generalizable ML models. We introduce ‘VR.net’, a dataset comprising 165-hour gameplay videos from 100 real-world games spanning ten diverse genres, evaluated by 500 participants. VR.net accurately assigns 24 motion sickness-related labels for each video frame, such as camera/object movement, depth of field, and motion flow. Building such a dataset is challenging since manual labeling would require an infeasible amount of time. Instead, we implement a tool to automatically and precisely extract ground truth data from 3D engines' rendering pipelines without accessing VR games' source code. We illustrate the utility of VR.net through several applications, such as risk factor detection and sickness level prediction. We believe that the scale, accuracy, and diversity of VR.net can offer unparalleled opportunities for VR motion sickness research and beyond.We also provide access to our data collection tool, enabling researchers to contribute to the expansion of VR.net.

Human-Computer Interaction, Department of Computer Science, University of Trier, Trier, Germany, Center for Advanced Computer Studies, University of Louisiana at Lafayette, Lafayette, LA, United States, Department of Software Engineering and Game Design and Development, Kennesaw State University, Marietta, GA, United States, School of Information Technology and Electrical Engineering, University of Queensland, Brisbane, QLD, Australia

We developed the Motion-Simulation Platform, a platform running within a game engine that is able to extract both RGB imagery and the corresponding intrinsic motion data (i.e., motion field). This is useful for motion-related computer vision tasks where large amounts of intrinsic motion data are required to train a model. We describe the implementation and design details of the Motion-Simulation Platform. The platform is extendable, such that any scene developed within the game engine is able to take advantage of the motion data extraction tools. We also provide both user and AI-bot controlled navigation, enabling user-driven input and mass automation of motion data collection.

The push towards a Metaverse is growing, with companies such as Meta developing their own interpretation of what it should look like. The Metaverse at its conceptual core promises to remove boundaries and borders, becoming a decentralised entity for everyone to use - forming a digital virtual layer over our own “real” world. However, creation of a Metaverse or “new world” presents the opportunity to create one which is inclusive and accessible to all. This challenge is explored and discussed in this workshop, with an aim of understanding how to create a Metaverse which is open and inclusive to people with physical and intellectual disabilities, and how interactions can be designed in a way to minimise disadvantage. The key outcomes of this workshop outline new opportunities for improving accessibility in the Metaverse, methodologies for designing and evaluating accessibility, and key considerations for designing accessible Metaverse environments and interactions.

This paper presents a novel solution for estimating simulator sickness in HMDs using machine learning and 3D motion data, informed by user-labeled simulator sickness data and user analysis. We conducted a novel VR user study, which decomposed motion data and used an instant dial-based sickness scoring mechanism. We were able to emulate typical VR usage and collect user simulator sickness scores. Our user analysis shows that translation and rotation differently impact user simulator sickness in HMDs. In addition, users' demographic information and self-assessed simulator sickness susceptibility data are collected and show some indication of potential simulator sickness. Guided by the findings from the user study, we developed a novel deep learning-based solution to better estimate simulator sickness with decomposed 3D motion features and user profile information. The model was trained and tested using the 3D motion dataset with user-labeled simulator sickness and profiles collected from the user study. The results show higher estimation accuracy when using the 3D motion data compared with methods based on optical flow extracted from the recorded video, as well as improved accuracy when decomposing the motion data and incorporating user profile information.

Training systems are used in many industries, ranging from surgery to space missions to rehabilitation. Virtual Reality (VR) is a technology that has been incorporated as an effective tool in such training systems to simulate the environment, especially in situations where the training can’t take place in the actual environment. For a training environment and task to be effective, it must sufficiently challenge the trainee. One parameter that can be used to measure this is cognitive load (CL), which is defined as the amount of working memory used while performing a learning task. This parameter needs to be sufficiently high to maximize learning but not too high as to overload the trainee. However, the challenge is to detect this state using objective physiological measures, which can be collected during the entire task. This paper presents a study to classify CL using a combination of Electroencephalogram (EEG) and Electrodermal Activity (EDA) signals during a procedural VR training task. Thirty participants undertook a study where they built a designated model within a given time over multiple levels that were constructed to induce low to high CL. Features generated from the data were subject to feature selection (FS), which was undertaken using the Mutual Information (MI) technique. Binary classification models were developed using Support Vector Machines (SVM), Random Forest (RF), k-Nearest Neighbors (kNN), Extreme Gradient Boosting (Xgboost) and Multi-Layer Perceptrons (MLP). Results illustrated that the Xgboost classifier performed the best with an F1-score of $0.831 \pm 0.030$ and accuracy of $0.805 \pm 0.033.$ SHAP analysis of the features illustrated greater contributions from the frontal and occipital regions of the brain and frequency domain features from tonic skin conductance.

Recent improvements in affordable virtual and augmented reality technology enable broader adoption in everyday contexts such as homes, schools, and health care settings. This Frontiers Research Topic presents articles that give insight into the everyday use of VR/AR or that provide techniques for interface improvements. Three articles report field studies of everyday applications. Studying VR and AR in everyday contexts provides practical insight into real-world deployments, with a level of ecological validity beyond artificial laboratory environments. Two articles discuss input methods that support improved interaction with minimal added cost (sketching, whole-hand interaction). Two Opinion and Perspective articles discuss considerations for older adults and for molecular science students.

We used VR as a speculative design tool to explore how Deaf people might interact with a sign language personal assistant through a Wizard of Oz pilot study. This allowed us to elicit concrete design requirements from potential users of the proposed personal assistant, and shows the potential of VR as a requirements elicitation tool.

Asymmetric systems enable remote collaboration using multiple different devices. In this paper, we have designed an asymmetric system that enables collaboration between Augmented Reality (AR) and Laptop users. Remote collaboration systems often lack mutual emotional understanding between the collaborators. Researchers in virtual reality have used physiological data sharing as a way to increase social awareness between collaborators. No such effort was made in AR and in asymmetric domains. A key contribution of this work is that we have enabled physiological (heart rate) feedback sharing between the collaborators to generate more awareness about one another in an asymmetric system. We executed a user study with 20 participants where they performed a collaborative and a competitive task. Our results indicated that providing heart rate feedback is beneficial for AR devices. We noticed the experimental tasks had an effect on galvanic skin response although no such effect was found on heart rate signal.

Virtual Reality (VR) interfaces provide an immersive medium to interact with the digital world. Most VR interfaces require physical interactions using handheld controllers, but there are other alternative interaction methods that can support different use cases and users. Interaction methods in VR are primarily evaluated based on their usability, however, their differences in neurological and physiological effects remains less investigated. In this paper—along with other traditional qualitative matrices such as presence, affect, and system usability—we explore the neurophysiological effects—brain signals and electrodermal activity—of using an alternative facial expression interaction method to interact with VR interfaces. This form of interaction was also compared with traditional handheld controllers. Three different environments, with different experiences to interact with were used—happy (butterfly catching), neutral (object picking), and scary (zombie shooting). Overall, we noticed an effect of interaction methods on the gamma activities in the brain and on skin conductance. For some aspects of presence, facial expression outperformed controllers but controllers were found to be better than facial expressions in terms of usability.

Virtual Reality (VR) could give users a more immersive experience than other non-immersive mediums. In this study, we explored differences in emotional and physiological effects between videos and VR using two different sets of contents to evoke happy and sad emotions. In this within-subjects controlled experiment we collected real-time heart rate, positive and negative affect schedule (PANAS), and Self-Assessment Manikin (SAM) to measure physiological and emotional effects. Our results showed that VR triggers stronger emotions and higher heart rate than videos.

Hyperscanning is an emerging method for measuring two or more brains simultaneously. This method allows researchers to simultaneously record neural activity from two or more people. While this method has been extensively implemented over the last five years in the real-world to study inter-brain synchrony, there is little work that has been undertaken in the use of hyperscanning in virtual environments. Preliminary research in the area demonstrates that inter-brain synchrony in virtual environments can be achieved in a manner similar to that seen in the real world. The study described in this paper proposes to further research in the area by studying how non-verbal communication cues in social interactions in virtual environments can affect inter-brain synchrony. In particular, we concentrate on the role eye gaze plays in inter-brain synchrony. The aim of this research is to explore how eye gaze affects inter-brain synchrony between users in a collaborative virtual environment.

Social Virtual Reality (VR) platforms enable multiple users to be present together in the same virtual environment (VE) and interact with each other in this space.These platforms are used in different application areas including teaching and learning, conferences, and meetings.To improve the engagement, safety, and overall positive experience in such platforms it is important to understand the effect they have on users' emotional states and self-awareness while being in the VE.In this work, we present a focus group study where we discussed users' opinions about social VR and we ran the focus group in a social VR platform created in Hubs by Mozilla.Our primary goal was to investigate users' emotional states and self-awareness while using this platform.We measured these effects using positive and negative affect schedule (PANAS) and Self-Assessment Questionnaire (SAQ).The experiment involved 12 adult participants who were volunteers from around the world with previous experience of VR.