Cross-disciplinary teams increasingly work with high-dimensional scientific datasets, yet fragmented toolchains and limited support for shared exploration hinder collaboration. Prior immersive visualization & analytics research has emphasized individual interaction, leaving open how multi-user collaboration can be supported at scale. To fill this critical gap, we conduct semi-structured interviews with 20 domain experts from diverse academic, government, and industry backgrounds. Using deductive–inductive hybrid thematic analysis, we identify four collaboration-focused themes: workflow challenges, adoption perceptions, prospective features, and anticipated usability and ethical risks. These findings show how current ecosystems disrupt coordination and shared understanding, while highlighting opportunities for effective multi-user engagement. Our study contributes empirical insights into collaboration practices for high-dimensional scientific data visualization & analysis, offering design implications to enhance coordination, mutual awareness, and equitable participation in next-generation collaborative immersive platforms. These contributions point toward future environments enabling distributed, cross-device teamwork on high-dimensional scientific data.

With advances in consumer-grade virtual reality (VR) devices, VR training gains unprecedented attention in research and industries. Although the nature of VR training encourages trainees to actively learn through exploring and gathering information in a simulated virtual environment, designing effective virtual training environments is non-trivial. We propose an adaptive approach that guides trainees to develop psychomotor skills in a simulated virtual environment. As a showcase, we demonstrate our novel approach for restaurant service using a game-based VR application. By incorporating the trainee’s performance and learning progress into optimization objectives, our approach uses mixed integer programming (MIP) to generate VR training sessions iteratively. Through collecting the trainee’s performance in VR training, our approach adapts the VR training sessions by considering the trainee’s strengths and weaknesses, guiding the trainee to improve over training sessions. We validated our approach through two experimental studies. In the first study, we compared our approach with a random training task assignment approach and a performance-only MIP approach through performing simulated restaurant service training. In the second study, we compared our approach with the random assignment approach by evaluating trainees’ skill developments in restaurant services. The results show that our skill-driven adaptive training approach outperforms the random assignment approach.

In the ever-evolving discipline of high-dimensional scientific data, collaborative immersive analytics (CIA) offers a promising frontier for domain experts in complex data visualization and interpretation. This research presents a comprehensive framework for conducting usability studies on the extended reality (XR) interface of ParaView, an open-source CIA system. By employing established human-computer interaction (HCI) principles, including Jakob Nielsen's Usability Heuristics, Cognitive Load Theory, NASA Task Load Index, System Usability Scale, Affordance Theory, and Gulf of Execution and Evaluation, this study aims to identify underlying usability issues and provide guidelines for enhancing user experience in scientific domains. Our findings reveal significant usability challenges in the ParaView XR interface that impede effective teamwork and collaboration. For instance, the lack of synchronous collaboration, limited communication methods, and the absence of role-based data access are critical areas that need attention. Additionally, inadequate error handling, insufficient feedback mechanisms, and limited support resources during application use require extensive improvement to fully utilize the system's potential. Our study suggests potential improvements to overcome the existing usability barriers of the collaborative immersive system.

Unified systems for multi-sensor devices, particularly eye-tracking in Virtual Reality (VR), are intricate and often require the listening and streaming of multichannel data. In this project, we propose a visual analysis framework for replicating a participant's viewing involvement by interpreting head movements as rotations and point-of-gaze (POG) as on-screen indicators. Our solution suggests an additional layer of system for near-real-time for processing and analyzing this multi-device data to connect with the data and enable both near-real-time or subsequent offline viewing of the entire VR eye-tracking session. Moreover, our method provides a no-batteries-need solution to create traditional eye-tracking visualization techniques. Finally, we apply three prior education technology analysis metrics: higher density gaze for students, shorter fixation time, and less fixation duration variance for students to determine expertise levels in this system. We systematically establish a ubiquitous, multi-device, eye-tracking solution to incorporate this approach. We evaluate the effectiveness of our system through a user study, using both expertise and non- expertise levels, and selectively surveying to ascertain the quality of the replicated experience and we test the system by running a real-world user study with sixty four different participants. We demonstrate the application's significance and potential to integrate prior analysis metrics using the collected data which this data collection and analysis have been approved by IRB.

Over the past 30 years, Many research organizations established large-scale inunersive visualization laboratories. Some have been very successful, but others are no longer around. With the recent availability of COTS virtual reality devices, the need for larger immersive visualization laboratories comes into question. Some immersive visualization laboratories have moved away from large scale facilities due to the cost of maintaining these more elaborate systems. But can immersive visualization requirements be satisfied solely using COTS VR devices used by researchers in their own office?

Deep learning techniques have emerged as valuable tools for video analysis and motion detection. Recent advancements in this field have shown promising results. Our objective is to leverage these video understanding techniques to aid teachers in evaluating their teaching quality and enhancing their effectiveness in the classroom. However, existing research on student behavior analysis primarily focuses on recognizing actions pertaining to classroom management, neglecting the identification of “learning behaviors” exhibited by students. To address this limitation, we introduce a novel video dataset specifically designed to capture the nuances of “learning behaviors” displayed by primary-grade students in the mathematics classroom, along with a dedicated student localization dataset focused on detecting the location of individuals. Our approach introduces a framework that utilizes deep learning-based object detection and action recognition techniques trained on our curated datasets to analyze and comprehend student learning behaviors in the classroom. To assess the performance of our approach, we conduct separate tests on our object detection and action recognition models. Sub-sequently, our framework is applied to a collection of recorded 360-degree classroom videos, enabling a thorough evaluation of its capabilities.

Over the past 30 years, many research organizations established large-scale immersive visualization laboratories. Some have been very successful, but others are no longer around. With the recent availability of COTS virtual reality devices, the need for larger immersive visualization laboratories comes into question. Some immersive visualization laboratories have moved away from large scale facilities due to the cost of maintaining these more elaborate systems. But can immersive visualization requirements be satisfied solely using COTS VR devices used by researchers in their own office?

IEEE VR 2023 is the 30th conference in the series, and 2023 is the 24th anniversary of the publication of Professor Frederick P. Brooks, Jr.'s paper "What's Real about Virtual Reality?".In light of those two things, and acknowledging the contribution the paper has made, We propose a panel of people who engage in activities using VR/AR outside of laboratory settings; people who confront, on a daily basis, what's real (and what's not) about virtual reality.Our four panelists will each briefly describe their application and use of VR and then address the pain point--the squeaky wheel--that impedes wider adoption in their application area.The presentations will be followed by Q&A both amongst the panelists and from the audience.Audience questions will be pre-screened.The panel will be recorded for archival purposes.

We present ongoing work to enhance ParaView with new immersive visualization capabilities and demonstrate its use in the development of our scientific workflow in support of visual analytics in an immersive environment to advance measurement science. Two case-studies of complex meteorology and radio-frequency (RF) data as part of the knowledge discovery process reveal the benefits of interactively exploring the three-dimensional data with immersive technologies such as the CAVE and head mounted display virtual reality. By adding support of immersive technology into Para View, researchers can more naturally investigate the data of three-dimensional simulations. These efforts to provide tools for immersive visualization are guided by the need to advance measurement science, standards, and technology. To make these tools more broadly accessible and impact-ful, we promote the use of software standards related to visualization and immersive systems.

A barrier to developing novel AI for complex reasoning is the lack of appropriate wargaming platforms for training and evaluating AIs in a multiplayer setting combining collaborative and adversarial reasoning under uncertainty with game theory and deception. An appropriate platform has several key requirements including flexible scenario design and exploration, extensibility across all five elements of Multi-Domain Operations (MDO), and capability for human-human and human-AI collaborative reasoning and data collection, to aid development of AI reasoning and the warrior-machinelike interface. Here, we describe the ARL Battlespace testbed which fulfills the above requirements for AI development, training and evaluation. ARL Battlespace is offered as an open source software platform (https://github.com/USArmyResearchLab/ARL_Battlespace). We present several example scenarios implemented in ARL Battlespace that illustrate different kinds of complex reasoning for AI development. We focus on 'gap' scenarios that simulate bridgehead and crossing tactics, and we highlight how they address key platform requirements including coordinated MDO actions, game theory and deception. We describe the process of reward shaping for these scenarios that will incentivize an agent to perform command and control (C2) tasks informed by human commanders' courses of action, as well as the key challenges that arise. The intuition presented will enable AI researchers to develop agents that will provide optimal policies for complex scenarios.

In an increasingly complex military operating environment, next generation wargaming platforms can reduce risk, decrease operating costs, and improve overall outcomes. Novel Artificial Intelligence (AI) enabled wargaming approaches, based on software platforms with multimodal interaction and visualization capacity, are essential to provide the decision-making flexibility and adaptability required to meet current and emerging realities of warfighting. We highlight three areas of development for future warfighter-machine interfaces: AI-directed decisional guidance, computationally informed decision-making, and realistic representations of decision spaces. Progress in these areas will enable development of effective human-AI collaborative decision-making, to meet the increasing scale and complexity of today's battlespace.

Recent years have witnessed an increment in the number of network components communicating through many network scenarios such as multi-layer and multi-domain, and it may result in a negative impact on resource utilization. An urgent requirement arises for routing the packets most efficiently and economically in large multi-domain networks. In tackling this complicated area, we consider the Inter-Domain Path Computation problem under Node-defined Domain Uniqueness Constraint (IDPC-NDU), which intends to find the minimum routing cost path between two nodes that traverses every domain at most once. Owing to the NP-Hard property of the IDPC-NDU, applying metaheuristic algorithms to solve this problem usually proves more efficient. In like manner, this paper proposes a Two-level Genetic Algorithm (PGA), where the first level determines the order of the visited domains, and the second level finds the shortest path between the two given nodes. Furthermore, to facilitate the finding process, a method to minimize the search space and a new chromosome encoding that would reduce the chromosome length to the number of domains are integrated into this proposed algorithm. To evaluate the efficiency of the proposal, experiments on various instances were conducted. The results demonstrated that PGA outperforms other algorithms and gives results no more than twice the optimal values.

We present an investigation using mixed reality technology to visualize decision-making dynamics for a Friendly vs Hostile wargame in a Multi-Domain Operation environment. The requirement of penetrate and dis-integrate phases under Multi-Domain Operations aligns well with the advantages of Artificial Intelligence/Machine Learning because of 1) very short planning timeframe for decision-making, 2) simultaneous planning requirement for multiple operations, and 3) interdependence of operations. In our decision dynamics research, we propose to advance the art/science for wargaming by leveraging brain science to extend the use of Artificial Intelligence/Machine Learning algorithms and the use of mixed reality technology to visualize complex battlespace scenarios requiring a better understand of the dynamics in a complex decision making process.

Despite foreseeing tremendous speedups over conventional deep neural networks, the performance advantage of binarized neural networks (BNNs) has merely been showcased on general-purpose processors such as CPUs and GPUs. In fact, due to being unable to leverage bit-level-parallelism with a word-based architecture, GPUs have been criticized for extremely low utilization (1 percent) when executing BNNs. Consequently, the latest tensorcores in NVIDIA Turing GPUs start to experimentally support bit computation. In this article, we look into this brand new bit computation capability and characterize its unique features. We show that the stride of memory access can significantly affect performance delivery and a data-format co-design is highly desired to support the tensorcores for achieving superior performance than existing software solutions without tensorcores. We realize the tensorcore-accelerated BNN design, particularly the major functions for fully-connect and convolution layers - bit matrix multiplication and bit convolution. Evaluations on two NVIDIA Turing GPUs show that, with ResNet-18, our BTC-BNN design can process ImageNet at a rate of 5.6K images per second, 77 percent faster than state-of-the-art. Our BNN approach is released on https://github.com/pnnl/TCBNN.

Two video analysis approaches (pose estimation and manual annotation) were applied to video recordings of two-person teams performing a mission planning task in a shared augmented reality (AR) environment. The analysis approaches calculated the distance relations between team members and annotated observed behaviors during the collaborative task. The 2D pose estimation algorithm lacked scene depth processing; therefore, we found some inconsistencies with the manual annotation. Although integration of the two analysis approaches was not possible, each approach by itself produced several insights on team behavior. The manual annotation analysis found four common team behaviors as well as behavior variations unique to particular teams and temporal situations. Comparing a behavior-based time on task percentage indicated behavior-type connections and some possible exclusions. The pose estimation analysis found the majority of the teams moved around the 3D scene at a similar distance apart on average with similar variation in fluctuation around a common distance range between team members. Outlying team behavior was detected by both analysis approaches and included: periods of very low distance relations, infrequent but very high distance relation spikes, significant task time spent adjusting the HoloLens device during wearing, and exceptionally long task time with gaps in pose estimation data processing.

Annotation are essential in using machine learning for object detection in images. However, good detection will require high quality annotations for supervised learning-based techniques. It is also known that manual annotation by human experts are laborious, tedious and error-prone. This problem is more so challenging in physical sciences images, such as liquid droplets or crystals. In this paper, we suggest a minimal effort approach to train YOLOv3 models using a relatively smaller set of training data for liquid droplet images. The proposed approach uses only minimal amount of annotations or annotated images, and naively build training data set using YOLOv3 models. The investigation compares two approach variants and discuss the results as an exploratory work in finding suitable semi-automatic image annotation techniques for object detection in liquid droplet images.

Future Multi Domain Operation (MDO) wargaming will rely on Artificial Intelligence/Machine Learning (AI/ML) algorithms to aid and accelerate complex Command and Control decision-making. This requires an interdisciplinary effort to develop new algorithms that can operate in dynamic environments with changing rules, uncertainty, individual biases, changing cognitive states, as well as the capability to rapidly mitigate unexpected hostile capabilities and exploit friendly technological capabilities. Building on recent advancements in AI/ML algorithms, we believe that new algorithms for learning, reasoning under uncertainty, game theory with three or more players, and interpretable AI can be developed to aid in complex MDO decision-making. To achieve these goals, we developed a new flexible MDO warfighter machine interface game, Battlespace, to investigate and understand how human decision-making principles can be leveraged by and synergized with AI. We conducted several experiments with human vs. random players operating in a fixed environment with fixed rules, where the overall goal of the human players was to collaborate to either capture the opponents’ flags or eliminate all of their units. Then, we analyzed the evolution of the games and identified key features that characterized the human players’ strategies and their overall goal. We then followed a Bayesian approach to model the human strategies and developed heuristic strategies for a simple AI agent. Preliminary analysis revealed that following the human agents’ strategy in the capture the flag games produced the greatest winning percentage and may be useful for gauging the value of intermediate game states for developing the coordinated action planning of reinforcement learning algorithms.