Cybersickness is one of the problems that users often encounter during virtual reality (VR) experiences. State-of-the-art machine learning (ML)/deep learning (DL) methods for predicting cybersickness often require massive amounts of high-quality training data and lengthy training times. These methods also suffer from limited transferability across different VR environments, performing well only in familiar environments and faltering in newones. To address these limitations, we propose an innovative cybersickness prediction and mitigation method that leverages knowledge from pre-trained large foundation models, namely a time-series generative pre-trained Transformer (TimeGPT) and Chronos, both trained on billions of diverse data points. To adapt these foundation models for the cybersickness prediction/forecasting task, we utilize two learning strategies: zero-shot learning (ZSL) (without fine-tuning TimeGPT and Chronos models’ weights and parameters) and few-shot learning (FSL) (with fine-tuning of TimeGPT and Chronos weights and parameters). We validate our approach on two open-source VR datasets: Simulations 2021 and APAL Head 2019, across multiple prediction horizons, multimodal data-fusion configurations, and demographic-based personalization via age- and gender-based grouping. Our results show that the proposed FSL-based fine-tuned TimeGPT and Chronos models significantly outperform traditional DL models (training from scratch) for both datasets. For instance, under the full multimodal fusion configuration, the fine-tuned TimeGPT model achievesRMSEvalues of 0.26 for the Simulations 2021 dataset and 2.69 for the APAL Head 2019 dataset, which are 1.31× and 1.39× lower than the traditional DL-based Transformer model under the same modalities configuration. Additionally, it reduces training time by 81% and 79% compared to the Transformer model, highlighting its efficiency and strong potential for practical deployment in cybersickness prediction. Finally, to demonstrate the effectiveness of our proposed method, we deploy the traditional DL-based Transformer model and FSL-based fine-tuned TimeGPT and Chronos models on a consumer-grade VR headset (HTC Vive Pro) and integrate them with a dynamic field-of-view (DFOV) mitigation method. Using streaming VR sensor data during gameplay, the system forecasts the onset of cybersickness severity (e.g., FMS score) at multiple prediction horizons and adaptively adjusts DFOV as predicted severity increases. A user study further confirms that shorter-horizon predictions provide a more responsive control signal for DFOV adaptation compared to a no-prediction/no-mitigation baseline and lead to lower perceived cybersickness during VR experiences.

Games and simulators can be a valuable platform to execute complex multi-agent, multiplayer, imperfect information scenarios with significant parallels to military applications: multiple participants manage resources and make decisions that command assets to secure specific areas of a map or neutralize opposing forces. These characteristics have attracted the artificial intelligence (AI) community by supporting development of algorithms with complex benchmarks and the capability to rapidly iterate over new ideas. The success of AI algorithms in real-time strategy games such as StarCraft II has also attracted the attention of the military research community aiming to explore similar techniques in military counterpart scenarios. Aiming to bridge the connection between games and military applications, this work discusses past and current efforts on how games and simulators, together with the AI algorithms, have been adapted to simulate certain aspects of military missions and how they might impact the future battlefield. This paper also investigates how advances in virtual reality and visual augmentation systems open new possibilities in human interfaces with gaming platforms and their military parallels.

The modern battlefield environment presents commanders and analysts with an overwhelming amount of information.Only portions of this information are useful at any given moment, often requiring human intervention to parse out what is meaningful and what is not.In an environment where every second counts, methods for accelerating the presentation of only useful information to decision makers is critical.Machine learning is widely used to predict patterns and outcomes in a variety of applications where data structures are complex and high-dimensional.Supervised learning is a traditional machine learning method wherein the algorithm is trained on a large set of data before performing predictions.On the other hand, online learning is a machine learning technique wherein the algorithm learns incrementally or whenever new data and feedback are available.This work seeks to develop a proof of concept for predicting the utility value of incoming sensor data for a user via an online learning method.It also investigates changes in model performance with respect to hyperparameter configuration and the conditions that cause a user to accept that piece of information on each trial presentation via simulated experiments.15.

Enabling leaders with the ability to make decisive actions in high operational tempo environments is key to achieving decision-superiority. Under stressful battlefield conditions with little to no time for communication, it is critical to acquire relevant tactical information quickly to inform decision-making. A potential augmentation to tactical information systems is access to real-time analytics on a unit's operating status and emergent behaviors inferred from soldier-worn or embedded sensors on their kit. Automatic human activity recognition (HAR) has been greatly achievable in recent years thanks to advancements in algorithms and ubiquitous low-cost, yet powerful processors, hardware and sensors. In this paper, we present weapon-born sensor measurement acquisition, processing, and HAR approaches to demonstrate Soldier state estimation in a target acquisition and tracking experiment. The Soldier states that were classified include whether the Soldier is resting, tracking a target, transitioning between potential targets, or firing a shot at the target. We implemented Multivariate Time Series Classification (TSC) using the SKTime toolkit to perform this task and discuss the performance from various classification methods. We also discuss a framework for efficient transference of this information to other tactical information systems on the network.

Organized by experts from across academia, industry, the Federal Labs, and SPIE, this meeting will highlight emerging capabilities in immersive technologies and degraded visual environments as critical enablers to future multi-domain operations.

Efforts are underway across the defense and commercial industries to develop cross-reality (XR), multi-user operation centers in which human users can perform their work while aided by intelligent systems. At their core is the objective to accelerate decision-making and improve efficiency and accuracy. However, presenting data to users in an XR, multi-dimensional environment results in a dramatic increase in extraneous information density. Intelligent systems offer a potential mechanism for mitigating information overload while ensuring that critical and anomalous data is brought to the attention of the human users in an immersive interface. This paper describes such a prototype system that combines real and synthetic motion sensors which, upon detection of an event, send a captured image for processing by a YOLO cluster. Finally, we describe how a future system can integrate a decision-making component for evaluation of the resulting metadata to determine whether to inject the results into an XR environment for presentation to human users.

Deployment of Internet of Things (IoT) devices and Data Fusion techniques have gained popularity in public and government domains. This usually requires capturing and consolidating data from multiple sources. As datasets do not necessarily originate from identical sensors, fused data typically results in a complex data problem. Because military is investigating how heterogeneous IoT devices can aid processes and tasks, we investigate a multi-sensor approach. Moreover, we propose a signal to image encoding approach to transform information (signal) to integrate (fuse) data from IoT wearable devices to an image which is invertible and easier to visualize supporting decision making. Furthermore, we investigate the challenge of enabling an intelligent identification and detection operation and demonstrate the feasibility of the proposed Deep Learning and Anomaly Detection models that can support future application that utilizes hand gesture data from wearable devices.

Immersive technologies, such as virtual and augmented reality, initially failed to live up to expectations, but have improved greatly, with many new head-worn displays and associated applications being released over the past few years. Unfortunately, 'cybersickness' remains as a common user problem that must be overcome if mass adoption is to be realized. This article evaluates the state of research on this problem, identifies challenges that must be addressed, and formulates an updated cybersickness research and development (R&D) agenda. The new agenda recommends prioritizing creation of powerful, lightweight, and untethered head-worn displays, reduction of visual latencies, standardization of symptom and aftereffect measurement, development of improved countermeasures, and improved understanding of the magnitude of the problem and its implications for job performance. Some of these priorities are unresolved problems from the original agenda which should get increased attention now that immersive technologies are proliferating widely. If the resulting R&D agenda is carefully executed, it should render cybersickness a challenge of the past and accelerate mass adoption of immersive technologies to enhance training, performance, and recreation.

One of the most significant challenges for the emerging operational environment addressed by Multi-Domain Operations (MDO) is the exchange of information between personnel in operating environments. Making information available for leveraging at the appropriate echelon is essential for convergence, a key tenet of MDO. Emergent cross-reality (XR) technologies are poised to have a significant impact on the convergence of the information environment. These powerful technologies present an opportunity to not only enhance the situational awareness of individuals at the "local" tactical edge and the decision-maker at the "global" mission command (C2), but to intensely and intricately bridge the information exchanged across all echelons. Complimentarily, the increasing use of autonomy in MDO, from autonomous robotic agents in the field to decision-making assistance for C2 operations, also holds great promise for human-autonomy teaming to improve performance at all echelon levels. Traditional research examines, at most, a small subset of these problems. Here, we envision a system that sees human-robot teams operating at the local edge communicating with human-autonomy teams at the global operations level. Both teams use a mixed reality (MR) system for visualization and interaction with a common operating picture (COP) to enhance situational awareness, sensing, and communication {but with highly different purposes and considerations. By creating a system that bridges across echelons, we are able to examine these considerations to determine their impact on information shared bi-directionally, between the global (C2) and local (tactical) levels, in order to understand and improve autonomous agents teamed with humans at both levels. We present a prototype system that includes an autonomous robot operating with a human teammate sharing sensory data and action plans with, and receiving commands and intelligence information from, a tactical operations team commanding from a remote location. We examine the challenges and considerations in creating such a system, and present initial findings.

Understanding quantified uncertainty through efficient visualization techniques is becoming increasingly important for the successful teaming of human and intelligent agents across many domains. For humans to make effective, well-informed decisions, visualizations must maximize the amount of critical information communicated in a way that complexity is not prohibitive of fast and accurate understanding. In this review, we first identify common approaches to uncertainty in multiple domains, including traditional graphical methods in the 1D and 2D Data Dimensions, and survey their techniques. We then analyze current challenges in the uncertainty visualization space pertaining to information complexity, presentation, added dimensionality, visual dominance, and multidisciplinary needs. Finally, we review the growing number of applications and the current state of uncertainty visualization, addressing the benefits from knowing uncertainty in each example and identifying the windows of opportunity in the future context of multi-domain use cases.

Decision-making is defined as a process resulting in the selection of a course of action from a number of alternatives based on variables that represent key considerations to the task. This is a complex process where the goal is to generate the "best" course of action given the data and knowledge obtained. As the use of intelligent systems increases, so too does the amount of data to be considered by human analysts and commanders. As the military looks toward integration of intelligent system like smart devices or internet of things, the devices and the data from these devices are important for decision making in highly dynamic situations. Of critical importance is the uncertainty of information associated with the data produced from such systems. Any uncertainty must be captured and communicated to aid the decision-making process. Our work focuses on how this process can be investigated to understand and analyze the impact of uncertainty for decision-making in multi-domain operational environments. We conducted user studies and present our results to discuss the presentation of uncertainty within the decision-making cycle for our tasks.

The resurgence of AI in the recent decade dramatically changes the design of modern sensor data fusion systems, leading to new challenges, opportunities, and research directions. One of these challenges is the management of uncertainty. This paper develops a framework to reason about sources of uncertainty, develops representations of uncertainty, and investigates uncertainty mitigation strategies in modern intelligent data processing systems. Insights are developed into workflow composition that maximizes efficacy at accomplishing mission goals despite the sources of uncertainty, while leveraging a collaboration of humans, algorithms, and machine learning components.

Collective intelligence is generally defined as the emergence and evolution of intelligence derived from the collective and collaborative efforts of several entities; to include humans and (dis)embodied intelligent agents. Recent advances in immersive technology have led to cost-effective tools that allow us to study and replicate interactions in a controlled environment. Combined together, immersive collective intelligence holds the promise of a symbiotic intelligence that could be greater than the sum of the individual parts. For the military, where the decision making process is typically characterized by high-stress and high-consequence, the concept of a distributive, immersive collective intelligence capability is game changing. Commanders and staff will now be able to remotely immerse themselves in their operational environment with subject matter expertise and advanced analytics. This paper presents the initial steps to understanding immersive collective intelligence with a demonstration designed to discern how military intelligence analysts benefit from an immersive data visualization.

The benefits and limitations of immersive technologies in military decision-making are not well understood. Here, we describe the framework of an experiment which seeks to empirically determine the effects of immersive and non-immersive technology on decision-making. In this experiment, users are shown tactical spatial information about a building layout and told they must decide which of three pre-determined breach points is optimal for maximizing mission success and minimizing risk to the ground team. To ensure observable effects are related to immersion and not simply perception of depth, we deploy a between-subjects design with three viewing conditions: data shown in 2D on a desktop display, data shown in 3D on desktop display, and data shown in 3D in a head-mounted display (HMD). Dependent variables include decision accuracy, time to task completion, decision confidence, and score on the System Usability Scale. In the VR version of the experiment, full telemetry is captured to track when and for how long users interacted with specific information in the scenario environment. Pilot results suggest that tracking these metrics will allow for intricate comparison of decision-making behaviors between display types.

Be it discussed as cybersickness, immersive sickness, simulator sickness, or virtual reality sickness, the ill effects of visuo-vestibular mismatch in immersive environments are of great concern for the wider adoption of virtual reality and related technologies. In this position paper, we discuss a unified research approach that may address motion sickness and identify critical research topics.

Head mounted displays (HMD) may prove useful for synthetic training and augmentation of military C5ISR decision-making. Motion sickness caused by such HMD use is detrimental, resulting in decreased task performance or total user dropout. The genesis of sickness symptoms is often measured using paper surveys, which are difficult to deploy in live scenarios. Here, we demonstrate a new way to track sickness severity using machine learning on data collected from heterogeneous, non-invasive sensors worn by users who navigated a virtual environment while remaining stationary in reality. We discovered that two models, one trained on heterogeneous sensor data and another trained only on electroencephalography ( EEG) data, were able to classify sickness severity with over 95% accuracy and were statistically comparable in performance. Greedy feature optimization was used to maximize accuracy while minimizing the feature subspace. We found that across models, the features with the most weight were previously reported in the literature as being related to motion sickness severity. Finally, we discuss how models constructed on heterogeneous vs homogeneous sensor data may be useful in different real-world scenarios.