A barrier to developing novel AI for complex reasoning is the lack of appropriate wargaming platforms for training and evaluating AIs in a multiplayer setting combining collaborative and adversarial reasoning under uncertainty with game theory and deception. An appropriate platform has several key requirements including flexible scenario design and exploration, extensibility across all five elements of Multi-Domain Operations (MDO), and capability for human-human and human-AI collaborative reasoning and data collection, to aid development of AI reasoning and the warrior-machinelike interface. Here, we describe the ARL Battlespace testbed which fulfills the above requirements for AI development, training and evaluation. ARL Battlespace is offered as an open source software platform (https://github.com/USArmyResearchLab/ARL_Battlespace). We present several example scenarios implemented in ARL Battlespace that illustrate different kinds of complex reasoning for AI development. We focus on 'gap' scenarios that simulate bridgehead and crossing tactics, and we highlight how they address key platform requirements including coordinated MDO actions, game theory and deception. We describe the process of reward shaping for these scenarios that will incentivize an agent to perform command and control (C2) tasks informed by human commanders' courses of action, as well as the key challenges that arise. The intuition presented will enable AI researchers to develop agents that will provide optimal policies for complex scenarios.

In an increasingly complex military operating environment, next generation wargaming platforms can reduce risk, decrease operating costs, and improve overall outcomes. Novel Artificial Intelligence (AI) enabled wargaming approaches, based on software platforms with multimodal interaction and visualization capacity, are essential to provide the decision-making flexibility and adaptability required to meet current and emerging realities of warfighting. We highlight three areas of development for future warfighter-machine interfaces: AI-directed decisional guidance, computationally informed decision-making, and realistic representations of decision spaces. Progress in these areas will enable development of effective human-AI collaborative decision-making, to meet the increasing scale and complexity of today's battlespace.

We present an investigation using mixed reality technology to visualize decision-making dynamics for a Friendly vs Hostile wargame in a Multi-Domain Operation environment. The requirement of penetrate and dis-integrate phases under Multi-Domain Operations aligns well with the advantages of Artificial Intelligence/Machine Learning because of 1) very short planning timeframe for decision-making, 2) simultaneous planning requirement for multiple operations, and 3) interdependence of operations. In our decision dynamics research, we propose to advance the art/science for wargaming by leveraging brain science to extend the use of Artificial Intelligence/Machine Learning algorithms and the use of mixed reality technology to visualize complex battlespace scenarios requiring a better understand of the dynamics in a complex decision making process.

Two video analysis approaches (pose estimation and manual annotation) were applied to video recordings of two-person teams performing a mission planning task in a shared augmented reality (AR) environment. The analysis approaches calculated the distance relations between team members and annotated observed behaviors during the collaborative task. The 2D pose estimation algorithm lacked scene depth processing; therefore, we found some inconsistencies with the manual annotation. Although integration of the two analysis approaches was not possible, each approach by itself produced several insights on team behavior. The manual annotation analysis found four common team behaviors as well as behavior variations unique to particular teams and temporal situations. Comparing a behavior-based time on task percentage indicated behavior-type connections and some possible exclusions. The pose estimation analysis found the majority of the teams moved around the 3D scene at a similar distance apart on average with similar variation in fluctuation around a common distance range between team members. Outlying team behavior was detected by both analysis approaches and included: periods of very low distance relations, infrequent but very high distance relation spikes, significant task time spent adjusting the HoloLens device during wearing, and exceptionally long task time with gaps in pose estimation data processing.

Future Multi Domain Operation (MDO) wargaming will rely on Artificial Intelligence/Machine Learning (AI/ML) algorithms to aid and accelerate complex Command and Control decision-making. This requires an interdisciplinary effort to develop new algorithms that can operate in dynamic environments with changing rules, uncertainty, individual biases, changing cognitive states, as well as the capability to rapidly mitigate unexpected hostile capabilities and exploit friendly technological capabilities. Building on recent advancements in AI/ML algorithms, we believe that new algorithms for learning, reasoning under uncertainty, game theory with three or more players, and interpretable AI can be developed to aid in complex MDO decision-making. To achieve these goals, we developed a new flexible MDO warfighter machine interface game, Battlespace, to investigate and understand how human decision-making principles can be leveraged by and synergized with AI. We conducted several experiments with human vs. random players operating in a fixed environment with fixed rules, where the overall goal of the human players was to collaborate to either capture the opponents’ flags or eliminate all of their units. Then, we analyzed the evolution of the games and identified key features that characterized the human players’ strategies and their overall goal. We then followed a Bayesian approach to model the human strategies and developed heuristic strategies for a simple AI agent. Preliminary analysis revealed that following the human agents’ strategy in the capture the flag games produced the greatest winning percentage and may be useful for gauging the value of intermediate game states for developing the coordinated action planning of reinforcement learning algorithms.

Existing studies on question answering on knowledge bases (KBQA) mainly operate with the standard i.i.d. assumption, i.e., training distribution over questions is the same as the test distribution. However, i.i.d. may be neither achievable nor desirable on large-scale KBs because 1) true user distribution is hard to capture and 2) randomly sampling training examples from the enormous space would be data-inefficient. Instead, we suggest that KBQA models should have three levels of built-in generalization: i.i.d., compositional, and zero-shot. To facilitate the development of KBQA models with stronger generalization, we construct and release a new large-scale, high-quality dataset with 64,331 questions, GrailQA, and provide evaluation settings for all three levels of generalization. In addition, we propose a novel BERT-based KBQA model. The combination of our dataset and model enables us to thoroughly examine and demonstrate, for the first time, the key role of pre-trained contextual embeddings like BERT in the generalization of KBQA.1

For decades, virtual reality (VR) and augmented reality (AR) technologies have remained absent from science and engineering workflows due to the custom-built nature of the hardware and software and oftentimes highly task-specific design. In addition to high cost, these characteristics create a critical entry barrier for most users to invest in VR and AR applications. Recent developments in both VR and AR technologies in the commercial sector have resulted in the availability of affordable consumer-grade and reliable VR and AR hardware and software tools. However, significant research is needed to adapt VR and AR technology use for data-intensive scientific and engineering applications such as network modeling research, human sciences research, and visualization of complex simulated environments. Our VR and AR applications provide examples of how scientists and engineers can use advanced visualization technologies to perform data analysis and assessment, thus transforming scientific discovery.

In this paper, we present results of an investigation comparing two visualization technologies: the Microsoft HoloLens and the Microsoft Surface Touch Table. Two-person teams (dyads) played the role of commander’s staff tasked with planning the most efficient and safest mission route for a squad of soldiers to extract a repository of intelligence documents from the ruins of a building located in enemy territory. Quantitative and qualitative measures of performance were collected. We focused on two performance measures: total mission planning time and mission execution time (the planned route run in a simulated execution mode). Surprisingly, there was a significant decrease in planning time when using the Surface Touch Table. The dyads needed on average 86% more time to plan the mission using the HoloLens. Additionally, this increase in mission planning time associated with the HoloLens did not produce a more optimal mission solution. In a search for understanding the unexpected results, a content analysis of a preferred visualization questionnaire is described. Analysis results suggested: a more realistic scene invited unnecessary exploration instead of focused time on task; becoming familiar with the HoloLens spilled over into task time; collaborative and communication difficulties stemmed from the HoloLens being designed as a single-user device.

We note some future areas for work with cognitive models and agents that as Colbert ( I am America (and so can you!) , 2007) notes, “so can you”. We present three approaches as something like design patterns, so they can be applied to other architectures and tasks. These areas are: (a) Interacting directly with the screen-as-world. It is now possible for models to interact with uninstrumented interfaces both on the machine that the model is running on as well as remote machines. Improved interaction can not only support a broader range of behavior but also make the interaction more accurately model human behavior on tasks that include interaction. Just one implication is that this will force models to have more knowledge about interaction, an area that has been little modeled but essential for all tasks. (b) Providing the cognitive architecture with more representation of the body. In our example, we provide a physiological substrate to implement behavioral moderators’ effects on cognition. Cognitive architectures can now be broader in the measurements they predict and correspond to. This approach provides a more complete and theoretically appropriate way to include new aspects of behavior including stressor effects and emotions in models. And (c) using machine learning techniques, particularly genetic algorithms (GAs), to fit models to data. Because of the model complexity, this is equivalent to performing a multi-variable non-linear stochastic multiple-output regression. Doing this by hand is completely inadequate. While there is a danger of overfitting using a GA, these fits can help provide a better understanding of the model and architecture, including how the architecture changes under moderators such stress. This paper also includes some notes on model maintenance and reporting.

Anger and aggression often result from attributing hostile intent to the ambiguous actions of other people. Two online double-blind parallel randomized pilot studies with healthy adult volunteers tested a novel online cognitive intervention called Hostile Bias Modification Training (HBMT). We hypothesized HBMT would reduce hostile attribution bias, anger, and aggression. In study one, 180 volunteers were randomized to one of three versions of HBMT (active, alternative, and placebo). In study two, 217 volunteers were randomized to either active or placebo HBMT and returned 24–96 h later to complete measures of hostile attribution bias, anger, and aggression. In study one, volunteers who completed active HBMT (vs. placebo) subsequently interpreted hypothetical vignettes as significantly less hostile (d = 0.64, p = .006) and reported significantly less imagined anger (d = 0.51, p = .006) and aggression (d = 0.73, p < .001). In study two, active HBMT (vs. placebo) was associated with reduced hostile attribution bias ( $${\upeta }_{p}^{2}$$ = .089, p < .001), less aggressive driving ( $${\upeta }_{p}^{2}$$ = .039, p = .027), and lower likelihood of cyber-aggression (OR 0.56, p = .004) 24–96 h after intervention. These results suggest HBMT may be an easily implemented intervention to improve anger-related outcomes. No harms to volunteers were observed. This trial was registered at ClinicalTrials.gov: NCT04015440 on July 11 2019.

The proliferation of immersive technologies has accentuated their potential utilization across a broad range of operational situations from strategic planning to the tactical edge. However, little is known about individual and team behavior associated with immersive technologies. We begin to address this challenge by conducting a user-based experiment comparing a 3D augmented reality (AR) device (HoloLens) to a traditional 2D flat screen display. Participant dyads used both technologies to interactively plan a mission to retrieve a repository of intelligence documents located within enemy-held territory. Survey and performance data were collected during the mission planning task. A survey instrument called the Shared Augmented Reality Experience (SARE) survey was developed to measure some aspects of "presence" as defined by factor categories from early virtual environment research. Tested during the mission planning task, a correlation analysis reveals several significant relationships between the survey items and the SARE total score. All relationships between the SARE total score and performance metrics are in the predicted direction; however, only one relationship is significant. Based on the limited data collected to date, experimental outcomes indicate that the level of shared AR experience should be associated with better performance and larger samples should be pursued.

Information Extraction (IE) research has made remarkable progress in Natural Language Processing using intrinsic measures, but little attention has been paid to human analysts as downstream processors. In one experiment, when participants were presented text with or without markup from an IE pipeline, they showed better text comprehension without markup. In a second experiment, the markup was hand-generated to be as relevant and accurate as possible to find conditions under which markup improves performance. This experiment showed no significant difference between performance with and without markup, but a significant majority of participants preferred working with markup to without. Further, preference for markup showed a fairly strong correlation with participants’ ratings of their own trust in automation. These results emphasize the importance of testing IE systems with actual users and the importance of trust in automation.

In this paper, we present a 3D User Interface design for a shared augmented reality setup to support strategic, tactical, and training platforms. An accurate 3D holographic object registration is necessary to enable a shared augmented reality experience using multiple Microsoft HoloLens devices. We developed a sensor data fusion framework which uses both external positional sensor data and Microsoft HoloLens to reduce augmented reality registration errors in our shared augmented reality application. Our shared augmented reality 3D User Interface design implemented in the application, together with our sensor data fusion framework, enables the use of multiple Microsoft HoloLens to support a mission planning scenario.

Automated systems such as information extraction (IE) pipelines are designed to facilitate situation awareness by providing human decision makers with relevant information, but beyond the validity of the pipeline itself, designing the output of the pipeline for optimal human understanding should be a goal. This paper presents results comparing comprehension of text documents with and without markup from a (simulated) IE pipeline in a simulated intelligence task. While previous work suggests that markup hurts both objective and subjective measures of performance and preference, this paper uses handgenerated markup designed to be maximally accurate and task relevant, finding more favorable results. These results, however, still point toward the limitations of markup and the importance of the task it is intended to facilitate.

Expert networks are formed by a group of expert-professionals with different specialties to collaboratively resolve specific queries posted to the network. In such networks, when a query reaches an expert who does not have sufficient expertise, this query needs to be routed to other experts for further processing until it is completely solved; therefore, query answering efficiency is sensitive to the underlying query routing mechanism being used. Among all possible query routing mechanisms, decentralized search, operating purely on each expert's local information without any knowledge of network global structure, represents the most basic and scalable routing mechanism, which is applicable to any network scenarios even in dynamic networks. However, there is still a lack of fundamental understanding of the efficiency of decentralized search in expert networks. In this regard, we investigate decentralized search by quantifying its performance under a variety of network settings. Our key findings reveal the existence of network conditions, under which decentralized search can achieve significantly short query routing paths (i.e., between O(log n) and O(log(2) n) hops, n: total number of experts in the network). Based on such theoretical foundation, we further study how the unique properties of decentralized search in expert networks are related to the anecdotal small-world phenomenon. In addition, we demonstrate that decentralized search is robust against estimation errors introduced by misinterpreting the required expertise levels. The developed performance bounds, confirmed by real datasets, are able to assist in predicting network performance and designing complex expert networks.

Social computing blends computational techniques such as statistics, machine learning, text mining, and graph theory, with psychological and organizational theories of process and structure, and social science theories of membership, engagement and communication. The application of social computing approaches for assessing and shaping the socio-cultural landscape within an area of operations is of growing interest to the defense and intelligence community. These approaches can enable the understanding of how patterns of relations among actors, their environment, and resources influence behavior, and how interventions might change those patterns so altering that behavior. Recently the Army conducted a Workshop on Social Computing Research at the U.S. Army Research Laboratory (ARL). The purpose of the workshop was to understand the strengths and limits of current computational research applied to socially-created data while identifying critical research needs and opportunities of interest to the Army. We discuss several social computing strategies resulting from the workshop and propose a set of recommendations for integrating social computing in key ARL basic research science and technology objectives.

Trust in automation has been linked to a multitude of performance improvements and implicated in the reduction of human error, stress, and workload. In the present study, trust in automation was examined in an experiment measuring the efficacy of linguistic annotation schemes for decision support and human performance. An automated aid provided decisional guidance to assist in intelligence task performance. Four hundred and fifty-eight participants were randomly assigned to one of three annotation schemes and then subsequently performed three simulated intelligence analysis task. The results indicated that trust played a significant role in intelligence task performance, though a significant trust by annotation scheme interaction did not emerge. Specifically, an increase in trust accompanied an increase in performance across the task types. We conclude with a discussion of trust and automated annotation schemes, which has implications for the intelligence operations community.

Accurate 3D holographic object registration for a shared augmented reality application is a challenging proposition with Microsoft HoloLens. We investigated using a sensor data fusion framework which uses both sensor data from an external positional tracking system and the Microsoft HoloLens to reduce augmented reality registration errors. In our setup, positional tracking data from the OptiTrack motion capture system was used to improve the registration of the 3D holographic object for a shared Augmented Reality application running on three Microsoft HoloLens displays. We showed an improved and more accurate 3D holographic object registration in our shared Augmented Reality application compared to the shared augmented reality application using HoloToolkit Sharing Service released by Microsoft. The result of our comparative study of the two applications also showed participants’ responses consistent with our initial assessment on the improved registration accuracy using our sensor data fusion framework. Using our sensor data fusion framework, we developed a shared augmented reality application to support a mission planning scenario using multiple holographic displays to illustrate details of the mission.