One of the central challenges associated with developing situational understanding originates at the information level. The simple fact is while all information may be created equal, the value of that information is not. Confounding this challenge is the fact that the true value of information is dependent not only on its source and latency, but just as importantly on the context in which it is being exercised. Towards this end, this paper presents a multi-faceted experiment meant to discern how Soldiers weigh the value of information against seven contextual variables under varying military scenarios. Initial results reveal that contextual variables play a significant role in how information is valued and more importantly provide a foundation for developing tailorable information strategies, increasing situational awareness, and reducing information- and cognitive-overload.

Collective intelligence is generally defined as the emergence and evolution of intelligence derived from the collective and collaborative efforts of several entities; to include humans and (dis)embodied intelligent agents. Recent advances in immersive technology have led to cost-effective tools that allow us to study and replicate interactions in a controlled environment. Combined together, immersive collective intelligence holds the promise of a symbiotic intelligence that could be greater than the sum of the individual parts. For the military, where the decision making process is typically characterized by high-stress and high-consequence, the concept of a distributive, immersive collective intelligence capability is game changing. Commanders and staff will now be able to remotely immerse themselves in their operational environment with subject matter expertise and advanced analytics. This paper presents the initial steps to understanding immersive collective intelligence with a demonstration designed to discern how military intelligence analysts benefit from an immersive data visualization.

Army Intelligence operates in a data rich environment with limited ability to operationalize exponentially increasing volumes of disparate structured and unstructured data to deliver timely, accurate, relevant, and tailored intelligence in support of mission command at echelon. The volume, velocity, variety, and veracity (the 4 Vs) of data challenge existing Army intelligence systems and processes, degrading the efficacy of the Intelligence Warfighting Function (IWfF). At the same time, industry has exploited the recent growth in data science technology to address the challenge of the 4 Vs and bring relevant data-driven insights to business leaders. To bring together the lessons from industry and the data science community, the US Army Research Laboratory (ARL) has collaborated with the US Army Intelligence Center of Excellence (USAICoE) to research these Military Intelligence (MI) challenges in an Army AR 5-5 Study entitled, "Application of Data Science within the Army Intelligence Warfighting Function." This paper summarizes the problem statement, research performed, key findings, and way forward for MI to effectively employ data science and data scientists to reduce the burden on Army Intelligence Analysts and increase the effectiveness of data exploitation to maintain a competitive edge over our adversaries.

The benefits and limitations of immersive technologies in military decision-making are not well understood. Here, we describe the framework of an experiment which seeks to empirically determine the effects of immersive and non-immersive technology on decision-making. In this experiment, users are shown tactical spatial information about a building layout and told they must decide which of three pre-determined breach points is optimal for maximizing mission success and minimizing risk to the ground team. To ensure observable effects are related to immersion and not simply perception of depth, we deploy a between-subjects design with three viewing conditions: data shown in 2D on a desktop display, data shown in 3D on desktop display, and data shown in 3D in a head-mounted display (HMD). Dependent variables include decision accuracy, time to task completion, decision confidence, and score on the System Usability Scale. In the VR version of the experiment, full telemetry is captured to track when and for how long users interacted with specific information in the scenario environment. Pilot results suggest that tracking these metrics will allow for intricate comparison of decision-making behaviors between display types.

Knowledge elicitation from a group versus a single source typically yields outcomes that range in agreement and promote a level of uncertainty. Recently the use of Type-2 fuzzy sets has been proffered as an alternative approach for capturing the difference of opinions expressed by a group of experts during the knowledge elicitation process. This paper examines the preliminary results associated with the transition of an existing Type-1 fuzzy set decision support system to a more flexible Interval Type-2 fuzzy approach.

Due to the ever-increasing size of data, construction, analysis and mining of universal massive networks are becoming forbidden and meaningless. In this work, we outline a novel framework called CubeNet, which systematically constructs and organizes real-world networks into different but correlated semantic cells, to support various downstream network analysis and mining tasks with better flexibility, deeper insights and higher efficiency. Particular, we promote our recent research on text and network mining with novel concepts and techniques to (1) construct four real-world large-scale multi-facet hierarchical heterogeneous networks; (2) enable insightful OLAP-style network analysis; (3) facilitate localized and contextual network mining. Although some functions have been covered individually in our previous work, a systematic and efficient realization of an organic system has not been studied, while some functions are still our on-going research tasks. By integrating them, CubeNet may not only showcase the utility of our recent research, but also inspire and stimulate future research on effective, insightful and scalable knowledge discovery under this novel framework.

Fuzzy systems are known to be excellent for reasoning where information is uncertain, incomplete, imprecise, and/or vague. Over the past several years, work has been done to prototype an automated value of information (VoI) decision support tool for military intelligence analysts. The system was constructed using the type-1 fuzzy sets. Recently, the use of type-2 fuzzy sets has been proffered as an alternative to account for the differing opinions expressed by the experts during the knowledge elicitation process. Compounding this challenge for the military is the fact that military science is as much an art as it is a science. The work described in this paper focuses on the investigation with respect to transitioning to type-2 fuzzy sets.

For the military, effective human-agent teaming requires a shared understanding between the human and the intelligent agents acting on their behalf. One of the central challenges associated with developing this shared understanding originates at the information level. The simple fact is while all information may be created equal, the value of information is not. Confounding this calculation is the knowledge that the true value of information is dependent not only on its source, content and latency, but just as importantly on the context of the situation in which it is being exercised. Building upon previous research aimed at codifying the value of information, this paper presents a multi-facetted experiment meant to discern a Soldier's value of information within varying military contexts. Initial results reveal that context plays a significant role in how information is valued and more importantly provides a foundation for strengthening human-agent information understanding and collaboration.

A commonality between simple, everyday tasks and complex, military operations is that they both are dependent on decision-making. In this era of big data, successful decision-making is reliant on the effective contextual exploitation of information. Understanding information and its value within context is a complicated task. For this research context is defined as the macro environment surrounding a decision space. With that understanding, the U.S. Army Research Laboratory (ARL) in collaboration with Towson University is investigating how humans perceive and judge information value within varying context. This paper highlights the experimental design and presents the early findings of an Amazon Mechanical Turk (MTurk) experiment where a human population judged the effects of varying information sources on three different online purchase contexts. Preliminary results indicate that information and situational context play a significant role in discerning information value.

Aspect-based sentiment analysis is an important tool to understand user opinions in a fine-grained manner. Although extensively studied, developing such a tool for a specific domain remains an expensive process. Most existing methods either rely on massive labeled data for training or external language resource and tools which are not necessarily available or accurate. We propose to study the aspect-based sentiment analysis with only a small set of aspect and sentiment seed words as guidance on a target corpus. We first expand the aspect and sentiment lexicons from the given seed words by features created by frequent pattern mining. Then, we develop a generative model to characterize the aspect and sentiment mentions based on their word embedding, and infer the sentiment polarity for sentiment words accordingly. The effectiveness of our method is verified by experiments on two real world data sets.

As a collaborative project funded by US Army Research Lab, our goal is to turn massive unstructured text data into structured heterogeneous information networks (HINs), on which actionable knowledge can be further uncovered flexibly and effectively based on user’s instructions. Taking advantage of open knowledge bases, we develop an end-to-end, data-driven system, AutoNet, with no additional human curation and annotation. AutoNet constructs a large-scale HIN from massive (user-provided) domain-specific text corpora (e.g., scientific papers) using our innovative phrase mining, entity typing, and relation extraction methods, and saves these models for later usage. After that, AutoNet supports two real-time functions: (1) discovery: given a few user-provided documents, AutoNet will construct a new HIN on the fly and highlight those new nodes (i.e., entities) and/or edges (i.e., relations), which are not in the pre-stored network; and (2) exploration: given some user-provided keywords, AutoNet will retrieve a related subnetwork from the large pre-stored HIN. We further design effective visualization tools for both functions. A demo video is available1.

The U.S. Army uses a standardized operation order (OPORD) for planning military operations. In this paper the U.S. Army Research Laboratory (ARL) considers using the OPORD as a basis for prioritizing information from the plethora of intelligence overwhelming an intelligence analyst. The OPORD would provide the input from which to calculate relevancy. To support this effort we review current approaches for calculating relevancy to improve existing information prioritization models, specifically value of information (VoI).

Sensemaking is the cognitive process of extracting information, creating schemata from knowledge, making decisions from those schemata, and inferring conclusions. Human analysts are essential to exploring and quantifying this process, but they are limited by their inability to process the volume, variety, velocity, and veracity of data. Visualization tools are essential for helping this human-computer interaction. For example, analytical tools that use graphical linknode visualization can help sift through vast amounts of information. However, assisting the analyst in making connections with visual tools can be challenging if the information is not presented in an intuitive manner. Experimentally, it has been shown that analysts increase the number of hypotheses formed if they use visual analytic capabilities. Exploring multiple perspectives could increase the diversity of those hypotheses, potentially minimizing cognitive biases. In this paper, we discuss preliminary research results that indicate an improvement in sensemaking over the traditional link-node visualization tools by incorporating an annotation enhancement that differentiates links connecting nodes. This enhancement assists by providing a visual cue, which represents the perceived value of reported information. We conclude that this improved sensemaking occurs because of the removal of the limitations of mentally consolidating, weighing, and highlighting data. This study aims to investigate whether line thickness can be used as a valid representation of VoI.

Today, military decision-making is dependent on the ability to amalgamate information across sources of varying degrees of agreement. Given the increasing volume of information, automated methods to assist in the identification and prioritization of the most valuable or relevant information has become paramount. Relevant information is not only critical to situational awareness and the military decision-making process, but vital to mission success. Towards this end, the US Army Research Laboratory (ARL) has undertaken a research initiative to model and test how analysts perceive the Value of Information (VoI) in varying military context. The goal of this effort is to develop methodologies useful in the development of automated information agents. As a part of the VoI initiative, ARL conducted an experiment with Subject Matter Experts (SMEs) at the US Army Intelligence Center of Excellence (ICOE), where data was collected on how intelligence analysts’ amalgamate information given information content and source reliability within complementary and contradictory conditional associations. The resulting experimental data was incorporated into an Adaptive Control of Thought-Rational (ACT-R) model. Exercising the ACT-R cognitive model resulted in some interesting response behaviors not observed in the initial experiment. In an effort to better understand the perceptions (cognitive underpinnings) of a military intelligence analyst, this paper extends the previous effort and utilizes a crowdsourced experiment within Amazon Mechanical Turk (Mturk). The experiment captures many of the same conditional ratings encountered by the military analysts. Data gained from the Mturk experiment will be examined using the ACT-R model as a simulation to determine whether the same data distributions exist within a wider audience and as a direct comparison to the analyst’s responses. This paper will examine the Mturk experimental design, discuss the experimental apparatus implementation and provide an overview of the ACT-R model utilized to replicate the amalgamation strategies.

Pattern-based methods have been successful in information extraction and NLP research. Previous approaches learn the quality of a textual pattern as relatedness to a certain task based on statistics of its individual content (e.g., length, frequency) and hundreds of carefully-annotated labels. However, patterns of good content-quality may generate heavily conflicting information due to the big gap between relatedness and correctness. Evaluating the correctness of information is critical in (entity, attribute, value)-tuple extraction. In this work, we propose a novel method, called TRUEPIE, that finds reliable patterns which can extract not only related but also correct information. TRUEPIE adopts the self-training framework and repeats the training-predicting-extracting process to gradually discover more and more reliable patterns. To better represent the textual patterns, pattern embeddings are formulated so that patterns with similar semantic meanings are embedded closely to each other. The embeddings jointly consider the local pattern information and the distributional information of the extractions. To conquer the challenge of lacking supervision on patterns' reliability, TruePIE can automatically generate high quality training patterns based on a couple of seed patterns by applying the arity-constraints to distinguish highly reliable patterns (i.e., positive patterns) and highly unreliable patterns (i.e., negative patterns). Experiments on a huge news dataset (over 25GB) demonstrate that the proposed TruePIE significantly outperforms baseline methods on each of the three tasks: reliable tuple extraction, reliable pattern extraction, and negative pattern extraction.

Visual analytics is a field of study which imparts knowledge through visual representations. The use of these visual representations provide a common method for analysts to sift through vast amounts of information and make informed decisions on critical matters. However, assisting the analyst in making connections with visual tools can be challenging if the information is not presented in an intuitive manner. This study aims to build upon our previous work and further investigate whether line thickness can be used as a valid visualization tool to improve situational awareness. In this paper, we follow-up on previous work to discuss research results exploring the impact that information complexity, measured as graph density, has on situational awareness. Our results indicate an increase in situational awareness, compared to non-enhanced visualizations for select graph densities. Furthermore, the results obtained in this study validate previous pilot study findings. The enhancement identified and validated with this research confirms that the line thickness visual cue represents a perceived information value tied to situational awareness. We conclude that this improved situational awareness and time savings occur from the decreased mental burden placed on the analyst.

Data cube is a cornerstone architecture in multidimensional analysis of structured datasets. It is highly desirable to conduct multidimensional analysis on text corpora with cube structures for various text-intensive applications in healthcare, business intelligence, and social media analysis. However, one bottleneck to constructing text cube is to automatically put millions of documents into the right cells in such a text cube so that quality multidimensional analysis can be conducted afterwards—it is too expensive to allocate documents manually or rely on massively labeled data. We propose Doc2Cube, a method that constructs a text cube from a given text corpus in an unsupervised way . Initially, only the label names (e.g., USA, China) of each dimension (e.g., location) are provided instead of any labeled data. Doc2Cube leverages label names as weak super-vision signals and iteratively performs joint embedding of labels, terms, and documents to uncover their semantic similarities. To generate joint embeddings that are discriminative for cube construction, Doc2Cube learns dimension-tailored document representations by selectively focusing on terms that are highly label-indicative in each dimension. Furthermore, Doc2Cube alleviates label sparsity by propagating the information from label names to other terms and enriching the labeled term set. Our experiments on a real news corpus demonstrate that Doc2Cube

Graph pattern mining methods can extract informative and useful patterns from large-scale graphs and capture underlying principles through the overwhelmed information. Contrast analysis serves as a keystone in various fields and has demonstrated its effectiveness in mining valuable information. However, it has been long overlooked in graph pattern mining. Therefore, in this paper, we introduce the concept of contrast subgraph, that is, a subset of nodes that have significantly different edges or edge weights in two given graphs of the same node set. The major challenge comes from the gap between the contrast and the informativeness. Because of the widely existing noise edges in real-world graphs, the contrast may lead to subgraphs of pure noise. To avoid such meaningless subgraphs, we leverage the similarity as the cornerstone of the contrast. Specifically, we first identify a coherent core, which is a small subset of nodes with similar edge structures in the two graphs, and then induce contrast subgraphs from the coherent cores. Moreover, we design a general family of coherence and contrast metrics and derive a polynomial-time algorithm to efficiently extract contrast subgraphs. Extensive experiments verify the necessity of introducing coherent cores as well as the effectiveness and efficiency of our algorithm. Real-world applications demonstrate the tremendous potentials of contrast subgraph mining.

For the military, one of the central challenges associated with developing good situational understanding originates with the information. The character of today’s battlefield features a high volume of information across multiple sources and threatens to overwhelm traditional methods of review.Confounding this effort is the fact that the information received often has varying degrees of agreement (i.e., complementary or contradictory). Timely review requires the development of automated assist methods that identify and prioritize the most valuable and applicable information for mission success. One method to improve military situational awareness and decisionmaking is the ability to amalgamate information across various sources and content. Towards this end, this paper presents the development and evaluation of an Adaptive Control of ThoughtRationale (ACT-R) model to understand the cognitive steps and processes employed by intelligence analysts as they value information.