Quantitative evaluations of human-machine teams (HMTs) are desperately needed to ensure technological implementations are helpful rather than harmful to overall system performance; however, as machines increasingly behave like active cognitive teammates, traditional evaluation strategies risk overestimating HMT capabilities. Areliable HMT evaluation method should include multiple high-resolution, continuous measures for both system performance and system challenges that can be implemented unobtrusively in real-time operations. In our prior work, we proposed joint activity testing (JAT) as acandidate evaluation framework to satisfy these requirements. Preliminary efforts with asingle dimension of performance and challenge have indicated that the method can identify the additive benefits of joint activity with aspecific technology. In this paper, we explore the operationalization of multi-dimensional JAT by synthesizing our work in two intelligence and two healthcare domains. The patterns observed between domains will guide future JAT, reveal paths towards real-time implementation, and spark future research evaluating resilience.

Although the majority of effort in Artificial Intelligence (AI) ideation, design, and development seeks to optimize the AI as the primary method of optimizing overall system performance, the evidence is clear that for risk-critical work in high-complexity, high-uncertainty settings, it is the interactions between human and machines that must be prioritized. Only be effectively coordinating the available machine and human agents can the system be resilient to an increasing set of system demands. This panel will convey the work that they are doing and obstacles they are facing in the following areas: (1) demonstrating the critical importance of human-machine teaming, (2) hardening design patterns that result in successful human- machine teams, (3) designing and evaluating new automation solutions for their ability to team, and (4) ensuring that new automation solutions are implemented and adopted for risk-critical work.

This chapter examines rigor as a measure of sensemaking in information analysis activity that captures how well an analytical process reduces this risk. It describes eight attributes of analytical rigor that were synthesized from empirical studies with professional intelligence analysts from a variety of specialties and agencies. The eight attributes namely hypothesis exploration, information search, information validation, stance analysis, sensitivity analysis, information synthesis, specialist collaboration and explanation critiquing. The concept of analytical rigor in information analysis warrants continued exploration and diverse application as a macrocognitive measure of analytical sensemaking activity. Judging the sufficiency of rigor depends upon many contextual factors—analyst characteristics such as experience, specialization, and organizational affiliation. The chapter provides an example that illustrates how these attributes were calibrated for two different studies of analytical activity. It discusses potential future directions using the attributes to assess, negotiate, and communicate the rigor of information analysis processes.

With an increase in data volume, variety, and velocity, Big Data advances tend to focus on technologies such as data gathering, processing, data storage, and analytics, all of which assume that technology is the limiting factor in leveraging Big Data to its fullest potential. The research framework proposed here takes a more holistic look at the Joint Cognitive System, identifying human attention as the limiting resource for employing Big Data for operational use. The framework leverages prior research in attention management, sensory perception, and joint cognitive systems to lay out a Human Centered Big Data Research agenda for designing attention direction support in Big Data environments.

Eight attributes of a rigorous intelligence analysis process were identified in prior research: hypothesis exploration, information search, information validation, stance analysis, sensitivity analysis, specialist collaboration, information synthesis, and explanation critiquing. Prior findings about new insights generated by augmenting traditional healthcare investigations with human factors expertise were categorized with respect to these attributes. This exploratory work shed light on how well the attributes generalize to a different domain and inspired suggestions for increasing the process rigor of healthcare investigations.

: The proliferation of data accessibility has exacerbated the risk of shallowness in information analysis, making it increasingly difficult to tell when analysis is sufficient for making decisions or changing plans, even as it becomes increasingly easy to find seemingly relevant data. In addressing the risk of shallow analysis, the concept of rigor emerges as an approach for coping with this fundamental uncertainty-motivating the need to better define and understand analytical rigor. The concept of rigor is explored in this thesis through a study that asks how professional analysts decide when there is sufficient rigor in an analytic process. Nine professional intelligence analysts participated in a scenario walkthrough in which they critiqued the analysis processes of two junior analysts-one representing a high-rigor analysis process and the other a low-rigor process. In the study, participants assumed the role of analyst supervisor, deciding if these analyses were of sufficient rigor to send to a decision maker-a fundamental judgment task characterized as the Supervisor's Dilemma. This study design validated and refined the Elicitation by Critiquing methodology, also developing the Liquified Natural Gas Scenario, based on security issues that challenge safety analyses, as a cognitive case for exploring themes in information analysis. This research identified three general findings on rigor in information analysis. First, it found that process insight influenced judgments of rigor. Second, it found that while similar cues were used in forming assessments of rigor, the way in which those cues were interpreted as indicating rigor tended to be more varied. Third, the results of the study suggest a revised definition of analytical rigor, reframing it as an emergent multi-attribute measure of sufficiency rather than as a measure of process deviation.

Across information analysis domains, it is often difficult to recognize when analysis is inadequate for a given context. A better understanding of rigor is an analytic broadening check to be leveraged against this uncertainty. The purpose of this research is to refine the understanding of rigor, exploring the concept within the domain of intelligence analysis. Nine professional intelligence analysts participated in a study of how analytic rigor is judged. The results suggest a revised definition of rigor, reframing it as an emergent multi-attribute measure of sufficiency rather than as a measure of process deviation. Based on this insight, a metric for assessing rigor was developed, identifying eight attributes of rigorous analysis. Finally, an alternative model of briefing interactions is proposed that integrates this framing of rigor into an applied context. This research, although specific in focus to intelligence analysis, shows the potential to generalize across forms of information analysis.

Detecting biased information, and deriving an accurate assessment of issues or situations from a pool of information representing multiple factions, are challenges for which intelligence analysts have developed strategies. These include researching sources, submitting products for peer review, explicitly contrasting pro vs. con positions, and comparing predictions along a spectrum of optimism/pessimism. We explore the concept of a Faction Display as a means to use visualization to support awareness of the location of a source along a spectrum of opinion. It can be used to place estimates or other assessments in the context of the source’s position among the set of factions in play. The Faction Display concept is illustrated as a “design seed” placed within the context of a safety analysis case study; the concept displays relative estimates for safe distances from Liquefied Natural Gas (LNG) pool fires resulting from a terrorist attack.

This study examines how professional intelligence analysts judge the rigor behind an analysis. The study investigates the challenges that inhibit the understanding of rigor in intelligence analysis and explores cues used by analysts to identify analytic rigor—or lack of rigor. Nine professional intelligence analysts participated in a modified elicitation by critiquing method study, embedded in a scenario walkthrough. Findings from the study indicate that, while professional intelligence analysts can make perceptive assessments about the quality of an analysis process based on product quality, these perceptions are apt to change with insight into the analytic process.

In this paper, we describe the use of an animated prototyping technique to elicit feedback on the usefulness of a set of modular design concepts for dealing with information dynamics in inferential analysis under data overload conditions. The design concepts were comprised of innovative solutions that utilized technology to assist in inferential analysis. The findings generally support the “promisingness” of the design directions for addressing information dynamics challenges in inferential analysis under data overload. Elicited feedback provides insight on how the concepts might prove useful to intelligence analysts in the field. Analysts recommended significant modifications that would be difficult to change post-implementation of software, suggesting that the animock technique was useful for exploring how design concepts could address challenging issues where no current software support exists.

University campuses, like many other public and private institutional settings, pose challenges to visitors and newcomers finding their way from place to place. In some cases, such campuses have grown to the size of a small town. Maps and tour guides have traditionally been the means used to assist visitors find their way; however, the recent development of high-power, low-cost mobile computing opens the door to portable electronic navigational aids. This paper focuses on user interface concerns in a personal digital assistant (PDA) based campus guide. Cognitive and visual display engineering principles are used to develop a preferred preliminary design. Subjective feedback and quantitative data on the user interface are gathered in a small pilot study. The appropriateness of the design and its implications for future work are also discussed.

Our recent research with professional information analysts has revealed promising directions for representing and sharing analyses to create process insight, suggesting two related directions for developing effective representations that support the assessment of rigor. First, our study suggests that effective representation incorporates a synthesis of critical process attributes, finding that analysts viewed analytic rigor as a multi-attribute assessment of sufficiency. Second, the research suggests that an effective representation for revealing analytic rigor facilitates the emergence of a participatory exchange among stakeholders. These two findings indicate that developing an effective representation for creating insight into an analytic process embeds indicators of critical process attributes into a participatory exchange interaction. Our approach for supporting rigor assessment builds a virtual map to the data space behind an analysis—a visualization that affords active shifting among organizing views of critical attributes of an analysis process.