With the increasing use and adoption of artificial intelligence (AI), the reliability of modern data systems will be driven by a tighter teaming between human experts and intelligent machine teammates. As in the case of human-human teams, the success of human-machine teams will also rely on clear communication about mutual goals and actions. In this paper, we combine related literature from cognitive psychology, human-machine teaming, uncertainty in data analysis, and multi-agent systems to propose a new form of uncertainty: interaction uncertainty for characterizing bidirectional communication in human-machine teams. We map the causes and effects of interaction uncertainty and outline potential ways to mitigate uncertainty for mutual trust in a high-consequence real-world scenario.

The overall goal of this workshop is to bring together researchers from across the CHI community to share their knowledge and build collaborations at the intersection of computational notebook and HCI research, focusing on both the effective design and effective use of interfaces and interactions within computational notebook environments. This includes innovating upon the computational notebook metaphor, designing new tools, interfaces, and interactions for use with computational notebooks, and more. We aim to pull expertise from across all fields of CHI to deliver novel research and generate open discussion about the current state of computational notebooks, how it can be improved from an HCI standpoint, and how these potential improvements can direct future research. To achieve this goal, we propose a full-day, hybrid workshop with discussions of challenges and opportunities, paper and demo presentations, lightning talks, and a keynote. Participants in this workshop will exchange ideas and help define a roadmap for future research at the intersection of HCI and computational notebook design.

Direct manipulation interactions on projections are often incorporated in visual analytics applications. These interactions enable analysts to provide incremental feedback to the system in a semi-supervised manner, demonstrating relationships that the analyst wishes to find within the data. However, determining the precise intent of the analyst is a challenge. When an analyst interacts with a projection, the inherent ambiguity of interactions can lead to a variety of possible interpretations that the system can infer. Previous work has demonstrated the utility of clusters as an interaction target to address this “With Respect to What” problem in dimension-reduced projections. However, the introduction of clusters introduces interaction inference challenges as well. In this work, we discuss the interaction space for the simultaneous use of semi-supervised dimension reduction and clustering algorithms. We introduce a novel pipeline representation to disambiguate between interactions on observations and clusters, as well as which underlying model is responding to those analyst interactions. We use a prototype visual analytics tool to demonstrate the effects of these ambiguous interactions, their properties, and the insights that an analyst can glean from each.

A novel set of system-state and control-action penalty functions are introduced as an alternative to traditional performance index contingency ranking. The novel system state penalty metrics are formulated based on piecewise linear functions of the system voltage and branch flow, guided by Weber’s Law of human cognition. Novel continuous and discrete control action metrics are also developed to measure the inherent cost and risk associated with every action taken by human power system operator to resolve violations on a pre-contingent basis. These new metrics are combined with traditional human factors indices for measuring human-machine trust and cognitive workload to create a systematic framework for measuring and evaluating operator trust and reliance on artificial intelligence (AI) algorithms for control room use. An existing AI-based contingency analysis recommender tool using a semi-supervised action algorithm is selected for a series of experiments with operations engineering staff using the IEEE 118 Bus System. The penalty metrics presented are demonstrated for both steady-state contingency analysis and transient stability studies, with the operations participants able to reduce the total system penalty in 85% of scenarios through remedial actions. A human-machine team was able to achieve equal or lower continuous control action penalty scores than the participant without availability of the recommender in 57% of experiment scenarios and lower continuous control action penalty scores than the AI tool alone in 83% of scenarios.

In this work, we present a collection of human-centered pitfalls that can occur when using machine learning tools and techniques in modern astronomical research, and we recommend best practices in order to mitigate these pitfalls. Human concerns affect the adoption and evolution of machine learning (ML) techniques in both existing workflows and work cultures. We use current and future surveys such as ZTF and LSST, the data that they collect, and the techniques implemented to process that data as examples of these challenges and the potential application of these best practices, with the ultimate goal of maximizing the discovery potential of these surveys.

Introducing machine learning (ML) assistance into any established process comes with adoption barriers, including entrenched procedures, technological and human readiness levels, human-machine trust, and work culture resistance to change. These barriers are even greater in critical operations such as operating a national or regional power grid, in which both regulatory frameworks and the importance of maintaining reliability levels causes additional resistance to the adoption of new computational support. Developers of future systems and job aides must consider not only technical aspects, but also whether new systems are usable by power system operators. This work presents the methodology and results of a study to evaluate the usability and readiness of a prototype recommender system for power grid contingency analysis. We explore operator cognitive load and evaluate operator performance when solving a collection of scenarios both with and without recommender assistance. We also examine operator trust in the system. We report insights gained on the readiness of the system using a collection of evaluation techniques.

This document presents a sample operations manual for the IEEE 118 Bus Model, which is a synthetic test case developed in 1962 from a section of the transmission grid operated by American Electric Power (AEP). The model is one of the most used synthetic test cases for development of power system applications and is referenced by over 9000 papers. However, the model lacks any context for use with real-time energy management system (EMS) applications, including common considerations, such as generator ramp rates, reactive capabilities, operating limits, and other information typically used by power system operators for real-time decision making. This manual divides the IEEE 118 Bus Model into three operating areas, defines various operating limits, and sets recommended operating procedures for responding to a few types of emergency operating conditions. The manual can be used in support of a wide variety of human-in-the-loop evaluation methodologies for new advanced power applications.

Representing branching and comparative analyses in computational notebooks is complicated by the 1-dimensional (1D), top-down list arrangement of cells. Given the ubiquity of these and other non-linear features, their importance to analysis and narrative, and the struggles current 1D computational notebooks have, enabling organization of computational notebook cells in 2 dimensions (2D) may prove valuable. We investigated whether and how users would organize cells in such a "2D Computational Notebook" through a user study and gathered feedback from participants through a follow-up survey and optional interviews. Through the user study, we found 3 main design patterns for arranging notebook cells in 2D: Linear, Multi-Column, and Workboard. Through the survey and interviews, we found that users see potential value in 2D Computational Notebooks for branching and comparative analyses, but the expansion from 1D to 2D may necessitate additional navigational and organizational aids.

This work presents the application of a methodology to measure domain expert trust and workload, elicit feedback, and understand the technological usability and impact when a machine learning assistant is introduced into contingency analysis for real-time power grid simulation. The goal of this framework is to rapidly collect and analyze a broad variety of human factors data in order to accelerate the development and evaluation loop for deploying machine learning applications. We describe our methodology and analysis, and we discuss insights gained from a pilot participant about the current usability state of an early technology readiness level (TRL) artificial neural network (ANN) recommender.

How do analysts think about grouping and spatial operations? This overarching research question incorporates a number of points for investigation, including understanding how analysts begin to explore a dataset, the types of grouping/spatial structures created and the operations performed on them, the relationship between grouping and spatial structures, the decisions analysts make when exploring individual observations, and the role of external information. This work contributes the design and results of such a study, in which a group of participants are asked to organize the data contained within an unfamiliar quantitative dataset. We identify several overarching approaches taken by participants to design their organizational space, discuss the interactions performed by the participants, and propose design recommendations to improve the usability of future high-dimensional data exploration tools that make use of grouping (clustering) and spatial (dimension reduction) operations.

Detect the expected, discover the unexpected was the founding principle of the field of visual analytics. This mantra implies that human stakeholders, like a domain expert or data analyst, could leverage visual analytics techniques to seek answers to known unknowns and discover unknown unknowns in the course of the data sense-making process. We argue that in the era of AI-driven automation, we need to recalibrate the roles of humans and machines (e.g., a machine learning model) as teammates. We posit that by realizing human-machine teams as a stakeholder unit, we can better achieve the best of both worlds: automation transparency and human reasoning efficacy. However, this also increases the burden on analysts and domain experts towards performing more cognitively demanding tasks than what they are used to. In this paper, we reflect on the complementary roles in a human-machine team through the lens of cognitive psychology and map them to existing and emerging research in the visual analytics community. We discuss open questions and challenges around the nature of human agency and analyze the shared responsibilities in human-machine teams.

Background: Occupational health professionals (OHPs) are in a unique position to impact the health and well-being of employees at work and outside of work. One way of achieving this holistic health goal is to integrate the concept of Total Worker Health® (TWH) into the organization’s culture. It is critical for OHPs to develop the ability to incorporate TWH into their practices, yet there are gaps in our understanding of OHP’s attitudes toward change and toward TWH, their level of TWH knowledge, and the number of OHPs who have adopted TWH. Methods: An electronic survey was administered to a national sample of 4,777. This cross-sectional study used Qualtrics to record survey responses measuring knowledge of TWH, attitude toward change, resistance to change, transformational leadership ability, perception of organizational readiness, and leadership commitment. Findings: The total sample size was 253 (5.3%). Most respondents were bachelors prepared nurses (75.1%) with greater than 10 years’ experience (71.5%) and employed in manufacturing (42.6%). Approximately 74% ( n = 125) of respondents knew about TWH, but did not have a program in place or were unsure of the existence of one. A high percentage (74.0%) were open to implementing TWH, had favorable attitudes toward change ( M = 3.9 on a 5-point Likert-type scale), but needed education on how to move forward (56.0%). Conclusions/Application to Practice: Findings suggest that most OHPs know about TWH, but generally have not adopted the TWH concept at their worksites. However, they are open to implementing TWH programs and have favorable attitudes toward change.

Interactive machine learning(ML)systems are difficult to design because of the"Two Black Boxes"problem that exists at the interface between human and machine.Many algorithms that are used in interactive ML systems are black boxes that are presented to users,while the human cognition represents a second black box that can be difficult for the algorithm to interpret.These black boxes create cognitive gaps between the user and the interactive ML model.In this paper,we identify several cognitive gaps that exist in a previously-developed interactive visual analytics(VA)system,Andromeda,but are also representative of common problems in other VA systems.Our goal with this work is to open both black boxes and bridge these cognitive gaps by making usability improvements to the original Andromeda system.These include designing new visual features to help people better understand how Andromeda processes and interacts with data,as well as improving the underlying algorithm so that the system can better implement the intent of the user during the data exploration process.We evaluate our designs through both qualitative and quantitative analysis,and the results confirm that the improved Andromeda system outperforms the original version in a series of high-dimensional data analysis tasks.

Generating useful network summaries is a challenging and important problem with several applications like sensemaking, visualization, and compression. However, most of the current work in this space do not take human feedback into account while generating summaries. Consider an intelligence analysis scenario, where the analyst is exploring a similarity network between documents. The analyst can express her agreement/disagreement with the visualization of the network summary via iterative feedback, e.g. closing or moving documents ("nodes") together. How can we use this feedback to improve the network summary quality? In this paper, we present NetReAct, a novel interactive network summarization algorithm which supports the visualization of networks induced by text corpora to perform sensemaking. NetReAct incorporates human feedback with reinforcement learning to summarize and visualize document networks. Using scenarios from two datasets, we show how NetReAct is successful in generating high-quality summaries and visualizations that reveal hidden patterns better than other non-trivial baselines.

There is fast-growing literature on provenance-related research, covering aspects such as its theoretical framework, use cases, and techniques for capturing, visualizing, and analyzing provenance data. As a result, there is an increasing need to identify and taxonomize the existing scholarship. Such an organization of the research landscape will provide a complete picture of the current state of inquiry and identify knowledge gaps or possible avenues for further investigation. In this STAR, we aim to produce a comprehensive survey of work in the data visualization and visual analytics field that focus on the analysis of user interaction and provenance data. We structure our survey around three primary questions: (1) WHY analyze provenance data, (2) WHAT provenance data to encode and how to encode it, and (3) HOW to analyze provenance data. A concluding discussion provides evidence-based guidelines and highlights concrete opportunities for future development in this emerging area.

The nascent field of Explainable AI seeks to unmask the underlying details of black box learning algorithms, enabling these algorithms to explain their state and results to human analysts. However, to truly enable interactive AI, we argue that there exists a second black box representing the cognitive process of the user, containing information which must be communicated to the algorithm. Using this “Two Black Boxes” problem as motivation, we present a symmetric, collaborative human-AI model using Semantic Interaction as a design philosophy to connect human and machine. We discuss challenges associated with each phase of communication between the pair of cooperatively-learning entities and the benefits that emerge from combining the expertise of the human and the AI. IN DATA analytics, the “black box” problem denotes the challenge that artificial intelligence (AI) algorithms in general, and neural network models in particular, suffer from opaqueness. These algorithms can supply useful results, such as finding novel latent structure in otherwise difficult to comprehend data. However, they typically do not provide any justification or rationale for their output. Users of these algorithms are therefore faced with the decision of whether to accept the results at face value, without the ability to question or understand the underlying process. This problem has resulted in the “Explainable AI” (XAI) research agenda, which seeks to open the black box of these algorithms and explain their results to human analysts. Analysts can thereby inspect the algorithms and gain insight into how the analytical results were discovered by the algorithm, the process trail, analytical provenance, and supporting data. This is represented by the right-pointing arrow in Figure 1. However, this is only half the problem in human-AI interaction for data analytics. We posit that there is another black box in the equation — the black box of human cognition. Analysts conduct cognitive sensemaking activities, and as a result of these thought processes, also want to be Computer Published by the IEEE Computer Society c © 2020 IEEE 1

Interactive data exploration and analysis is an inherently personal process. One's background, experience, interests, cognitive style, personality, and other sociotechnical factors often shape such a process, as well as the provenance of exploring, analyzing, and interpreting data. This Viewpoint posits both what personal information and how such personal information could be taken into account to design more effective visual analytic systems, a valuable and under-explored direction.

We present a symmetric, collaborative human-artificial intelligence (AI) model using semantic interaction as a design philosophy. We discuss challenges associated with each phase of communication between the pair of cooperatively learning entities and the benefits that emerge from combining the expertise of humans and AI.

Visual analytics tools integrate provenance recording to externalize analytic processes or user insights. Provenance can be captured on varying levels of detail, and in turn activities can be characterized from different granularities. However, current approaches do not support inferring activities that can only be characterized across multiple levels of provenance. We propose a task abstraction framework that consists of a three stage approach, composed of 1) initializing a provenance task hierarchy, 2) parsing the provenance hierarchy by using an abstraction mapping mechanism, and 3) leveraging the task hierarchy in an analytical tool. Furthermore, we identify implications to accommodate iterative refinement, context, variability, and uncertainty during all stages of the framework. We describe a use case which exemplifies our abstraction framework, demonstrating how context can influence the provenance hierarchy to support analysis. The article concludes with an agenda, raising and discussing challenges that need to be considered for successfully implementing such a framework.