Deep learning has recently seen rapid development and received significant attention due to its state-of-the-art performance on previously-thought hard problems. However, because of the internal complexity and nonlinear structure of deep neural networks, the underlying decision making processes for why these models are achieving such performance are challenging and sometimes mystifying to interpret. As deep learning spreads across domains, it is of paramount importance that we equip users of deep learning with tools for understanding when a model works correctly, when it fails, and ultimately how to improve its performance. Standardized toolkits for building neural networks have helped democratize deep learning; visual analytics systems have now been developed to support model explanation, interpretation, debugging, and improvement. We present a survey of the role of visual analytics in deep learning research, which highlights its short yet impactful history and thoroughly summarizes the state-of-the-art using a human-centered interrogative framework, focusing on the Five W's and How (Why, Who, What, How, When, and Where). We conclude by highlighting research directions and open research problems. This survey helps researchers and practitioners in both visual analytics and deep learning to quickly learn key aspects of this young and rapidly growing body of research, whose impact spans a diverse range of domains.

We are working on a scalable, interactive visualization system, called Carina, for people to explore million-node graphs. By using latest web browser technologies, Carina offers fast graph rendering via WebGL, and works across desktop (via Electron) and mobile platforms. Different from most existing graph visualization tools, Carina does not store the full graph in RAM, enabling it to work with graphs with up to 69M edges. We are working to improve and open-source Carina, to offer researchers and practitioners a new, scalable way to explore and visualize large graph datasets.

Finding patterns in graphs has become a vital challenge in many domains from biological systems, network security, to finance (e.g., finding money laundering rings of bankers and business owners). While there is significant interest in graph databases and querying techniques, less research has focused on helping analysts make sense of underlying patterns within a group of subgraph results. Visualizing graph query results is challenging, requiring effective summarization of a large number of subgraphs, each having potentially shared node-values, rich node features, and flexible structure across queries. We present VIGOR, a novel interactive visual analytics system, for exploring and making sense of query results. VIGOR uses multiple coordinated views, leveraging different data representations and organizations to streamline analysts sensemaking process. VIGOR contributes: (1) an exemplar-based interaction technique, where an analyst starts with a specific result and relaxes constraints to find other similar results or starts with only the structure (i.e., without node value constraints), and adds constraints to narrow in on specific results; and (2) a novel feature-aware subgraph result summarization. Through a collaboration with Symantec, we demonstrate how VIGOR helps tackle real-world problems through the discovery of security blindspots in a cybersecurity dataset with over 11,000 incidents. We also evaluate VIGOR with a within-subjects study, demonstrating VIGOR's ease of use over a leading graph database management system, and its ability to help analysts understand their results at higher speed and make fewer errors.

Locating and extracting subgraphs from large network datasets is a challenge in many domains, one that often requires learning new querying languages. We will present the first demonstration of VISAGE, an interactive visual graph querying approach that empowers analysts to construct expressive queries, without writing complex code (see our video: https://youtu.be/l2L7Y5mCh1s). VISAGE guides the construction of graph queries using a data-driven approach, enabling analysts to specify queries with varying levels of specificity, by sampling matches to a query during the analyst's interaction. We will demonstrate and invite the audience to try VISAGE on a popular film-actor-director graph from Rotten Tomatoes.

Large data-rich network datasets are ubiquitous. Increasing research and development in graph databases offers analysts the ability to query large networks for interesting patterns. Graph querying allows analysts to quickly find matches for desired subgraphs from a large network (e.g., a clique of suspiciously communicating machines on a company intranet). Many of these systems do not support the exploration of results and leave the analyst with ineffective visualizations like tables and lists. We present our work in progress, VIGOR, an interactive visual system for summarizing and investigating large numbers of graph querying results.

Robocalling, voice phishing, and caller ID spoofing are common cybercrime techniques used to launch scam campaigns through the telephony channel, which unsuspecting users have long trusted. More reliable than online complaints, a telephony honeypot provides complete, accurate and timely information about unwanted phone calls across the United States. Our first goal is to provide a large-scale data-driven analysis of the telephony spam and fraud ecosystem. Our second goal is to uniquely identify bad actors potentially operating several phone numbers. We collected about 40,000 unsolicited calls. Our results show that only a few bad actors, robocallers or telemarketers, are responsible for the majority of the spam and scam calls, and that they can be uniquely identified based on audio features from their calls. This discovery has major implications for law enforcement and businesses that are presently engaged in combatting the rise of telephony fraud. In particular, since our system allows endusers to detect fraudulent behavior and tie it back to existing fraud and spam campaigns, it can be used as the first step towards designing and deploying intelligent defense strategies.

As the bulk electric grid becomes more complex, power system operators and engineers have more information to process and interpret than ever before. The information overload they experience can be mitigated by effective visualizations that facilitate rapid and intuitive assessment of the system state. With the introduction of non-dispatchable renewable energy, flexible loads, and energy storage, the ability to temporally explore system states becomes critical. This paper introduces STEPS, a new 3D Spatio-temporal Electric Power Systems visualization tool suitable for steady-state operational applications.

Who can spot the next Google, Facebook, or Twitter? Who can discover the next billion-dollar startups? Measuring investor success is a challenging task, as investment strategies can vary widely. We propose InvestorRank, a novel method for identifying successful investors by analyzing how an investor's collaboration network change over time. InvestorRank captures the intuition that a successful investor achieves increasingly success in spotting great startups, or is able to keep doing so persistently. Our results show potential in discovering relatively unknown investors that may be the success stories of tomorrow.

Making sense of large graph datasets is a fundamental and challenging process that advances science, education and technology. We survey research on graph exploration and visualization approaches aimed at addressing this challenge. Different from existing surveys, our investigation highlights approaches that have strong potential in handling large graphs, algorithmically, visually, or interactively; we also explicitly connect relevant works from multiple research fields — data mining, machine learning, human-computer ineraction, information visualization, information retrieval, and recommender systems — to underline their parallel and complementary contributions to graph sensemaking. We ground our discussion in sensemaking research; we propose a new graph sensemaking hierarchy that categorizes tools and techniques based on how they operate on the graph data (e.g., local vs global). We summarize and compare their strengths and weaknesses, and highlight open challenges. We conclude with future research directions for graph sensemaking.

Visualization is a powerful paradigm for exploratory data analysis. Visualizing large graphs, however, often results in a meaningless hairball. In this paper, we propose a different approach that helps the user adaptively explore large million-node graphs from a local perspective. For nodes that the user investigates, we propose to only show the neighbors with the most subjectively interesting neighborhoods. We contribute novel ideas to measure this interestingness in terms of how surprising a neighborhood is given the background distribution, as well as how well it fits the nodes the user chose to explore. We are currently designing and developing AdaptiveNav, a fast and scalable method for visually exploring large graphs. By implementing our above ideas, it allows users to look into the forest through its trees.

Visualization is a powerful paradigm for exploratory data analysis. Visualizing large graphs, however, often results in a meaningless hairball. In this paper, we propose a different approach that helps the user adaptively explore large million-node graphs from a local perspective. For nodes that the user investigates, we propose to only show the neighbors with the most subjectively interesting neighborhoods. We contribute novel ideas to measure this interestingness in terms of how surprising a neighborhood is given the background distribution, as well as how well it fits the nodes the user chose to explore. We introduce FACETS, a fast and scalable method for visually exploring large graphs. By implementing our above ideas, it allows users to look into the forest through its trees. Empirical evaluation shows that our method works very well in practice, providing rankings of nodes that match interests of users. Moreover, as it scales linearly, FACETS is suited for the exploration of very large graphs.

Visualization is a powerful paradigm for exploratory data analysis. Visualizing large graphs, however, often results in a meaningless hairball. In this paper, we propose a different approach that helps the user adaptively explore large million-node graphs from a local perspective. For nodes that the user investigates, we propose to only show the neighbors with the most subjectively interesting neighborhoods. We contribute novel ideas to measure this interestingness in terms of how surprising a neighborhood is given the background distribution, as well as how well it fits the nodes the user chose to explore. We are currently designing and developing AdaptiveNav, a fast and scalable method for visually exploring large graphs. By implementing our above ideas, it allows users to look into the forest through its trees.

Given the explosive growth of modern graph data, new methods are needed that allow for the querying of complex graph structures without the need of a complicated querying languages; in short, interactive graph querying is desirable. We describe our work towards achieving our overall research goal of designing and developing an interactive querying system for large network data. We focus on three critical aspects: scalable data mining algorithms, graph visualization, and interaction design. We have already completed an approximate subgraph matching system called MAGE in our previous work that fulfills the algorithmic foundation allowing us to query a graph with hundreds of millions of edges. Our preliminary work on visual graph querying, Graphite, was the first step in the process to making an interactive graph querying system. We are in the process of designing the graph visualization and robust interaction needed to make truly interactive graph querying a reality.

Mobile devices have become increasingly central to our everyday activities, due to their portability, multi-touch capabilities, and ever-improving computational power. Such attractive features have spurred research interest in leveraging mobile devices for computation. We explore a novel approach that aims to use a single mobile device to perform scalable graph computation on large graphs that do not fit in the device's limited main memory, opening up the possibility of performing on-device analysis of large datasets, without relying on the cloud. Based on the familiar memory mapping capability provided by today's mobile operating systems, our approach to scale up computation is powerful and intentionally kept simple to maximize its applicability across the iOS and Android platforms. Our experiments demonstrate that an iPad mini can perform fast computation on large real graphs with as many as 272 million edges (Google+ social graph), at a speed that is only a few times slower than a 13” Macbook Pro. Through creating a real world iOS app with this technique, we demonstrate the strong potential application for scalable graph computation on a single mobile device using our approach.

(PSERC) is a multi-university Center conducting research on challenges facing the electric power industry and educating the next generation of power engineers. More information about PSERC can be found at the Center's website: Notice Concerning Copyright Material PSERC members are given permission to copy without fee all or part of this publication for internal use if appropriate attribution is given to this document as the source material. This report is available for downloading from the PSERC website. project entitled " Seamless Energy Management Systems " (S-53G for 2013-2014). The project has been sponsored by EPRI. We express our appreciation to EPRI and in particular Paul Myrda for his support and guidance for the project. We also express our appreciation for the support provided by PSERC's industrial members, as well as EPRI members, and by the Energy management systems (EMS), control centers that manage the transmission-generation grid, have existed since the 1960s. They have gradually evolved over the years mostly in an incremental manner. Now, major transformation is of EMS systems is essential to support emerging behavior the power system, affected by variable and less predictable renewable energy penetration, availability of newer types of sensors, and communication systems, and powerful computation platforms. As part of this project, various " seams " have been identified that are of interest and that are apparent in current power system operations and control practices. These seams are: 1. Communication architecture of current systems is not suitable for enhanced sensing capability enabled by synchronized phasor measurement units (PMU) 2. The difficulty in comparing application results due to the utilization of different models in operations and planning, and to tie these models to PMU data 3. Repetitiveness of simulations in contingency analysis applications 4. Continued utilization of legacy code in today's software applications, which rely primarily on sequential computing rather than taking advantage of modern high performance computing technology 5. A lack of look-ahead visualization capabilities, which are critical for today's modern electric grid with its non-dispatchable renewables, increasing levels of demand response, and deployment of energy storage. There exists a need for next generation power system management tools that use unified geo-spatial models, more efficiently handle massive scenario evaluation such as contingency analysis, and provide look-ahead visualization capabilities. This report focuses on attempting to address some of these limitations. Chapter II of this report provides a summary of advances in fast communication systems for PMU applications, …

Given a large graph with millions of nodes and edges, say a social network where both its nodes and edges have multiple attributes (e.g., job titles, tie strengths), how to quickly find subgraphs of interest (e.g., a ring of businessmen with strong ties)? We present MAGE, a scalable, multicore subgraph matching approach that supports expressive queries over large, richly-attributed graphs. Our major contributions include: (1) MAGE supports graphs with both node and edge attributes (most existing approaches handle either one, but not both); (2) it supports expressive queries, allowing multiple attributes on an edge, wildcards as attribute values (i.e., match any permissible values), and attributes with continuous values; and (3) it is scalable, supporting graphs with several hundred million edges. We demonstrate MAGE's effectiveness and scalability via extensive experiments on large real and synthetic graphs, such as a Google+ social network with 460 million edges.

Scale-free networks have received much attention in recent years due to their prevalence in many important applications such as social networks, biological systems, and the Internet. We consider the use of conservative parallel discrete event simulation techniques in network simulation applications involving scale-free networks. An analytical model is developed to study the parallelism available in simulations using a conservative time window synchronization algorithm. The performance of scale-free network simulations using two variants of the Chandy/Misra/Bryant synchronization algorithm are evaluated. These results demonstrate the importance of topology in the performance of synchronization protocols when developing parallel discrete event simulations involving scale-free networks, and highlight important challenges such as performance bottlenecks that must be addressed to achieve efficient parallel execution. These results suggest that new approaches to parallel simulation of scale-free networks may offer significant benefit.