Controlling the spread of infectious diseases in large populations is an important societal challenge. Mathematically, the problem is best captured as a certain class of reaction-diffusion processes (referred to as contagion processes) over appropriate synthesized interaction networks. Agent-based models have been successfully used in the recent past to study such contagion processes. We describe EpiSimdemics, a highly scalable, parallel code written in Charm++ that uses agent-based modeling to simulate disease spreads over large, realistic, co-evolving interaction networks. We present a new parallel implementation of EpiSimdemics that achieves unprecedented strong and weak scaling on different architectures - Blue Waters, Cori and Mira. EpiSimdemics achieves five times greater speedup than the second fastest parallel code in this field. This unprecedented scaling is an important step to support the long term vision of realtime epidemic science. Finally, we demonstrate the capabilities of EpiSimdemics by simulating the spread of influenza over a realistic synthetic social contact network spanning the continental United States (~280 million nodes and 5.8 billion social contacts).

Computational epidemiology seeks to develop computational methods to study the distribution and determinants of health-related states or events (including disease), and the application of this study to the control of diseases and other health problems. Recent advances in computing and data sciences have led to the development of innovative modeling environments to support this important goal. The datasets used to drive the dynamic models as well as the data produced by these models presents unique challenges owing to their size, heterogeneity and diversity. These datasets form the basis of effective and easy to use decision support and analytical environments. As a result, it is important to develop scalable data management systems to store, manage and integrate these datasets. In this paper, we develop EpiK—a knowledge base that facilitates the development of decision support and analytical environments to support epidemic science. An important goal is to develop a framework that links the input as well as output datasets to facilitate effective spatio-temporal and social reasoning that is critical in planning and intervention analysis before and during an epidemic. The data management framework links modeling workflow data and its metadata using a controlled vocabulary. The metadata captures information about storage, the mapping between the linked model and the physical layout, and relationships to support services. EpiK is designed to support agent-based modeling and analytics frameworks—aggregate models can be seen as special cases and are thus supported. We use semantic web technologies to create a representation of the datasets that encapsulates both the location and the schema heterogeneity. The choice of RDF as a representation language is motivated by the diversity and growth of the datasets that need to be integrated. A query bank is developed—the queries capture a broad range of questions that can be posed and answered during a typical case study pertaining to disease outbreaks. The queries are constructed using SPARQL Protocol and RDF Query Language (SPARQL) over the EpiK. EpiK can hide schema and location heterogeneity while efficiently supporting queries that span the computational epidemiology modeling pipeline: from model construction to simulation output. We show that the performance of benchmark queries varies significantly with respect to the choice of hardware underlying the database and resource description framework (RDF) engine.

Data movement in high-performance computing systems accelerated by graphics processing units (GPUs) remains a challenging problem. Data communication in popular parallel programming models, such as the Message Passing Interface (MPI), is currently limited to the data stored in the CPU memory space. Auxiliary memory systems, such as GPU memory, are not integrated into such data movement standards, thus providing applications with no direct mechanism to perform end-to-end data movement. We introduce MPI-ACC, an integrated and extensible framework that allows end-to-end data movement in accelerator-based systems. MPI-ACC provides productivity and performance benefits by integrating support for auxiliary memory spaces into MPI. MPI-ACC supports data transfer among CUDA, OpenCL and CPU memory spaces and is extensible to other offload models as well. MPI-ACC's runtime system enables several key optimizations, including pipelining of data transfers, scalable memory management techniques, and balancing of communication based on accelerator and node architecture. MPI-ACC is designed to work concurrently with other GPU workloads with minimum contention. We describe how MPI-ACC can be used to design new communication-computation patterns in scientific applications from domains such as epidemiology simulation and seismology modeling, and we discuss the lessons learned. We present experimental results on a state-of-the-art cluster with hundreds of GPUs; and we compare the performance and productivity of MPI-ACC with MVAPICH, a popular CUDA-aware MPI solution. MPI-ACC encourages programmers to explore novel application-specific optimizations for improved overall cluster utilization.

Agent-based modeling is a computational modeling framework for simulating the interactions of multiple diverse agents representing a complex system or phenomenon. The appoach has been successfully used to model a number of social, biological, and technological systems. In this chapter, we present ENteric Immunity SImulator (ENISI), an agent-based modeling framework for studying the inflammatory and regulatory immune pathways triggered by interactions among microbes and immune cells in the gut. In ENISI, individual cells move through simulated tissues and engage in context-dependent interactions with the other cells with which they are in contact. The scale of ENISI is unprecedented in this domain, with the ability to simulate 107–109 cells for 250 simulated days in 90 min on a modest cluster. We describe the formal representation of the immune system as an agent-based model for modeling mucosal immune responses to gastrointestinal pathogens. We also describe performance and sensitivity analysis techniques and demonstrate the utility of ENISI in guiding the design of wet-lab experiments.

Agent-based models (ABM) are widely used to study immune systems, providing a procedural and interactive view of the underlying system. The interaction of components and the behavior of individual objects is described procedurally as a function of the internal states and the local interactions, which are often stochastic in nature. Such models typically have complex structures and consist of a large number of modeling parameters. Determining the key modeling parameters which govern the outcomes of the system is very challenging. Sensitivity analysis plays a vital role in quantifying the impact of modeling parameters in massively interacting systems, including large complex ABM. The high computational cost of executing simulations impedes running experiments with exhaustive parameter settings. Existing techniques of analyzing such a complex system typically focus on local sensitivity analysis, i.e. one parameter at a time, or a close “neighborhood” of particular parameter settings. However, such methods are not adequate to measure the uncertainty and sensitivity of parameters accurately because they overlook the global impacts of parameters on the system. In this article, we develop novel experimental design and analysis techniques to perform both global and local sensitivity analysis of large-scale ABMs. The proposed method can efficiently identify the most significant parameters and quantify their contributions to outcomes of the system. We demonstrate the proposed methodology for ENteric Immune SImulator (ENISI), a large-scale ABM environment, using a computational model of immune responses to Helicobacter pylori colonization of the gastric mucosa.

Cost estimation is crucial in the performance modeling of parallel algorithms and allocation of computational resources on distributed systems. This paper presents a novel methodology for estimating the cost of constrained producer-consumer (CPC) algorithms. In CPC algorithms, the computation is performed by classes of nodes (tasks), separated in time. The methodology combines data flow analysis with communication latencies to determine the production and consumption of data on different processors, which helps in determining the amount of computations and communication. The cost metric that we develop in this paper uses computational imbalances and communication load, and determines a single cost value. The resulting metric is unique, as it provides the first model that targets CPC algorithms. It has wide application in Genetic Algorithms, molecular dynamics, scheduling schemes and computational epidemiology. We provide a general method for determining the application-specific constants of the cost metric. As an example, we extract the constants for EpiSimdemics (a highly scalable contagion simulator), and give guidelines for applying the procedure to other CPC algorithms. Our evaluations show that the cost metric estimated the execution times of a contagion simulator with less than a 6.5% error. The metric can be used in optimal assignment of computational resources.

We describe the methodology for generating a synthetic population of the United States. A synthetic population integrates a variety of databases from commercial and public sources into a common architecture for data exchange. The process preserves the confidentiality of the individuals in the original data sets, yet produces realistic attributes and demographics for the synthetic individuals. The synthetic population is a set of synthetic people and households, located geographically, each associated with demographic variables recorded in the census. Joint demographic distributions are reconstructed from the marginal distributions available in typical census data using an iterative proportional fitting (IPF) technique [5]. Each synthetic individual is placed in a household with other synthetic individuals. Each household is located geographically using land-use data and data pertaining to transportation networks. The process guarantees that a census of our synthetic population is statistically indistinguishable from the original census. The basic process can be extended to assign other personal and behavioral attributes using additional data sources. Table 1 shows some of the data we have used in constructing previous synthetic populations. The methodology for generating synthetic populations is illustrated in Figure 1. The following list describes the sequence of steps involved: 1. population synthesis, in which a synthetic representation of each household in a region is created from Census data. The individuals in the households are endowed with individual and household level characteristics such as age, gender, marital status, household income, household size, and location; 2. activity assignment, in which each synthetic person in a household is assigned a set of activities to perform during the day, along with the times when the activities begin and end, as given by activity or time-use survey data; 3. location choice, in which an appropriate real location is chosen for each activity for every synthetic person based on a gravity model and data sources such as land use patterns, tax data or commercial location data; ∗NDSSL Technical Report 15-009

Recent advances in social media and other platforms have facilitated collection of near real-time surveillance data on flu outbreaks. This data can not only be used for assessing the current extent of flu epidemics in different regions, but also to project the propagation of the disease in the near future. We have developed FluCaster, a pervasive and scalable web application for situation assessment and forecasting of Influenza-like Illness (ILI), commonly referred to as the flu. FluCaster can be used for assessing the prevalence of ILI at highly resolved spatio-temporal levels. Importantly, FluCaster can also provide the user with short and long term forecasts, in the presence of interventions applied on specific sub-populations. FluCaster is comprised of three basic components: (i) a web-enabled user-interface, (ii) a middleware that coordinates interactions between the UI components and the back end models and data store, and (iii) a back end that is comprised of high resolution epidemic simulations, combined with optimization routines for forecasting and situation assessment. The back end also stores and operates on many types of data, including the GIS data consisting of maps and geographic locations, the synthetic population data corresponding to population demographics, and synthetic contact network data for the different regions under consideration. The underlying mathematical models that we have implemented involve highly resolved information on different regional demographics to compute the forecasting output, and form the basis of the forecasting pipeline. We have described the mathematical formulation of the forecasting models in our previous work. Here we describe the overall systems architecture of FluCaster that supports scalability and accessibility for forecasting of flu outbreaks.

Public health decision makers need access to high resolution situation assessment tools for understanding the extent of various epidemics in different regions of the world. In addition, they need insights into the future course of epidemics by way of forecasts. Such forecasts are essential for planning the allocation of limited resources and for implementing several policy-level and behavioral intervention strategies. The need for such forecasting systems became evident in the wake of the recent Ebola outbreak in West Africa.

Public health decision makers need access to high resolution situation assessment tools for understanding the extent of various epidemics in different regions of the world. In addition, they need insights into the future course of epidemics by way of forecasts. Such forecasts are essential for planning the allocation of limited resources and for implementing several policy-level and behavioral intervention strategies. The need for such forecasting systems became evident in the wake of the recent Ebola outbreak in West Africa. We have developed EpiCaster, an integrated Web application for situation assessment and forecasting of various epidemics, such as Flu and Ebola, that are prevalent in different regions of the world. Using EpiCaster, users can assess the magnitude and severity of different epidemics at highly resolved spatio-temporal levels. EpiCaster provides time-varying heat maps and graphical plots to view trends in the disease dynamics. EpiCaster also allows users to visualize data gathered through surveillance mechanisms, such as Google Flu Trends (GFT) and the World Health Organization (WHO). The forecasts provided by EpiCaster are generated using different epidemiological models, and the users can select the models through the interface to filter the corresponding forecasts. EpiCaster also allows the users to study epidemic propagation in the presence of a number of intervention strategies specific to certain diseases. Here we describe the modeling techniques, methodologies and computational infrastructure that EpiCaster relies on to support large-scale predictive analytics for situation assessment and forecasting of global epidemics.

Modeling dynamical systems represents an important application class covering a wide range of disciplines including but not limited to biology, chemistry, finance, national security, and health care. Such applications typically involve large-scale, irregular graph processing, which makes them difficult to scale due to the evolutionary nature of their workload, irregular communication and load imbalance. EpiSimdemics is such an application simulating epidemic diffusion in extremely large and realistic social contact networks. It implements a graph-based system that captures dynamics among co-evolving entities. This paper presents an implementation of EpiSimdemics in Charm++ that enables future research by social, biological and computational scientists at unprecedented data and system scales. We present new methods for application-specific processing of graph data and demonstrate the effectiveness of these methods on a Cray XE6, specifically NCSA's Blue Waters system.

We present a high performance computing (HPC) based modeling approach to reason about mobile malware. The ubiquity of smart phones and devices and the use of local protocols for disseminating information over such devices has raised new security challenges. The HPC approach to study mobile malware propagation problem involves: (i) a realistic and detailed representation of mobile devices, their time varying location, their usage patterns and the urban environment within which they are operated, leading to dynamic interaction networks over which malware can spread - these networks are large, heterogeneous and time varying; and (ii) a high performance computing based simulation environment that can study diffusion of malware over such networks. We use EpiCure, an individual based high performance simulation tool for malware modeling, that scales to networks spanning urban regions with over 10M individuals. We find that malware dynamics in realistic networks are very different from those in random waypoint (RWP) mobility models. Next, we study the impact of some of the worm model parameters and properties associated with population mobility and social contact networks; we use detailed statistical analysis to identify the significant parameters and their interaction. Finally, we use EpiCure to study SMS/MMS based malware and hybrid malware that spread using both proximity based Bluetooth networks and infrastructure based cellular networks.

We present an integrated interactive modeling environment to support public health epidemiology. The environment combines a high resolution individual-based model with a user-friendly Web-based interface that allows analysts to access the models and the analytics backend remotely from a desktop or a mobile device. The environment is based on a loosely coupled service-oriented-architecture that allows analysts to explore various counterfactual scenarios. As the modeling tools for public health epidemiology are getting more sophisticated, it is becoming increasingly difficult for noncomputational scientists to effectively use the systems that incorporate such models. Thus an important design consideration for an integrated modeling environment is to improve ease of use such that experimental simulations can be driven by the users. This is achieved by designing intuitive and user-friendly interfaces that allow users to design and analyze a computational experiment and steer the experiment based on the state of the system. A key feature of a system that supports this design goal is the ability to start, stop, pause, and roll back the disease propagation and intervention application process interactively. An analyst can access the state of the system at any point in time and formulate dynamic interventions based on additional information obtained through state assessment. In addition, the environment provides automated services for experiment set-up and management, thus reducing the overall time for conducting end-to-end experimental studies. We illustrate the applicability of the system by describing computational experiments based on realistic pandemic planning scenarios. The experiments are designed to demonstrate the system’s capability and enhanced user productivity.

We describe ISIS, a high-performance-computing-based application to support computational epidemiology of infectious diseases. ISIS has been developed over the last seven years in close coordination with public health and policy experts. It has been used in a number of important federal planning and response exercises. ISIS grew out of years of experience in developing and using HPC-oriented models of complex socially coupled systems. This identified the guiding principle that complex models will be used by domain experts only if they can do realistic analysis without becoming computing experts. Using ISIS, one can carry out detailed computational experiments as they pertain to planning and response in the event of a pandemic. ISIS is designed to support networked epidemiology -- study of epidemic processes over social contact networks. The current system can handle airborne infectious diseases such as influenza, pertussis, and smallpox. ISIS is comprised of the following basic components: (i) a web app that serves as the user-interface, (ii) a middleware that coordinates user interaction via the web app with backend models and databases, (iii) a backend computational modeling framework that is comprised of highly resolved epidemic simulations combined with highly realistic control strategies that include pharmaceutical as well as non-pharmaceutical interventions and (iv) a backend data management framework that manages complex unstructured and semi-structured data. ISIS has been used in over a dozen case studies defined by the DoD, DHHS, NIH, BARDA and NSC. We describe three recent studies illustrating the use of ISIS in real-world settings:(i) uses of ISIS during the H1N1 pandemic, Cii) supporting a US military planning exercise, and (iii) distribution of limited stockpile of pharmaceuticals using public and private outlets.

We describe the design and prototype implementation of I ndemics (_Interactive; Epi_demic; _Simulation;)—a modeling environment utilizing high-performance computing technologies for supporting complex epidemic simulations. I ndemics can support policy analysts and epidemiologists interested in planning and control of pandemics. I ndemics goes beyond traditional epidemic simulations by providing a simple and powerful way to represent and analyze policy-based as well as individual-based adaptive interventions . Users can also stop the simulation at any point, assess the state of the simulated system, and add additional interventions. I ndemics is available to end-users via a web-based interface. Detailed performance analysis shows that I ndemics greatly enhances the capability and productivity of simulating complex intervention strategies with a marginal decrease in performance. We also demonstrate how I ndemics was applied in some real case studies where complex interventions were implemented.

Fine-grained communication in supercomputing applications often limits performance through high communication overhead and poor utilization of network bandwidth. This paper presents Topological Routing and Aggregation Module (TRAM), a library that optimizes fine-grained communication performance by routing and dynamically combining short messages. TRAM collects units of fine-grained communication from the application and combines them into aggregated messages with a common intermediate destination. It routes these messages along a virtual mesh topology mapped onto the physical topology of the network. TRAM improves network bandwidth utilization and reduces communication overhead. It is particularly effective in optimizing patterns with global communication and large message counts, such as all-to-all and many-to-many, as well as sparse, irregular, dynamic or data dependent patterns. We demonstrate how TRAM improves performance through theoretical analysis and experimental verification using benchmarks and scientific applications. We present speedups on petascale systems of 6x for communication benchmarks and up to 4x for applications.