AMReX is a C++ software framework that supports the development of block-structured adaptive mesh refinement (AMR) algorithms for solving systems of partial differential equations (PDEs) with complex boundary conditions on current and emerging architectures.

We introduce a topology-aware performance optimization and modeling workflow for AMR simulation that includes two new modeling tools, ProgrAMR and Mota Mapper, which interface with the BoxLib AMR framework and the SSTmacro network simulator. ProgrAMR allows us to generate and model the execution of task dependency graphs from high-level specifications of AMR-based applications, which we demonstrate by analyzing two example AMR-based multigrid solvers with varying degrees of asynchrony. Mota Mapper generates multiobjective, network topology-aware box mappings, which we apply to optimize the data layout for the example multigrid solvers. While the sensitivity of these solvers to layout and execution strategy appears to be modest for balanced scenarios, the impact of better mapping algorithms can be significant when performance is highly constrained by network hop latency. Furthermore, we show that network latency in the multigrid bottom solve is the main contributing factor preventing good scaling on exascale-class machines.

Modern cosmological simulations have reached the trillion-element scale, rendering data storage and subsequent analysis formidable tasks. To address this circumstance, we present a new MPI-parallel approach for analysis of simulation data while the simulation runs, as an alternative to the traditional workflow consisting of periodically saving large data sets to disk for subsequent 'offline' analysis. We demonstrate this approach in the compressible gasdynamics/N-body code Nyx, a hybrid MPI + OpenMP code based on the BoxLib framework, used for large-scale cosmological simulations. We have enabled on-the-fly workflows in two different ways: one is a straightforward approach consisting of all MPI processes periodically halting the main simulation and analyzing each component of data that they own ('in situ'). The other consists of partitioning processes into disjoint MPI groups, with one performing the simulation and periodically sending data to the other 'sidecar' group, which post-processes it while the simulation continues ('in-transit'). The two groups execute their tasks asynchronously, stopping only to synchronize when a new set of simulation data needs to be analyzed. For both the in situ and in-transit approaches, we experiment with two different analysis suites with distinct performance behavior: one which finds dark matter halos in the simulation using merge trees to calculate the mass contained within iso-density contours, and another which calculates probability distribution functions and power spectra of various fields in the simulation. Both are common analysis tasks for cosmology, and both result in summary statistics significantly smaller than the original data set. We study the behavior of each type of analysis in each workflow in order to determine the optimal configuration for the different data analysis algorithms.

The ability to predict the performance of irregular, asynchronous applications on future hardware is essential to the exascale co-design process. Adaptive Mesh Refinement (AMR) applications are inherently irregular and dynamic in their computation and communication patterns, resulting in complex hardware/software interactions. We have developed a methodology to use architectural simulators to assess the performance of different AMR data placement strategies on a selection of potential hardware interconnect topologies for exascale-class supercomputers. We use our framework to study the CASTRO AMR compressible astrophysics code for the simulation of supernovae. The results show a performance improvement of up to 18 percent may be obtained through the use of locality-aware data distributions for some network topologies on an exascale-class supercomputer.

Extending on a companion paper in this colloquium, the dispersion, ignition an d combustion characteristics of aluminum particle clouds is investigated numerically behind reflected shoc k waves. It is observed that a higher proportion of the Al cloud by mass burns for a higher initial cloud concen tration. Vorticity from the cloud wake and from that deposited by the reflected shock cause the particle cloud to c onv lute, and this effect is particularly very significant for higher concentration clouds. Faster ignition delay time s and higher overall Al burning by mass are observed for stronger incident shock Mach numbers due to th consequent hotter gas temperatures behind the reflected shock wave. A mass-weighted ignition parameter is intr oduced in this study and is identified to be particularly useful to determine overall cloud ignition trends.

An empirical model for the ignition of aluminum particle clouds is developed and applied to the study of particle ignition and combustion behavior resulting from explosive blast waves. This model incorporates both particle ignition time delay as well as cloud concentration effects on ignition. The total mass of aluminum that burns is found to depend on the model, with shorter ignition delay times resulting in increased burning of the cloud. A new mass-averaged ignition parameter is defined and is observed to serve as a useful parameter to compare cloud ignition behavior. Investigation of this variable reveals that both peak ignition as well as time required to attain peak ignition are sensitive to the model parameters. Overall, this study demonstrates that the new ignition model developed captures effects not included in other combustion models for the investigation of shock-induced ignition of aluminum particle clouds.

We have developed adaptive high-resolution methods for numerical simulations of turbulent combustion of chemical/biological (C/B) clouds in thermobaric explosions. The code is based on our AMR (Adaptive Mesh Refinement) technology that was used successfully to simulate distributed energy release in explosions, such as: afterburning in TNT explosions and turbulent combustion of Shock-Dispersed Fuel (SDF) charges in confined explosions. Versions of the methodology specialized for low-Mach number flows have also been developed and extensively validated on a number of laboratory scale laminar and turbulent flames configurations. In our formulation, we model the gas phase by the multi-component form of the reacting gas-dynamics equations, while the particle-phase is modeled by continuum mechanics laws for 2-phase reacting flows, as formulated by Nigmatulin. Mass, momentum, and energy interchange between phases are taken into account using Khasainov's model. Both the gas and particle phase conservation laws are integrated with their own second-order Godunov algorithms that incorporate the non-linear wave structure associated with such hyperbolic systems. Specialized ordinary differential equation (ODE) methods are used to integrate chemical kinetics and interphase terms. Adaptive grid methods are used to capture the energy-bearing scales of the turbulent flow (the MILES approach of J. Boris) without resorting to traditional turbulence models. The code is built on an AMR framework that manages the grid hierarchy. Our work-based load-balancing algorithm is designed to run efficiently on massively-parallel computers. Gas-phase combustion in the explosion products (EP) cloud is modeled in the fast-chemistry limit, while Aluminum particle combustion in the EP cloud is based on the finite-rate empirical burning law of Ingignoli. The thermodynamic properties of the components are specified by the Cheetah code. At the 19th HPCUG meeting in 2009, we summarized recent progress in: "AMR Code Simulations of Turbulent Combustion in Confined and Unconfined SDF Explosions". These models were used successfully to simulate the simultaneous after-burning of booster products and combustion of Aluminum (Al) in SDF explosion clouds. Computed pressure histories were shown to be in excellent agreement with the data -- thereby proving the validity of our combustion modeling of such explosions. This year, the modeling has been extended to include the mixing and combustion of C/B clouds in such explosion fields. Here we will establish how the cloud consumption by combustion depends on chamber environments.

A heterogeneous continuum model is proposed to describe the dispersion and combustion of an aluminum particle cloud in an explosion. It combines the gasdynamic conservation laws for the gas phase with a continuum model for the dispersed phase, as formulated by Nigmatulin. Inter-phase mass, momentum and energy exchange are prescribed by phenomenological models. It incorporates a combustion model based on the mass conservation laws for fuel, air and products; source/sink terms are treated in the fast-chemistry limit appropriate for such gasdynamic fields, along with a model for mass transfer from the particle phase to the gas. The model takes into account both the afterburning of the detonation products of the booster with air, and the combustion of the Al particles with air. The model equations were integrated by high-order Godunov schemes for both the gas and particle phases. Numerical simulations of the explosion fields from 1.5-g Shock-Dispersed-Fuel (SDF) charge in a 6.6 liter calorimeter were used to validate the combustion model. Then the model was applied to 10-kg Al-SDF explosions in a vented two-room structure and in an unconfined height-of-burst explosion. Computed pressure histories are in reasonable (but not perfect) agreement with measured waveforms. Differences are caused by physical-chemical kinetic effects of particle combustion which induce ignition delays in the initial reactive blast wave and quenching of reactions at late times. Current simulations give initial insights into such modeling issues.

Laboratory experiments have been performed with 1.5-gram Shock-Dispersed-Fuel (SDF) charges [1]. The charge consisted of a 0.5-g PETN booster surrounded by 1-g of flake Aluminum (Al) powder. The SDF charge was placed at the center of a bomb calorimeter. Detonation of the booster disperses the fuel, ignites it, and induces an exothermic energy release via a turbulent combustion process. Over-pressure histories in air were considerably larger than those measured in nitrogen atmospheres; impulses histories show factors of 2 to 4 increase due to Al-air combustion. At the 21 st

\Ve present a numerical method for solving the multifluid equations of gas dynamics using an operator-split second-order Godunoy method for flow in complex geometries in t'\\'O and three dimensions, The multifluid system treats the fluid components as thermodynamically distinct entities and correctly models fluids with different compressibilities, This treatment allows a general equation-of-state (EOS) specification and the method is implemented so that the EOS references are minimized, The current method is complementary to volume-of-fluid (YOF) methods in the sense that a VOF representation is used 1 but no interface reconstruction is performed. The Oodunm' integrator captures the interface during the solution process. The basic multifluid integrator is coupled to a Cartesian grid algorithm that also uses a VOF representation of the fluidbody interface. This representation of the fluidbody interface allows the algorithm to easily accommodate arbitrarily complex geometries. The • This \\'ork was performed under the auspices of the U.S, Department of Energy by the Lawrence Livermore National Laboratory under contract \V-7405-Eng48. Support under contract W -7 405-Eng-48 was provided by the Applied Mathematical Sciences Program and the HPCC Grand Challenge Program of the Office of Scientific Computing at DOE and by thE' Defense Nuclear Agency under IACRO 95-204.5. Prof. Colella was supported at CC Berkeley by DARPA and the :\ ational Science Foundation under grant DMS-8919074: and by a National Science Foundation Presidential Young Investigator award under grant ACS-8958522; and by the Department of Energy High Performance Computing and Communications Program under grant DE-FG03-92ER25140. 1 resulting single grid multifluid-Cartesian grid integration scheme is coupled to a local adaptive mesh refinement algorithm that dynamically refines selectf'd regions of the computational grid to achie\"e a desired level of accuracy. The overall method is fully conserYatiyp with respect to the total mixture. The method will be used for a simple nozzle problem in two-dimensional axisymmetric coordinates. Introduction and Overview Compressible flows in which the fluid is made up of a number of thermodynamically distinct species, an extreme system being liquid-gas, arise in a wide Yariety of engineering applications requiring realistic geometries, In this paper we describe an algorithm for modeling inviscid compressible multifluid flows containing complex geometries in two and three space dimensions. The basic algorithm is an operator split second-order Godunoy method used to solve the Euler equations for multiftuid flow. The algorithm captures rather than tracks the interfaces between distinct materials while maintaining a volumeof-fluid (VOF) representation of the constituent materials. That is, the interface is obtained during the coarse of the Oodunoy solution without recourse to an interface reconstruction. As such. the present multifluid method provides a complementary approach to '"OF interface tracking algorithms. \Vhile there are numerous approaches to tracking interfaces: we shall only mention those in the class of "OF techniques. The simplest VOF interface reconstruction algorithms are those based on the Simple Line Interface Calculation (SLIC) method [12]. There are numerous other first -order variations on this such as the eenter of mass method [16], central differences [11], and Youngs' method [17]. To obtain a second-order reconstruction, there is an algorithm based on a least squares fit to the local volume fractions profile [15]. In all of the interface tracking methods. a sub-grid scale method of reconstructing the interface must be used to compute t.he current location of the material interface and as a result the interface remains sharp. The primary disad,"antage to using these methods is the expense. Having to reconstruct the interface adds extra computation m"er simply advancing the flow in time. In addition, the reconstruction process is performed on a cell by eel ( basis hence requiring some coding sophistication so that vectoriza.tion can be achieved on modern supercomputers. However. if the number of cells occupied by the interface is small, then this cost may be minimized. \Vhen complex geometry is included, there are added difficulties in coupling a reconstruction algorithm more complex than SLle. Another point of consideration is that interface tracking techniques may not be appropriate for all problems. If the interface is initially sharp and retains its integrity over time then tracking the interface is appropriate. However, if the fluids become mixed either by diffusion, by large-scale motions or are initially mixed, then treating the interface as a discontinuity gives a representation that is inconsistent and probably meaningless. This leads to consideration of the current method since it does not require tracking the interface~ yet has the ability to distinguish thermodynamically distinct fluid components and compute mixture properties using the VOF formulation. Furthermore, the ability to describe such ftows in arbitrarily complex geometry provides a computational capability important for real world engineering applications. \Ve refer to the methodology for treating complex geometry as a Cartesian grid method [5], The basic multifluid method is coupled to a Cartesian grid algorithm which also uses a '"OF representation of the fluid-body interface. This representation of the fluid-body interface allows 2 the algorithm to easily accommodate arbitrarily complex geometries. The resulting single grid multifluid-Cartesian grid integration scheme is coupled to a local Adaptive ~Jesh Refinement (AMR) This is a code based on the original ideas found in [4] and later in [3]. The current version [1], [10] is an object.-oriented (C++) code framework for managing a hierarchy of logically rectangular refined grids that is hybridized with Fortran routines that provide 10,,' level support and integrator instantiation. In regions where errors are deemed unacceptable, a grid is locally refined. This has the two-fold result of increasing accuracy locally where it is required as well as concentrating the computational effort where it is needed. "'hat follows is a description of the multifluid VOF representation and the predictor-corrector Godunm" solution in one dimension. Then there is an oven"ie,," of the previously documented Cartesian grid method, followed by a discussion of the modifications necessary to couple the multifluid-Cartesian grid method into A~IR. A simple nozzle problem in axisymmetric coordinates illustrates the adaptive code results. M ultiftuid Algorithm

For many explosives, only a fraction of the chemical energy is released in the detonation. Calorimetry data for TNT from Ornellas [8] shows that when the ambient gas is inert, there is substantially less total energy released than when the ambient gas is air. This data indicates that burning of the explosion byproducts plays a key role in the overall energetics of the system. The basic concept of shock-dispersed fuel (SDF) charges is to directly exploit this idea. More precisely, in an SDF charge a small charge is used to disperse a fuel and create a turbulent environment in which the fuel can mix with ambient air and burn. Here, we consider a prototype SDF charge in which a 0.5 g PETN booster charge is used to disperse 1.0 g of TNT which plays the role of the fuel [6]. The hot detonation products and dispersed material are rich in fuel (C(s), CO and H2), and when they mix with air and burn, they release 2,500 cal/g (in addition to the 1,100 cal/g released by the detonation) in a non-premixed turbulent combustion process [3]. The goal in this paper is to explore the dependence of the total energy release on the geometry of the calorimeter. As a baseline case, we consider a charge in air in a 6.6 l calorimeter [6]. We first present two simulations of this baseline case corresponding to filling the calorimeter with air and filling the calorimeter with an inert gas. Comparison of these two cases illustrates the role of burning on the overall system energetics. We then discuss two series of simulations focused on exploring the dependence of the system response on problem geometry. In the first set of simulations, the aspect ratios of the calorimeter are held constant but the volume is increased. In the second set the volume is held constant but the aspect ratios are changed. In the next two sections, we briefly discuss the computational model and the numerical method used for the simulations. In the final section we present the computational results and discuss the implications of the results to the design of SDF charges.

We describe the parallelization of a computer program for the adaptive mesh refinement simulation of variable density, viscous, incompressible fluid flows for low Mach number combustion. The adaptive methodology is based on the use of local grids superimposed on a coarse grid to achieve sufficient resolution in the solution. The key elements of the approach to parallelization are a dynamic load-balancing technique to distribute work to processors and a software methodology for managing data distribution and communications. The methodology is based on a message-passing model that exploits the coarse-grained parallelism inherent in the algorithms. A method is presented for parallelizing weakly sequential loops--loops with sparse dependencies among iterations.

. We describe an approach to parallelization of structured adaptive mesh refinement algorithms. This type of adaptive methodology is based on the use of local grids superimposed on a coarse grid to achieve sufficient resolution in the solution. The key elements of the approach to parallelization are a dynamic load-balancing technique to distribute work to processors and a software methodology for managing data distribution and communications. The methodology is based on a message-passing model that exploits the coarse-grained parallelism inherent in the algorithms. The approach is illustrated for an adaptive algorithm for hyperbolic systems of conservation laws in three space dimensions. A numerical example computing the interaction of a shock with a helium bubble is presented. We give timings to illustrate the performance of the method.

Embedded boundary methods model e uid e ows in complex geometries by treating boundaries as tracked interfaces in a regular mesh. Though often referred to as Cartesian grid methods, they are equally well-suited to axisymmetric problems. This paper describes a formulation of the discrete ordinates method for radiative transfer calculations with embedded boundaries. The method uses diamond-difference stencils in the interior with a conservative extension to boundary cells based on a volume-of-e uid approach. Numerical examples are presented in both two-dimensional Cartesian and axisymmetric geometries, including a model of the BERL 300-kW natural gas burner.

We present a numerical method for solving the multifluid equations of gas dynamics using an operator-split second-order Godunov method for flow in complex geometries in two and three dimensions. The multifluid system treats the fluid components as thermodynamically distinct entities and correctly models fluids with different compressibilities. This treatment allows a general equation-of-state (EOS) specification and the method is implemented so that the EOS references are minimized. The current method is complementary to volume-of-fluid (VOF) methods in the sense that a VOF representation is used, but no interface reconstruction is performed. The Godunov integrator captures the interface during the solution process. The basic multifluid integrator is coupled to a Cartesian grid algorithm that also uses a VOF representation of the fluid-body interface. This representation of the fluid-body interface allows the algorithm to easily accommodate arbitrarily complex geometries. The resulting single grid multifluid-Cartesian grid integration scheme is coupled to a local adaptive mesh refinement algorithm that dynamically refines selected regions of the computational grid to achieve a desired level of accuracy. The overall method is fully conservative with respect to the total mixture. The method will be used for a simple nozzle problem in two-dimensional axisymmetric coordinates.