HPC trends favor algorithms and implementations that reduce data motion relative to FLOPS. We investigate the use of lossy compressed data arrays in place of traditional IEEE floating point arrays to store the primary data of calculations. Simulation is fundamentally an exercise in controlled approximation, and error introduced by finite-precision arithmetic (or lossy compression) is just one of several sources of error that need to be managed to ensure sufficient accuracy in a computed result. We describe ZFP, a compressed numerical format designed for in-memory storage of multidimensional arrays, and summarize theoretical results that demonstrate that the error of repeated lossy compression can be bounded and controlled. Furthermore, we establish a relationship between grid resolution and compression-induced errors and show that, contrary to conventional floating point, ZFP reduces finite-difference errors with finer grids. We present example calculations that demonstrate data reduction by 4x or more with negligible impact on solution accuracy. Our results further demonstrate several orders-of-magnitude increase in accuracy using ZFP over IEEE floating point and Posits for the same storage budget.

We introduce stage-local partitioned two-step Runge-Kutta methods are an extension of standard two-step Runge-Kutta methods, which are an alternative to the standard additive two-step Runge-Kutta methods currently existing in the literature. These new schemes are designed with an eye towards truly N-partitioned systems and leverage local stage approximations to make several computationally interesting approximations viable. Specifically, the focus on local stage approximations makes possible the construction of truly asynchronous schemes, in the parallel sense, possible. In addition, we show that an implicit-explicit approach to these schemes can lead to methods that require the inversion of only local nonlinear systems.

An accurate description of turbulence up to the transport time scale is essential for predicting core plasma profiles and enabling reliable calculations for designing advanced scenarios and future devices. Here, we exploit the gap separation between turbulence and transport time scales and couple the global gyrokinetic code GENE to the transport-solver Tango, including kinetic electrons, collisions, realistic geometries, toroidal rotation and electromagnetic effects for the first time. This approach overcomes gyrokinetic codes' limitations and enables high-fidelity profile calculations in experimentally relevant plasma conditions, significantly reducing the computational cost. We present numerical results of GENE-Tango for two ASDEX Upgrade discharges, one of which exhibits a pronounced peaking of the ion temperature profile not reproduced by TGLF-ASTRA. We show that GENE-Tango can correctly capture the ion temperature peaking observed in the experiment. By retaining different physical effects in the GENE simulations, e.g., collisions, toroidal rotation and electromagnetic effects, we demonstrate that the ion temperature profile's peaking is due to electromagnetic effects of submarginal MHD instability. Based on these results, the expected GENE-Tango speedup for the ITER standard scenario is larger than two orders of magnitude compared to a single gyrokinetic simulation up to the transport time scale, possibly making first-principles ITER simulations feasible on current computing resources.

For decades, Lawrence Livermore National Laboratory (LLNL) has been engaged in significant research, development, and support for software to enable scientific computing and, particularly, the use of high performance computing (HPC) in the NNSA mission space. In particular, the move in the mid-1990’s to simulation as a leading component of stockpile stewardship through the ASCI and the successor ASC programs, as well as the need for reliable data acquisition and control software for the National Ignition Facility, have been important drivers in building expertise in production-quality software development at LLNL. LLNL has also been a leader in the DOE SciDAC FASTMath Institute and the DOE Exascale Computing Project (ECP), both of which have striven to make scientific computing software – in particular, the enabling technologies underpinning simulation capabilities – more widely adopted and sustainable. As such, we believe that our experience can inform the broader goal of software stewardship for scientific and high-performance computing. LLNL strongly supports the formation of a new DOE ASCR program element in software stewardship and sustainment. Historically, DOE ASCR has funded applied mathematics and computer science research that has led to the development of important new capabilities and algorithms that are expressed as artifacts in research software. Such frameworks, libraries, and tools have seldom been directly funded to address the important issues of code maintenance, documentation, robustness, and community building. Software engineering and support have typically been done on the side in support of the ASCR-driven research products. DOE funding priorities have been slow to recognize that good software engineering, the kind that ensures research investments have more adoption and longevity, requires significant resources. Based upon our experiences, we have prepared this response to highlight the concerns and issues we believe to be important as DOE ASCR considers its role in scientific software stewardship. We believe that role is important and will require a significant investment of new funding to legitimately support the technologies past and future DOE ASCR investments have and will produce to facilitate their uptake and adoption in the broader scientific computing community. Following a summary of our involvement in scientific software development, the remainder our response is organized around the nine topics specifically identified in the RFI.

We present a scheme that spatially couples two gyrokinetic codes using first-principles. Coupled equations are presented and a necessary and sufficient condition for ensuring accuracy is derived. This new scheme couples both the field and the particle distribution function. The coupling of the distribution function is only performed once every few time-steps, using a five-dimensional (5D) grid to communicate the distribution function between the two codes. This 5D grid interface enables the coupling of different types of codes and models, such as particle and continuum codes, or delta-f and total-f models. Transferring information from the 5D grid to the marker particle weights is achieved using a new resampling technique. Demonstration of the coupling scheme is shown using two XGC gyrokinetic simulations for both the core and edge. We also apply the coupling scheme to two continuum simulations for a one-dimensional advection–diffusion problem.

Two schemes for coupling gyrokinetic simulations of microturbulence in tokamaks are proposed. The first scheme is based on an additive Schwarz domain decomposition. We show that, because the goal of turbulence is long-time averages of the dynamics rather than temporal accuracy, the iteration to self-consistency across domains, which is typically required by Schwarz schemes, can be avoided, thereby accelerating the computation. Second, we propose a coupling scheme that relies entirely on the addition of source terms, leaving the boundary conditions arbitrary. The practical motivations for such a scheme are discussed, and forms of the source terms that ensure consistency and stability are derived. The schemes are tested on a nonlinear, one-dimensional model problem, and the first scheme is further tested on the Hasegawa–Wakatani model.

As scientific applications strive towards increasingly realistic modeling of complex phenomena, they are integrating multiple models and simulations into complex, coupled scientific workflows. As a result, ensuring that existing codes can be combined and recombined correctly and flexibly as part of these workflows is essential. In this paper, we propose Benesh, a programming system for creating in-situ scientific workflows. Benesh provides a domain-specific abstraction that enables a programmer to instrument an existing simulation code to be used as a building block in defining complex workflows. Using Benesh, developers define a workflow-level shared specification of data objects over common or partitioned data domains. This permits dependency-based execution to be specified at the workflow level, distinct from the independent operation of the component simulations. We additionally describe features of a scalable runtime that builds on a distributed data services layer to implement the Benesh programming system.

Machine learning and artificial intelligence (ML/AI) methods have been used successfully in recent years to solve problems in many areas, including image recognition, unsupervised and supervised classification, game-playing, system identification and prediction, and autonomous vehicle control. Data-driven machine learning methods have also been applied to fusion energy research for over 2 decades, including significant advances in the areas of disruption prediction, surrogate model generation, and experimental planning. The advent of powerful and dedicated computers specialized for large-scale parallel computation, as well as advances in statistical inference algorithms, have greatly enhanced the capabilities of these computational approaches to extract scientific knowledge and bridge gaps between theoretical models and practical implementations. Large-scale commercial success of various ML/AI applications in recent years, including robotics, industrial processes, online image recognition, financial system prediction, and autonomous vehicles, have further demonstrated the potential for data-driven methods to produce dramatic transformations in many fields. These advances, along with the urgency of need to bridge key gaps in knowledge for design and operation of reactors such as ITER, have driven planned expansion of efforts in ML/AI within the US government and around the world. The Department of Energy (DOE) Office of Science programs in Fusion Energy Sciences (FES) and Advanced Scientific Computing Research (ASCR) have organized several activities to identify best strategies and approaches for applying ML/AI methods to fusion energy research. This paper describes the results of a joint FES/ASCR DOE-sponsored Research Needs Workshop on Advancing Fusion with Machine Learning, held April 30–May 2, 2019, in Gaithersburg, MD (full report available at https://science.osti.gov/-/media/fes/pdf/workshop-reports/FES_ASCR_Machine_Learning_Report.pdf ). The workshop drew on broad representation from both FES and ASCR scientific communities, and identified seven Priority Research Opportunities (PRO’s) with high potential for advancing fusion energy. In addition to the PRO topics themselves, the workshop identified research guidelines to maximize the effectiveness of ML/AI methods in fusion energy science, which include focusing on uncertainty quantification, methods for quantifying regions of validity of models and algorithms, and applying highly integrated teams of ML/AI mathematicians, computer scientists, and fusion energy scientists with domain expertise in the relevant areas.

Currently, the dominating constraint in many high performance computing applications is data capacity and bandwidth, in both inter-node communications and even more-so in on-node data motion. A new approach to address this limitation is to make use of data compression in the form of a compressed data array. Storing data in a compressed data array and converting to standard IEEE-754 types as needed during a computation can reduce the pressure on bandwidth and storage. However, repeated conversions (lossy compression and decompression) introduce additional approximation errors, which need to be shown to not significantly affect the simulation results. We extend recent work [J. Diffenderfer, et al., Error Analysis of ZFP Compression for Floating-Point Data, SIAM Journal on Scientific Computing, 2019] that analyzed the error of a single use of compression and decompression of the ZFP compressed data array representation [P. Lindstrom, Fixed-rate compressed floating-point arrays, IEEE Transactions on Visualization and Computer Graphics, 2014] to the case of time-stepping and iterative schemes, where an advancement operator is repeatedly applied in addition to the conversions. We show that the accumulated error for iterative methods involving fixed-point and time evolving iterations is bounded under standard constraints. An upper bound is established on the number of additional iterations required for the convergence of stationary fixed-point iterations. An additional analysis of traditional forward and backward error of stationary iterative methods using ZFP compressed arrays is also presented. The results of several 1D, 2D, and 3D test problems are provided to demonstrate the correctness of the theoretical bounds.

provides power to bit slices in the arithmetic logic unit and a clock signal generator provides clock signals to the bit slices. Gate logic is configured to gate the power or the clock signals provided to a subset of the bit slices.

We present an extension of a linearized Coulomb collision operator, previously used in several Eulerian kinetic codes for like-species collisions and unlike-species collisions in the case where the backgrounds about which the linearization is made all are in collisional equilibrium, to the situation of interpenetrating plasma streams. In the latter case, the backgrounds cannot be taken to be in equilibrium and a significant generalization is required. Our development is targeted toward the Eulerian kinetic plasma code LOKI, which evolves the Vlasov–Poisson or Vlasov–Maxwell system in a Cartesian “2 + 2-dimensional” phase space. The extended operator has been implemented in a test code, and results of both quantitative verification and qualitative “realizability” tests are presented.

Compression of floating-point data will play an important role in high-performance computing as data bandwidth and storage become dominant costs. Lossy compression of floating-point data is powerful, but theoretical results are needed to bound its errors when used to store look-up tables, simulation results, or even the solution state during the computation. \black{In this paper, we analyze the round-off error introduced by ZFP, a %state-of-the-art lossy compression algorithm.} The stopping criteria for ZFP depends on the compression mode specified by the user; either fixed rate, fixed accuracy, or fixed precision [P. Lindstrom, Fixed-rate compressed floating-point arrays, IEEE Transactions on Visualization and Computer Graphics, 2014]. While most of our discussion is focused on the fixed precision mode of ZFP, we establish a bound on the error introduced by all three compression modes. In order to tightly capture the error, we first introduce a vector space that allows us to work with binary representations of components. Under this vector space, we define operators that implement each step of the ZFP compression and decompression to establish a bound on the error caused by ZFP. To conclude, numerical tests are provided to demonstrate the accuracy of the established bounds.

Axisymmetric (4D) simulations using the finite-volume code COGENT are performed to explore the role of ion kinetic effects in tokamak edge plasmas. The simulation model solves the long wavelength limit of the full-F gyrokinetic equation for ion species coupled to the 2D quasi-neutrality equation for electrostatic potential variations, where a fluid model is used for an electron response. The ion-ion collisions are described by the nonlinear Fokker-Plank operator and the effects of anomalous transport are included via a radial diffusion model. Illustrative simulations are performed for the parameters characteristic of the LTX tokamak. The development of 5D COGENT for edge plasma turbulence modelling is also reported. To that end, the slab-geometry 5D version has been developed and successfully verified in simulations of the collisionless drift-wave instability that involve gyrokinetic equations for both ion and electron species coupled to the long-wavelength limit of the 3D gyro-Poisson equation. Recent work is focused on extending the 5D code to include the effects of a tokamak edge magnetic geometry.

HPC applications use floating point arithmetic operations extensively to solve computational problems. Mixed-precision computing seeks to use the lowest precision data type that is sufficient to achieve a desired accuracy, improving performance and reducing power consumption. Manually optimizing a program to use mixed precision is challenging as it not only requires extensive knowledge about the numerical behavior of the algorithm but also estimates of the rounding errors. In this work, we present ADAPT, a scalable approach for mixed-precision analysis on HPC workloads using algorithmic differentiation to provide accurate estimates about the final output error. ADAPT provides a floating-point precision sensitivity profile while incurring an overhead of only a constant multiple of the original computation irrespective of the number of variables analyzed. The sensitivity profile can be used to make algorithmic choices and to develop mixed-precision configurations of a program. We evaluate ADAPT on six benchmarks and a proxy application (LULESH) and show that we are able to achieve a speedup of 1.2× on the proxy application.

We describe a new spatial discretization of a continuum gyrokinetic Vlasov model in axisymmetric tokamak edge plasma geometries. The geometries are represented using a multiblock decomposition in which logically distinct blocks are smoothly mapped from rectangular computational domains and are aligned with magnetic flux surfaces to accommodate strong anisotropy induced by the magnetic field. We employ a fourth-order, finite-volume discretization in mapped coordinates to mitigate the computational expense associated with discretization on 4D phase space grids. Applied to a conservative formulation of the gyrokinetic system, a finite-volume approach expresses local conservation discretely in a natural manner involving the calculation of normal fluxes at cell faces. In the approach presented here, the normal fluxes are computed in terms of face-averaged velocity normals in such a way that (i) the divergence-free property of the gyrokinetic velocity is preserved discretely to machine precision, (ii) the configuration space normal velocities are independent of mapping metrics, and (iii) the configuration space normal velocities are computed from exact pointwise evaluation of magnetic field data except for one term. The algorithms described in this paper form the foundation of a continuum gyrokinetic edge code named COGENT, which is used here to perform a convergence study verifying the accuracy of the spatial discretization.

The paper discusses high-order finite-volume numerical modeling of drift waves, which is an ubiquitous phenomenon in magnetized plasmas. It is found that some standard discretization methods applied to the conservative form of the governing equations can lead to a numerical instability. A method to stabilize high-order discretization is proposed and demonstrated to work in numerical simulations performed with the fourth-order finite-volume code COGENT. As practical examples, a stable drift-wave solution with adiabatic electrons and the collisionless (universal) drift-wave instability driven by electron kinetic effects are considered. Application of the present analysis to a broader range of computational fluid dynamics systems is discussed.

COGENT (Continuum Gyrokinetic Edge New Technology) is a kinetic plasma simulation code that is being developed by the edge simulation laboratory (ESL) collaboration. The original version of the code has been developed in a 4D phase space (2D configuration space and 2D velocity space) to address kinetic plasma phenomena in a complex magnetic field geometry including the core, magnetic separatrix, and scrape-off layer regions. This work is focused on extending the original 4D phase space to a 5D phase space (3D2V) to address full kinetic turbulences in a tokamak edge region. Here, we report on the current status of 5D COGENT, which presently operates in a shear-less, simple slab geometry. As a verification study, we use the problems of collision-less drift wave instability (universal instability) including its modification in the presence of collisional effects. The simulation model includes the gyrokinetic equations for ion and electron species coupled to the long-wavelength limit of the gyro-Poisson equation. Collisional effects are represented by the Krook collision model. Linear analytical results for the drift mode growth rate and real frequency are recovered, and the non-linear stage is modelled and analysed as well. In addition, extensive 5D runs have been performed to address the effects of the drift wave instability on blob/filamentary structures characteristic of a tokamak edge. A helical-shaped potential perturbation is observed to grow exponentially in time while spinning around the filament axis with electron drift frequency.

Plasmas with varying collisionalities occur in many applications, such as tokamak edge regions, where the flows are characterized by significant variations in density and temperature. While a kinetic model is necessary for weakly-collisional high-temperature plasmas, high collisionality in colder regions render the equations numerically stiff due to disparate time scales. In this paper, we propose an implicit–explicit algorithm for such cases, where the collisional term is integrated implicitly in time, while the advective term is integrated explicitly in time, thus allowing time step sizes that are comparable to the advective time scales. This partitioning results in a more efficient algorithm than those using explicit time integrators, where the time step sizes are constrained by the stiff collisional time scales. We implement semi-implicit additive Runge–Kutta methods in COGENT, a high-order finite-volume gyrokinetic code and test the accuracy, convergence, and computational cost of these semi-implicit methods for test cases with highly-collisional plasmas.