In this paper we introduce EMPIRE-PIC, a finite element method particle-in-cell (FEM-PIC) application developed at Sandia National Laboratories. The code has been developed in C++ using the Trilinos library and the Kokkos Performance Portability Framework to enable running on multiple modern compute architectures while only requiring maintenance of a single codebase. EMPIRE-PIC is capable of solv -ing both electrostatic and electromagnetic problems in two-and three-dimensions to second-order accuracy in space and time. In this paper we validate the code against three benchmark problems - a simple electron orbit, an electrostatic Langmuir wave, and a transverse electromagnetic wave propagating through a plasma. We demon-strate the performance of EMPIRE-PIC on four different architectures: Intel Haswell CPUs, Intel's Xeon Phi Knights Landing, ARM Thunder-X2 CPUs, and NVIDIA Tesla V100 GPUs attached to IBM POWER9 processors. This analysis demonstrates scala-bility of the code up to more than two thousand GPUs, and greater than one hundred thousand CPUs.

This report describes the high-level accomplishments from the Plasma Science and Engineering Grand Challenge LDRD at Sandia National Laboratories. The Laboratory has a need to demonstrate predictive capabilities to model plasma phenomena in order to rapidly accelerate engineering development in several mission areas. The purpose of this Grand Challenge LDRD was to advance the fundamental models, methods, and algorithms along with supporting electrode science foundation to enable a revolutionary shift towards predictive plasma engineering design principles. This project integrated the SNL knowledge base in computer science, plasma physics, materials science, applied mathematics, and relevant application engineering to establish new cross-laboratory collaborations on these topics. As an initial exemplar, this project focused efforts on improving multi-scale modeling capabilities that are utilized to predict the electrical power delivery on large-scale pulsed power accelerators. Specifically, this LDRD was structured into three primary research thrusts that, when integrated, enable complex simulations of these devices: (1) the exploration of multi-scale models describing the desorption of contaminants from pulsed power electrodes, (2) the development of improved algorithms and code technologies to treat the multi-physics phenomena required to predict device performance, and (3) the creation of a rigorous verification and validation infrastructure to evaluate the codes and models across a range of challenge problems. These components were integrated into initial demonstrations of the largest simulations of multi-level vacuum power flow completed to-date, executed on the leading HPC computing machines available in the NNSA complex today. These preliminary studies indicate relevant pulsed power engineering design simulations can now be completed in (of order) several days, a significant improvement over pre-LDRD levels of performance.

that the simplest 2D/1D equation has certain desirable properties, (ii) systematically discretize this equation, and (iii) derive a stable iteration scheme for solving the discrete system of equations. In a companion paper [1], we give numerical results that confirm the theoretical predictions of accuracy and iterative stability. (authors)

The azimuthally periodic Z pulsed power facility at Sandia National Laboratories drives megajoules of electrical energy stored at its outer perimeter through a set of radially-convergent parallel transmission lines towards a load at the machine center. The individual line currents are combined in a double post-hole convolute configuration to deliver peak currents up to 26 MA over an ∼ 150 ns pulse width (terawatts). This power flow is associated with a vastness of scales in time (discharge length versus electron cyclotron frequencies), space (transmission line lengths versus electron gyroradius), and velocity (outgassed neutrals versus the propagation speeds of the EM waves in each medium), as well as highly coupled multi-physics which make straightforward numerical approaches not viable. Heterogeneous methods are required. In this talk, we overview our model which bridges the distinct regions in the Z accelerator, i.e. the upstream water/vacuum sections carrying EM waves to the downstream vacuum region comprised of magnetically-insulated transmission lines where crucial 3D effects from emitted particles must be included to predict the power flow characteristics and load current correctly. This complex 3D region spans the kinetic low density plasma regime to high field regions near the load containing high density, highly collisional plasmas. We showcase results from detailed 3D kinetic simulations incorporating electromagnetic wave propagation, relativistic plasma dynamics, field-emitted electrons, and self-consistent surface heating with neutral desorption physics using an in-house particle-in-cell code, EMPIRE, provide cross-code comparisons with results from CHICAGO1, and compare with measurements from actual Z shots.

Recent work has demonstrated that block preconditioning can scalably accelerate the performance of iterative solvers applied to linear systems arising in implicit multiphysics PDE simulations. The idea of block preconditioning is to decompose the system matrix into physical sub-blocks and apply individual specialized scalable solvers to each sub-block. It can be advantageous to block into simpler segregated physics systems or to block by discretization type. This strategy is particularly amenable to multiphysics systems in which existing solvers, such as multilevel methods, can be leveraged for component physics and to problems with disparate discretizations in which scalable monolithic solvers are rare. This work extends our recent work on scalable block preconditioning methods for structure-preserving discretizatons of the Maxwell equations and our previous work in MHD system solvers to the context of multifluid electromagnetic plasma systems. We argue how a block preconditioner can address both the disparate discretization, as well as strongly-coupled off-diagonal physics that produces fast time-scales (e.g. plasma and cyclotron frequencies). We propose a block preconditioner for plasma systems that allows reuse of existing multigrid solvers for different degrees of freedom while capturing important couplings, and demonstrate the algorithmic scalability of this approach at time-scales of interest.

This report reviews a hierarchy of formal mathematical models for describing plasma phenomena. Starting with the Boltzmann equation, a sequence of approximations and modeling assumptions can be made that progressively reduce to the equations for magnetohydrodynamics. Understanding the assumptions behind each of these models and their mathematical form is essential to appropriate use of each level of the hierarchy. A sequence of moment models of the Boltzmann equation are presented, then focused into a generalized three-fluid model for neutral species, electrons, and ions. This model is then further reduced to a two-fluid model, for which Braginskii described a useful closure. Further reduction of the two-fluid model yields a Generalized Ohm's Law model, which provides a connection to magnetohydrodynamic approaches. A verification approach based on linear plasma waves is presented alongside the model hierarchy, which is intended as an initial and necessary but not sufficient step for verification of plasma models within this hierarchy.

Multi-fluid plasma models, where an electron fluid is modeled in addition to multiple ion and neutral species as well as the full set of Maxwell's equations, are useful for representing physics beyond the scope of classic MHD. This advantage presents challenges in appropriately dealing with electron dynamics and electromagnetic behavior characterized by the plasma and cyclotron frequencies and the speed of light. For physical systems, such as those near the MHD asymptotic regime, this requirement drastically increases runtimes for explicit time integration even though resolving fast dynamics may not be critical for accuracy. Implicit time integration methods, with efficient solvers, can help to step over fast time-scales that constrain stability, but do not strongly influence accuracy. As an extension, Implicit-explicit (IMEX) schemes provide an additional mechanism to choose which dynamics are evolved using an expensive implicit solve or resolved using a fast explicit solve. In this study, in addition to IMEX methods we also consider a physics compatible exact sequence spatial discretization. This combines nodal bases (H-Grad) for fluid dynamics with a set of vector bases (H-Curl and H-Div) for Maxwell's equations. This discretization allows for multi-fluid plasma modeling without violating Gauss' laws for the electric and magnetic fields. This initial study presents a discussion of the major elements of this formulation and focuses on demonstrating accuracy in the linear wave regime and in the MHD limit for both a visco-resistive and a dispersive ideal MHD problem.

In this work we present a computational capability featuring a hierarchy of models with different fidelities for the solution of electrokinetics problems at the micro-/nano-scale. A multifidelity approach allows the selection of the most appropriate model, in terms of accuracy and computational cost, for the particular application at hand. We demonstrate the proposed multifidelity approach by studying the mobility of a colloid in a micro-channel as a function of the colloid charge and of the size of the ions dissolved in the fluid.

Time-independent solutions to electron flows in MITLs have been known for many years1. The motivation behind these calculations was to understand experiments by predicting the integrated qualities such as flow current, anode current, cathode current, and voltage. In the process of verifying Finite Element electromagnetic Particle-in-Cell codes developed at Sandia, EMPHASIS and EMPIRE, it was found that these global quantities were very close to the theoretical results for coarse meshing. This is consistent with earlier codes used for the original designs of HERMES, Saturn, and Z2. However, looking at spatial profiles (charge density, current density, velocity, electric and magnetic fields) a finer mesh was required for good comparison to theory. These details matter to the quality of the electron beam for many applications and surface plasma formation. This work focuses on showing the order of convergence for the spatial profiles for MITL flow as a function of the element count, time step, and number of macroparticles used. Most importantly, the impact of the algorithms used for the Space Charge Limited (SCL) emission boundary are examined3, The purpose of this verification is to have increased confidence in the predictive capability of these codes for improved Z power flow designs, High Power Microwave (HPM) sources, and advanced bremsstrahlung accelerators.

This report documents the outcome from the ASC ATDM Level 2 Milestone 6358: Assess Status of Next Generation Components and Physics Models in EMPIRE. This Milestone is an assessment of the EMPIRE (ElectroMagnetic Plasma In Realistic Environments) application and three software components. The assessment focuses on the electromagnetic and electrostatic particle-in-cell solu- tions for EMPIRE and its associated solver, time integration, and checkpoint-restart components. This information provides a clear understanding of the current status of the EMPIRE application and will help to guide future work in FY19 in order to ready the application for the ASC ATDM L 1 Milestone in FY20. It is clear from this assessment that performance of the linear solver will have to be a focus in FY19.