The gyrokinetic (GK) field equation is a three-dimensional (3D) elliptic equation, but it is often simplified to a set of two-dimensional (2D) equations by assuming that the field does not vary along a specific direction. However, this simplification can introduce inevitable 0th-order numerical errors, as nonlinear mode coupling in toroidal geometry can produce undesirable harmonic modes that violate the assumption. In this work, we propose a novel directional finite difference method (FDM) with a local coordinate transformation to better resolve the target field of interest. The directional FDM can accurately solve 3D GK field equations without simplifications, which can overcome the limitations of conventional methods. The accuracy and efficiency of different FDMs are analyzed in great detail for a variety of geometries, from simple 2D Cartesian coordinates to realistic 3D curvilinear coordinates. The 0th-order numerical errors of simplified 2D GK equations were found to be more problematic for low-harmonic modes and low aspect ratio geometries such as spherical tokamaks. On the other hand, the directional 3D FDM can accurately resolve a much wider range of harmonic modes aligned to the direction of interest, including the low-harmonic modes. We demonstrate that the directional 3D FDM is a highly effective algorithm for solving the 3D GK field equations, achieving accuracy improvements of 10 to 100 times or more, particularly for low-harmonic modes in spherical tokamaks.

Magnetic island perturbations may cause a reduction in plasma self-driven current that is needed for tokamak operation. A novel effect on tokamak self-driven current revealed by global gyrokinetic simulations is due to magnetic-island-induced 3D electric potential structures, which have the same dominant mode numbers as that of the magnetic island, whereas centered at both the inner and outer edge of the island. The non-resonant potential islands are shown to drive a current through an efficient nonlinear parallel acceleration of electrons. In large aspect ratio (large-A) tokamak devices, this new effect can result in a significant global reduction of the electron bootstrap current when the island size is sufficiently large, in addition to the local current loss across the island region due to the pressure profile flattening. It is shown that there exists a critical magnetic island width for large-A tokamaks beyond which the electron bootstrap current loss is global and increases rapidly with the island size. As such, this process may introduce a size limit for tolerable magnetic islands in large-A tokamak devices in the context of steady state operation. On the other hand, the current loss caused by magnetic islands in low-A tokamaks such as spherical tokamak (ST) NSTX/U is minor. The reduction of the axisymmetric current by magnetic islands scales with the square of island width. However, the loss of the current is mainly local to the island region, and the pace of current loss as the island size increases is substantially slower compared to large-A tokamaks. In particular, the bootstrap current reduction in STs is even smaller in the reactor-relevant high-beta(p) regime where neoclassical tearing modes are more likely to develop.

Recently, the numerical scheme presented by Mishchenko et al. [Phys. Plasmas 21, 052113 (2014); 21, 092110 (2014)] enabled explicit gyrokinetic simulations of low-frequency electromagnetic instabilities in tokamaks at experimentally relevant values of plasma beta. This scheme resolved the long-standing cancellation problem that previously hindered gyrokinetic particle-in-cell code simulations of magnetohydrodynamic phenomena with inherently small parallel electric fields. Moreover, the scheme did not employ approximations that eliminate critical tearing-type instabilities. Here, we report on the implementation of this numerical scheme in the global gyrokinetic particle-in-cell code GTS. This implementation allows for a more complete and accurate picture of interaction between small scale turbulence and MHD modes in tokamaks. Additionally, we present a comprehensive set of verification simulations of numerous electromagnetic instabilities relevant to present-day tokamaks. These simulations encompass the kinetic ballooning mode, the internal kink mode, the tearing mode, the micro-tearing mode, and the toroidal Alfven eigenmode destabilized by energetic ions, which are all instrumental in understanding tokamak physics. We will also showcase the preliminary nonlinear simulations of kinetic ballooning instabilities and (2,1) island formation due to tearing mode instability. These simulations validate the accuracy of the scheme implementation and pave the way for studying how these instabilities affect plasma confinement and performance.

Plasmas generated using energetic electron beams are well known for their low electron temperature (T (e)) and plasma potential, which makes them attractive for atomic-precision plasma processing applications such as atomic layer etch and deposition. A 2-dimensional particle-in-cell model for an electron beam-generated plasma in argon confined by a constant applied magnetic field is described in this article. Plasma production primarily occurs in the path of the beam electrons in the center of the chamber. The resulting plasma spreads out in the chamber through non-ambipolar diffusion with a short-circuiting effect allowing unequal electron and ion fluxes to different regions of the bounding conductive chamber walls. The cross-field transport of the electrons (and thus the steady-state characteristics of the plasma) are strongly impacted by the magnetic field. T (e) is anisotropic in the electron beam region, but low and isotropic away from the plasma production zone. The plasma density increases and the plasma becomes more confined near the region of production when the magnetic field strengthens. The magnetic field reduces both electron physical and energy transport perpendicular to the magnetic field. T (e) is uniform along the magnetic field lines and slowly decreases perpendicular to it. Electrons are less energetic in the sheath regions where the sheath electric field repels and confines the low-energy electrons from the bulk plasma. Even though electron and ion densities are similar in the bulk plasma due to quasi-neutrality, electron and ion fluxes on the grounded chamber walls are unequal at most locations. Electron confinement by the magnetic field weakens with increasing pressure, and the plasma spread out farther from the electron beam region.

High-energy particle resonances can modify particle distributions and even cause significant particle loss. Resonances can be present in any toroidal confinement device and can easily be found numerically. Many stellarators have weak magnetic shear so that large islands and large chaotic regions can be produced by resonant perturbations with small amplitudes. While the choice of the field line helicity profile in the plasma can limit the presence of resonances at low particle energy, the resonance location is energy-dependent, and they can move into the plasma at higher energy. If resonances match the toroidal variation of the equilibrium, they can produce wide islands in the phase space of orbits even in the absence of perturbations due to instabilities. These islands increase in size with particle energy and can seriously affect the confinement of high-energy ions.

The thermal quench triggered by locked modes is known to be mainly due to open stochastic magnetic field lines connected to the wall boundary. It is essential to understand the 3D structure of open stochastic field lines since it determines the overall plasma dynamics in the system. In this study, we analyze the 3D magnetic topology for two key concepts, the connection length Lc and the effective magnetic mirror ratio Meff, and present a comprehensive picture of electron and ion dynamics related to the magnetic topology. The connection length determines the 3D structure of the ambipolar potential, and a sharp potential drop across distinct Lc regions induces the E × B transport and mixing across the field line. The confinement of electrons and ions along the field line is determined by the ambipolar potential and Meff configuration. Electron and ion temperatures in magnetic hills (Meff<1) are lower than in magnetic wells (Meff>1) because particles in magnetic hills are more likely to escape toward the wall boundary along the field line. The mixing between the magnetic wells and hills by E × B and magnetic drift motions results in collisionless detrapping of electrons and ions, which reduces their temperature efficiently. Numerical simulations of two different magnetic configurations demonstrate the importance of the collisionless detrapping mechanism, which could be the main cause of plasma temperature drop during the thermal quench.

As the growth of data sizes continues to outpace computational resources, there is a pressing need for data reduction techniques that can significantly reduce the amount of data and quantify the error incurred in compression. Compressing scientific data presents many challenges for reduction techniques since it is often on non-uniform or unstructured meshes, is from a high-dimensional space, and has many Quantities of Interests (QoIs) that need to be preserved. To illustrate these challenges, we focus on data from a large scale fusion code, XGC. XGC uses a Particle-In-Cell (PIC) technique which generates hundreds of PetaBytes (PBs) of data a day, from thousands of timesteps. XGC uses an unstructured mesh, and needs to compute many QoIs from the raw data, f. One critical aspect of the reduction is that we need to ensure that QoIs derived from the data (density, temperature, flux surface averaged momentums, etc.) maintain a relative high accuracy. We show that by compressing XGC data on the high-dimensional, nonuniform grid on which the data is defined, and adaptively quantizing the decomposed coefficients based on the characteristics of the QoIs, the compression ratios at various error tolerances obtained using a multilevel compressor (MGARD) increases more than ten times. We then present how to mathematically guarantee that the accuracy of the QoIs computed from the reduced f is preserved during the compression. We show that the error in the XGC density can be kept under a user-specified tolerance over 1000 timesteps of simulation using the mathematical QoI error control theory of MGARD, whereas traditional error control on the data to be reduced does not guarantee the accuracy of the QoIs.

We present the Exascale Framework for High Fidelity coupled Simulations (EFFIS), a workflow and code coupling framework developed as part of the Whole Device Modeling Application (WDMApp) in the Exascale Computing Project. EFFIS consists of a library, command line utilities, and a collection of run-time daemons. Together, these software products enable users to easily compose and execute workflows that include: strong or weak coupling, in situ (or offline) analysis/visualization/monitoring, command-and-control actions, remote dashboard integration, and more. We describe WDMApp physics coupling cases and computer science requirements that motivate the design of the EFFIS framework. Furthermore, we explain the essential enabling technology that EFFIS leverages: ADIOS for performant data movement, PerfStubs/TAU for performance monitoring, and an advanced COUPLER for transforming coupling data from its native format to the representation needed by another application. Finally, we demonstrate EFFIS using coupled multi-simulation WDMApp workflows and exemplify how the framework supports the project’s needs. We show that EFFIS and its associated services for data movement, visualization, and performance collection does not introduce appreciable overhead to the WDMApp workflow and that the resource-dominant application’s idle time while waiting for data is minimal.

We present a scheme that spatially couples two gyrokinetic codes using first-principles. Coupled equations are presented and a necessary and sufficient condition for ensuring accuracy is derived. This new scheme couples both the field and the particle distribution function. The coupling of the distribution function is only performed once every few time-steps, using a five-dimensional (5D) grid to communicate the distribution function between the two codes. This 5D grid interface enables the coupling of different types of codes and models, such as particle and continuum codes, or delta-f and total-f models. Transferring information from the 5D grid to the marker particle weights is achieved using a new resampling technique. Demonstration of the coupling scheme is shown using two XGC gyrokinetic simulations for both the core and edge. We also apply the coupling scheme to two continuum simulations for a one-dimensional advection–diffusion problem.

The Exascale Computing Project (ECP) is invested in co-design to assure that key applications are ready for exascale computing. Within ECP, the Co-design Center for Particle Applications (CoPA) is addressing challenges faced by particle-based applications across four “sub-motifs”: short-range particle–particle interactions (e.g., those which often dominate molecular dynamics (MD) and smoothed particle hydrodynamics (SPH) methods), long-range particle–particle interactions (e.g., electrostatic MD and gravitational N-body), particle-in-cell (PIC) methods, and linear-scaling electronic structure and quantum molecular dynamics (QMD) algorithms. Our crosscutting co-designed technologies fall into two categories: proxy applications (or “apps”) and libraries. Proxy apps are vehicles used to evaluate the viability of incorporating various types of algorithms, data structures, and architecture-specific optimizations and the associated trade-offs; examples include ExaMiniMD, CabanaMD, CabanaPIC, and ExaSP2. Libraries are modular instantiations that multiple applications can utilize or be built upon; CoPA has developed the Cabana particle library, PROGRESS/BML libraries for QMD, and the SWFFT and fftMPI parallel FFT libraries. Success is measured by identifiable “lessons learned” that are translated either directly into parent production application codes or into libraries, with demonstrated performance and/or productivity improvement. The libraries and their use in CoPA’s ECP application partner codes are also addressed.

GENE solves the five-dimensional gyrokinetic equations to simulate the development and evolution of plasma microturbulence in magnetic fusion devices. The plasma model used is close to first principles and computationally very expensive to solve in the relevant physical regimes. In order to use the emerging computational capabilities to gain new physics insights, several new numerical and computational developments are required. Here, we focus on the fact that it is crucial to efficiently utilize GPUs (graphics processing units) that provide the vast majority of the computational power on such systems. In this paper, we describe the various porting approaches considered and given the constraints of the GENE code and its development model, justify the decisions made, and describe the path taken in porting GENE to GPUs. We introduce a novel library called gtensor that was developed along the way to support the process. Performance results are presented for the ported code, which in a single node of the Summit supercomputer achieves a speed-up of almost 15× compared to running on central processing unit (CPU) only. Typical GPU kernels are memory-bound, achieving about 90% of peak. Our analysis shows that there is still room for improvement if we can refactor/fuse kernels to achieve higher arithmetic intensity. We also performed a weak parallel scalability study, which shows that the code runs well on a massively parallel system, but communication costs start becoming a significant bottleneck.

Resonances of high energy particles in magnetic confinement devices due to electromagnetic instabilities can strongly modify the particle distribution, leading to a reduction in fusion power and even discharge termination and particle loss to the device walls through an avalanche. The existence of a mode particle resonance depends on the properties of the equilibrium and particle parameters, and their number, location, and density can vary with device design. Recently, the advent of more powerful computing capabilities and advanced theoretical understanding has led to the design of non-axisymmetric devices or stellarators, which could prove to be more advantageous than tokamaks. Stellarators have the advantage of being immune to major disruptions because of the very low plasma current. One of the problems shared by both types of devices is the existence of resonances in particle orbits, which can lead to large amplitude high frequency instabilities and subsequent induced particle loss. We examine the number of resonances, their location, and dependence on particle energy for some stellarator designs.

Two existing particle-in-cell gyrokinetic codes, GEM for the core region and XGC for the edge region, have been successfully coupled with a spatial coupling scheme at the interface in a toroidal geometry. A mapping technique is developed for transferring data between GEM's structured and XGC's unstructured meshes. Two examples of coupled simulations are presented to demonstrate the coupling scheme. The optimization of GEM for graphics processing unit is also presented.

Global gyrokinetic simulations with self-consistent coupling of neoclassical and turbulent dynamics show that turbulence can significantly affect plasma self-driven mean current generation in tokamaks. The current amplitude, profile and associated phase space structures can all be modified. Turbulence can significantly reduce the current generation in the collisionless regime, generate current profile corrugation near the rational magnetic surface and nonlocally drive current in the linearly stable region-all these are expected to have a radical impact on broad tokamak physics. Both electron parallel acceleration and residual stress from turbulence play crucial roles in turbulence-induced current generation.

Plasma turbulence is considered one of the main mechanisms for driving anomalous thermal transport in magnetic confinement fusion devices. Based on first-principle model, gradient-driven gyrokinetic simulations have often been used to explain turbulence-driven transport in present fusion devices, and in fact, many present predictive codes are based on the assumption that turbulence is gradient-driven. However, using the electrostatic global particle-in-cell gyrokinetic tokamak simulation (GTS) code (Wang et al 2010 Phys. Plasmas 17 072511), we will show that while global gradient-driven gyrokinetic simulations provide decent agreement in ion thermal transport with a set of NBI-heated NSTX (Ono et al 2000 Nucl. Fusion 40 557) H-mode plasmas, they are not able to explain the observed electron thermal transport variation in a set of RF-heated L-mode plasmas, where a factor of 2 decrease in electron heat flux is observed after the cessation of the RF heating. Thus, identifying the regime of validity of the gradient-driven assumption is essential for first-principle gyrokinetic simulation. This understanding will help us to more confidently predict the confinement performance of ITER and future magnetic confinement devices.

Part of the promise of exascale computing and the next generation of scientific simulation codes is the ability to bring together time and spatial scales that have traditionally been treated separately. This enables creating complex coupled simulations and in situ analysis pipelines, encompassing such things as "whole device" fusion models or the simulation of cities from sewers to rooftops. Unfortunately, the HPC analysis tools that have been built up over the preceding decades are ill suited to the debugging and performance analysis of such computational ensembles. In this paper, we present a new vision for performance measurement and understanding of HPC codes, MonitoringAnalytics (MONA). MONA is designed to be a flexible, high performance monitoring infrastructure that can perform monitoring analysis in place or in transit by embedding analytics and characterization directly into the data stream, without relying upon delivering all monitoring information to a central database for post-processing. It addresses the trade-offs between the prohibitively expensive capture of all performance characteristics and not capturing enough to detect the features of interest. We demonstrate several uses of MONA; capturing and indexing multi-executable performance profiles to enable later processing, extraction of performance primitives to enable the generation of customizable benchmarks and performance skeletons, and extracting communication and application behaviors to enable better control and placement for the current and future runs of the science ensemble. Relevant performance information based on a system for MONA built from ADIOS and SOSflow technologies is provided for DOE science applications and leadership machines.