The volume of data generated by exascale simulations requires scalable tools for analysis and visualization. Due to the relatively low I/O bandwidth of modern HPC systems, it is crucial to work as close as possible with simulated data via in situ approaches. In situ visualization provides insights into simulation data and, with the help of additional interactive analysis tools, can support the scientific discovery process at an early stage. Such in situ visualization tools need to be hardware-independent given the ever-increasing hardware diversity of modern supercomputers. We present a new in situ 3D vector field visualization algorithm for particle-in-cell (PIC) simulations and performance evaluation of the solution developed at large-scale. We create a solution in a hardware-agnostic approach to support high throughput and interactive in situ processing on leadership class computing systems. To that end, we demonstrate performance portability on Summit's and the Frontier's pre-exascale testbed at the Oak Ridge Leadership Computing Facility.

We present EZ, a novel current deposition algorithm for particle-in-cell (PIC) simulations. EZ calculates the current density on the electromagnetic grid due to macro-particle motion within a time step by solving the continuity equation of electrodynamics. Being a charge conserving hybridization of Esirkepov's method and ZigZag, we refer to it as “EZ” as shorthand for “Esirkepov meets ZigZag”. Simulations of a warm, relativistic plasma with PIConGPU show that EZ achieves the same level of charge conservation as the commonly used method by Esirkepov, yet reaches higher performance for macro-particle assignment-functions up to third-order. In addition to a detailed description of the functioning of EZ, reasons for the expected and observed performance increase are given, and guidelines for its implementation aiming at highest performance on GPUs are provided.

This paper assesses and reports the experience of ten teams working to port, validate, and benchmark several High Performance Computing applications on a novel GPU-accelerated Arm testbed system. The testbed consists of eight NVIDIA Arm HPC Developer Kit systems, each one equipped with a server-class Arm CPU from Ampere Computing and two data center GPUs from NVIDIA Corp. The systems are connected together using InfiniBand interconnect. The selected applications and mini-apps are written using several programming languages and use multiple accelerator-based programming models for GPUs such as CUDA, OpenACC, and OpenMP offloading. Working on application porting requires a robust and easy-to-access programming environment, including a variety of compilers and optimized scientific libraries. The goal of this work is to evaluate platform readiness and assess the effort required from developers to deploy well-established scientific workloads on current and future generation Arm-based GPU-accelerated HPC systems. The reported case studies demonstrate that the current level of maturity and diversity of software and tools is already adequate for large-scale production deployments.

It is common in the HPC community that the achieved performance with just CPUs is limited for many computational cases. The EuroHPC pre-exascale and the coming exascale systems are mainly focused on accelerators, and some of the largest upcoming supercomputers such as LUMI and Frontier will be powered by AMD Instinct™ accelerators. However, these new systems create many challenges for developers who are not familiar with the new ecosystem or with the required programming models that can be used to program for heterogeneous architectures. In this paper, we present some of the more well-known programming models to program for current and future GPU systems. We then measure the performance of each approach using a benchmark and a mini-app, test with various compilers, and tune the codes where necessary. Finally, we compare the performance, where possible, between the NVIDIA Volta (V100), Ampere (A100) GPUs, and the AMD MI100 GPU.

HPC systems employ a growing variety of compute accelerators with different architectures and from different vendors. Large scientific applications are required to run efficiently across these systems but need to retain a single code-base in order to not stifle development. Directive-based offloading programming models set out to provide the required portability, but, to existing codes, they themselves represent yet another API to port to. Here, we present our approach of porting the GPU-accelerated particle-in-cell code PIConGPU to OpenACC and OpenMP target by adding two new backends to its existing C++-template metaprogramming-based offloading abstraction layer alpaka and avoiding other modifications to the application code. We introduce our approach in the face of conflicts between requirements and available features in the standards as well as practical hurdles posed by immature compiler support.

Due to the recent announcement of the Frontier supercomputer, many scientific application developers are working to make their applications compatible with AMD (CPU-GPU) architectures, which means moving away from the traditional CPU and NVIDIA-GPU systems. Due to the current limitations of profiling tools for AMD GPUs, this shift leaves a void in how to measure application performance on AMD GPUs. In this article, we design an instruction roofline model for AMD GPUs using AMD’s ROCProfiler and a benchmarking tool, BabelStream (the HIP implementation), as a way to measure an application’s performance in instructions and memory transactions on new AMD hardware. Specifically, we create instruction roofline models for a case study scientific application, PIConGPU, an open source particle-in-cell simulations application used for plasma and laser-plasma physics on the NVIDIA V100, AMD Radeon Instinct MI60, and AMD Instinct MI100 GPUs. When looking at the performance of multiple kernels of interest in PIConGPU we find that although the AMD MI100 GPU achieves a similar, or better, execution time compared to the NVIDIA V100 GPU, profiling tool differences make comparing performance of these two architectures hard. When looking at execution time, GIPS, and instruction intensity, the AMD MI60 achieves the worst performance out of the three GPUs used in this work.

The paper concerns the elimination of a set of variables from a system of linear inequalities. We employ the widely used Fourier–Motzkin elimination method extended with the Chernikov rules. A straightforward implementation of the algorithm results in extensive enumeration during the most computationally demanding stage. We propose a new way of checking Chernikov rules using bit pattern trees as an accelerating data structure to avoid extensive enumeration. The bit pattern tree is a data structure based on k-d tree used to accelerate the double description method. First we describe an adaptation of that approach to check the second Chernikov rule in Fourier–Motzkin elimination. We also propose a new algorithm that employs bit pattern trees to accelerate both Chernikov rules. Presented results of computational evaluation prove competitiveness of the proposed algorithms.

PIConGPU, Particle In Cell on GPUs, is an open source simulations framework for plasma and laser-plasma physics used to develop advanced particle accelerators for radiation therapy of cancer, high energy physics and photon science. While PIConGPU has been optimized for at least 5 years to run well on NVIDIA GPU-based clusters (1), there has been limited exploration by the development team of potential scalability bottlenecks using recently updated and new tools including NVIDIA’s NVProf tool and the brand-new NVIDIA NSight Suite (Systems and Compute) tools. PIConGPU is a highly optimized application that runs production jobs at scale on a system Oak Ridge Leadership Facility’s (OLCF) Summit supercomputer (using the full machine at 4600 nodes; at 98% of GPU utilization on all ∼28000 NVIDIA Volta GPUs). PIConGPU has been selected as one of the the eight applications for OLCF’s coveted Center for Accelerated Application Readiness (CAAR) program aimed at the facility’s Frontier supercomputer (OLCF’s first exascale system to launch in 2021), to partner with our vendors (primary vendors: AMD and Cray/HPE) ensuring that Frontier will be able to perform large-scale science when it opens to users in 2022 (2). To this effect, performance engineers on the PIConGPU team wanted to dive deep into the application to understand at the finest granularity, which portions of the code could be further optimized to exploit the hardware on Summit at it’s maximum potential and also to elucidate which key kernels should be tracked and optimized for the CAAR effort to port this code to Frontier. Any bottlenecks that are observed via performance profiling on Summit are likely to also impact scalability on the Frontier-dev system and the Frontier Early Access (EA) system. Additionally, the engineers wanted to take a closer look at the newest NVIDIA profiling tools which allows us to identify the most useful features on these tools and will provide an opportunity to compare it to new AMD and Cray’s performance analysis tool releases and provide feedback to our vendor partners on what features are most important and mission critical for CAAR efforts. Figure 1. A timeline view of PIConGPU, captured using NSight Systems The primary goal of this report is to focus on the evaluation of PIConGPU’s most time-intensive kernels using NVProf and NSight Suite. Three kernels, Current Deposition (also known as Compute Current), Particle Push (Move and Mark), and Shift Particles are known to be some of the most time-consuming kernels in PIConGPU. The Current

In recent years, the prospects of performing fundamental and applied studies at the next-generation high-intensity laser facilities have greatly stimulated the interest in performing large-scale simulations of laser interaction with matter with the account for quantum electrodynamics (QED) processes such as emission of high energy photons and decay of such photons into electron-positron pairs. These processes can be modelled via probabilistic routines that include frequent computation of synchrotron functions and can constitute significant computational demands within accordingly extended Particle-in-Cell (QED-PIC) algorithms. In this regard, the optimization of these routines is of great interest. In this paper, we propose and describe two modifications. First, we derive a more accurate upper-bound estimate for the rate of QED events and use it to arrange local sub-stepping of the global time step in a significantly more efficient way than done previously. Second, we present a new high-performance implementation of synchrotron functions. Our optimizations made it possible to speed up the computations by a factor of up to 13.7 depending on the problem. Our implementation is integrated into the PICADOR and Hi-Chi codes, the latter of which is distributed publicly (https://github.com/hi-chi/pyHiChi).

State-of-the-art numerical simulations of laser plasma by means of the Particle-in-Cell method are often extremely computationally intensive. Therefore there is a growing need for development of approaches for efficient utilization of resources of modern supercomputers. In this paper, we address the problem of a substantially non-uniform and dynamically varying distribution of macroparticles in a computational area in simulating quantum electrodynamic (QED) cascades. We propose and evaluate a load balancing scheme for shared memory systems, which allows subdividing individual cells of the computational domain into work portions with subsequent dynamic distribution of these portions between OpenMP threads. Computational experiments on 1D, 2D, and 3D QED simulations show that the proposed scheme outperforms the previously developed standard and custom schemes in the PICADOR code by 2.1 to 10 times when employing several Intel Cascade Lake CPUs.

The cascaded production and dynamics of electron-positron plasma in ultimately focused laser fields of extreme intensity are studied by three-dimensional particle-in-cell simulations with the account of the relevant processes of quantum electrodynamics (QED). We show that, if the laser facility provides a total power above 20 PW, it is possible to trigger not only a QED cascade but also pinching in the produced electron-positron plasma. The plasma self-compression in this case leads to an abrupt rise of the peak density and magnetic (electric) field up to at least 10^{28}cm^{-3} and 1/20 (1/40) of the Schwinger field, respectively. Determining the actual limits and physics of this process might require quantum treatment beyond the used standard semiclassical approach. The proposed setup can thus provide extreme conditions for probing and exploring fundamental physics of the matter and vacuum.

This paper concerns checking adjacency in the double description method for constructing a generating system of a polyhedral cone. We propose two new variations of the graph test. Theoretical estimates and experimental results show that the new variations are usually superior to the original algorithm in terms of speed.

Particle-in-cell plasma simulation is an important area of computational physics. The particle-in-cell method naturally allows parallel processing on distributed and shared memory. In this paper we address the problem of load balancing on multicore systems. While being well-studied for many traditional applications of the method, it is a relevant problem for the emerging area of particle-in-cell simulations with account for effects of quantum electrodynamics. Such simulations typically produce highly non-uniform, and sometimes volatile, particle distributions, which could require custom load balancing schemes. In this paper we present a computational evaluation of several standard and custom load balancing schemes for the particle-in-cell method on a high-end system with 96 cores on shared memory. We use a test problem with static non-uniform particle distribution and a real problem with account for quantum electrodynamics effects, which produce dynamically changing highly non-uniform distributions of particles and workload. For these problems the custom schemes result in increase of scaling efficiency by up to 20% compared to the standard OpenMP schemes.