𝚂𝚝𝚘𝚌𝚑𝚊𝚜𝚝𝚒𝚌𝙶𝚆 is a code for computing accurate Quasi-Particle (QP) energies of molecules and material systems in the GW approximation. 𝚂𝚝𝚘𝚌𝚑𝚊𝚜𝚝𝚒𝚌𝙶𝚆 utilizes the stochastic Resolution of the Identity (sROI) technique to enable a massively-parallel implementation with computational costs that scale semi-linearly with system size, allowing the method to access systems with tens of thousands of electrons. We introduce a new implementation, 𝚂𝚝𝚘𝚌𝚑𝚊𝚜𝚝𝚒𝚌𝙶𝚆-𝙶𝙿𝚄, for which the main bottleneck steps have been ported to GPUs and which gives substantial performance improvements over previous versions of the code. We showcase the new code by computing band gaps of hydrogenated silicon clusters (Si_xH_y) containing up to 10001 atoms and 35144 electrons, and we obtain individual QP energies with a statistical precision of better than ±0.03 eV with times-to-solution on the order of minutes.

The numerical solution of the Kadanoff-Baym nonlinear integro-differential equations, which yields the non-equilibrium Green's functions (NEGFs) of quantum many-body systems, poses significant computational challenges due to its high computational complexity. In this work, we present efficient implementations of a numerical method for solving these equations on distributed-memory architectures, including many-core CPUs and multi-GPU systems. For CPU-based platforms, we adopt a hybrid MPI/OpenMP programming model to exploit both inter-node and intra-node parallelism. On GPU-accelerated systems, we implement the method using two distinct approaches: MPI/OpenACC and MPI/CUDA FORTRAN. Several optimization strategies are employed to enhance GPU performance, including techniques to maximize computational resource utilization and minimize the overhead associated with kernel launches and memory management. Although OpenACC is easy to use, CUDA FORTRAN provides more advanced features for configuring and managing multiple levels of concurrency, while also simplifying memory allocation and data movement between host and device. This flexibility translates into significant performance improvements. We compare the performance of the three implementations and demonstrate that the GPU-based approaches achieve substantial speedups over CPU-based implementations. Furthermore, both CPU and GPU versions exhibit excellent strong and weak scaling, confirming the scalability and efficiency of our approach for large-scale NEGF computations.

StochasticGW is a code for computing accurate quasi-particle (QP) energies of molecules and material systems in the GW approximation. StochasticGW utilizes the stochastic Resolution of the Identity (sROI) technique to enable a massively parallel implementation with computational costs that scale semilinearly with system size, allowing the method to access systems with tens of thousands of electrons. We introduce a new implementation, StochasticGW-GPU, for which the main bottleneck steps have been ported to GPUs and give substantial performance improvements over previous versions of the code. We showcase the new code by computing band gaps of hydrogenated silicon clusters (SixHy) containing up to 10,001 atoms and 35,144 electrons, and we obtain individual QP energies with a statistical precision of better than ±0.03 eV with times-to-solution of less than 1 h.

The Particle-In-Cell (PIC) algorithm coupled with binary collision modules is a widely applicable method to simulate plasmas over a broad range of regimes (from the collisionless kinetic regime to the collisional regime). While several popular PIC codes implement binary collision modules, their performance on GPUs can be constrained by the default parallelization strategy, which assigns one GPU thread per simulation cell. This approach can underutilize GPU resources for simulations with many macroparticles per cell, and relatively few cells per GPU. To address this limitation, we propose an alternative parallelization strategy that instead GPU distributes threads based on independent pairs of colliding particles. Our proposed strategy shows a speed improvement of up to similar to 4x for cases with relatively few cells per GPU and a similar performance otherwise.

Bioinformatics workloads differ significantly from traditional scientific computing and AI workloads because they consist primarily of integer-only operations and string comparisons rather than floating-point operations. The underlying algorithms usually have low arithmetic intensity, irregular memory access patterns, and non-deterministic workloads. Local Assembly is an essential step in large-scale genome assembly software and is typically implemented using de Bruijn graphs. This paper examines the performance, portability, and productivity of a local assembly GPU kernel from a metagenome assembly pipeline implemented using hash table data structures on NVIDIA, AMD, and Intel GPUs. We focus on the challenges of achieving portability while maintaining performance for a complex bioinformatics GPU kernel that relies on hardware-specific optimizations. In this paper, we evaluate the local assembly kernel's performance and portability across different GPU architectures, identify performance bottlenecks, and propose modifications in existing tools and methods for performance modeling and analysis of integer-heavy bioinformatics application kernels.

MILC-Dslash is a benchmark that is derived from the MILC code which simulates lattice-gauge theory on a four-dimensional hypercube. This paper outlines a gradual progression in increasing the granularity of parallelism in the MILC-Dslash kernel using the SYCL programming model, transitioning from a simple to a fully parallel implementation. We explore the impact of various parallel strategies on the MILC-Dslash performance on an NVIDIA A100 GPU. This investigation encompasses different work-item index orders, work-group sizes, and memory access patterns that arise from these strategies. Examples of components intertwined with the parallel strategies include atomic memory operations, shared variables, divergent instructions, synchronization barrier, scenarios with and without dependencies between iterations, as well as versions with and without using the SYCL complex library (SyclCPLX) and the SYCLomatic tool. The best parallel strategy is twice as fast as the simplest strategy and shows a 10% improvement over the QUDA baseline, thanks to enhanced parallelism and the use of work-group local memory. This, along with other findings — such as optimizing GPU resource utilization even at the expense of concurrency, prioritizing the use of work-item indexing methods that favor more local-ized memory access patterns, and maximizing both the number of active work-items per warp and the sequence of successive active work-items — could provide valuable guidance for researchers and developers seeking to optimize parallel computing applications.

The US Department of Energy Office of Science and the National Nuclear Security Administration initiated the Exascale Computing Project (ECP) in 2016 to prepare mission-relevant applications and scientific software for the delivery of the exascale computers starting in 2023. The ECP currently supports 24 efforts directed at specific applications and six supporting co-design projects. These 24 application projects contain 62 application codes that are implemented in three high-level languages—C, C++, and Fortran—and use 22 combinations of graphical processing unit programming models. The most common implementation language is C++, which is used in 53 different application codes. The most common programming models across ECP applications are CUDA and Kokkos, which are employed in 15 and 14 applications, respectively. This article provides a survey of the programming languages and models used in the ECP applications codebase that will be used to achieve performance on the future exascale hardware platforms.

includes characterizing which modules have been ported to multi-accelerator nodes, initial performance analyses, the status of software integration, and a current vision of successes, obstacles, and next steps. As such this report contains not only an accurate snapshot of each sub-project's current status, but also represents an unprecedentedly broad account of experiences porting large scientific applications to next-generation HPC architectures.

Computing the GW quasiparticle bandstructure and Bethe-Salpeter Equation (BSE) absorption spectra for materials with spin-orbit coupling has commonly been done by treating GW corrections and spin-orbit coupling as separate perturbations to density-functional theory. However, accurate treatment of materials with strong spin-orbit coupling often requires a fully relativistic approach using spinor wavefunctions in the Kohn-Sham equation and GW/BSE. Such calculations have only recently become available, in particular for the BSE. We have implemented this approach in the plane-wave pseudopotential GW/BSE code BerkeleyGW, which is highly parallelized and widely used in the electronic-structure community. We present reference results for quasiparticle bandstructures and optical absorption spectra of solids with different strengths of spin-orbit coupling, including Si, Ge, GaAs, GaSb, CdSe, Au, and Bi_2Se_3. The calculated quasiparticle band gaps of these systems are found to agree with experiment to within a few tens of meV. The absorption spectrum of GaSb calculated with the fully-relativistic GW-BSE captures the large spin-orbit splitting of peaks in the spectrum. For Bi_2Se_3, we find a drastic change in the low-energy bandstructure compared to that of DFT, with the fully-relativistic treatment of the GW approximation correctly capturing the parabolic nature of the valence and conduction bands after including off-diagonal self-energy matrix elements. We present the detailed methodology, approach to spatial symmetries for spinors, comparison against other codes, and performance compared to spinless GW/BSE calculations and perturbative approaches to SOC. This work aims to spur further development of spinor GW/BSE methodology in excited-state research software.

Computing the GW quasiparticle bandstructure and Bethe-Salpeter Equation (BSE) absorption spectra for materials with spin-orbit coupling has commonly been done by treating GW corrections and spin-orbit coupling as separate perturbations to density-functional theory. However, accurate treatment of materials with strong spin-orbit coupling (such as many topological materials of recent interest, and thermoelectrics) often requires a fully relativistic approach using spinor wavefunctions in the Kohn-Sham equation and GW/BSE. Such calculations have only recently become available, in particular for the BSE. We have implemented this approach in the plane-wave pseudopotential GW/BSE code BerkeleyGW, which is highly parallelized and widely used in the electronic-structure community. We present reference results for quasiparticle bandstructures and optical absorption spectra of solids with different strengths of spin-orbit coupling, including Si, Ge, GaAs, GaSb, CdSe, Au, and Bi2Se3. The calculated quasiparticle band gaps of these systems are found to agree with experiment to within a few tens of meV. SOC splittings are found to be generally in better agreement with experiment, including quasiparticle corrections to band energies. The absorption spectrum of GaAs is not significantly impacted by the inclusion of spin-orbit coupling due to its relatively small value (0.2 eV) in the Λ direction, while the absorption spectrum of GaSb calculated with the fully-relativistic GW -BSE captures the large spin-orbit splitting of peaks in the spectrum. For the prototypical topological insulator Bi2Se3, we find a drastic change in the low-energy bandstructure compared to that of DFT, with the fully-relativistic treatment of the GW approximation correctly capturing the parabolic nature of the valence and conduction bands after including off-diagonal self-energy matrix elements. We present the detailed methodology, approach to spatial symmetries for spinors, comparison against other codes, and performance compared to spinless GW/BSE calculations and perturbative approaches to SOC. This work aims to spur further development of spinor GW/BSE methodology in excited-state research software, and enables more accurate and detailed exploration of electronic and optical properties of materials containing elements with large atomic number. a bbarker6@ucmerced.edu

Excitons localized around point defects in semiconductors are promising candidates for long-lived and photon-addressable qubits. However, their microscopic origin is difficult to characterize due to the computational complexity of studying large systems with defects. Here we study the quasiparticle and optical absorption spectrum of the divacancy defect in 3C-SiC, a prototypical defect for quantum information applications, by means of large-scale GW and GW plus Bethe-Salpeter equation calculations. Despite the presence of localized unoccupied quasiparticle states in the gap, we find that the low-energy excitonic states are made primarily of transitions from occupied defect states to continuum conduction states from SiC, especially from the X point of the Brillouin zone. The mixed character of defect states and bulk states of these low-energy exciton states is in contrast with the NV- center in diamond and the divacancy in 4H-SiC, where the deep defect levels are well separated from bulk states. Our calculations provide a quantitative prediction of the defect quasiparticle energy levels and a physical understanding of the zero-phonon absorption. They highlight the important role of frontier conduction bands in the optical properties and formation of low-energy excitons in 3C-SiC divacancy.

In this paper, we show that OpenMP 4.5 based implementation of TestSNAP, a proxy-app for the Spectral Neighbor Analysis Potential (SNAP) in LAMMPS, can be ported across the NVIDIA, Intel, and AMD GPUs. Roofline analysis is employed to assess the performance of TestSNAP on each of the architectures. The main contributions of this paper are two-fold: 1) Provide OpenMP as a viable option for application portability across multiple GPU architectures, and 2) provide a methodology based on the roofline analysis to determine the performance portability of OpenMP implementations on the target architectures. The GPUs used for this work are Intel Gen9, AMD Radeon Instinct MI60, and NVIDIA Volta V100.

Parallel and distributed computing systems have been seeing rapid growth in the number of processing cores as progress on single-core performance has stagnated. The larger the system, the greater the challenge for application scalability and system stability. Aiming at addressing both challenges in the context of distributed metaheuristic optimization algorithms, in this work, we propose a scalable and fault-tolerant peer-to-peer communication algorithm tailored for population-based metaheuristics. In the algorithm, messages exchanging are carried out by multiple threads asynchronously in background and the minimal algorithm's overhead can be entirely hidden by overlapping communication with computation. Results from controlled benchmarks corroborate the efficiency of the algorithm and also hint that thread oversubscription can further improve scalability thanks to the high degree of idleness of communication operations. The proposed algorithm contributes to the important yet not sufficiently explored performance aspects of distributed metaheuristics.

In this work, we evaluate the utility of adding two new ceilings to the roofline model related to PCI-E and effective Network bandwidths to provide insights into how communication impacts the performance of large-scale parallel applications. The roofline performance analysis is based on two benchmark problems: scalar dense matrix addition and dense symmetric eigen-problem with complex matrix. The experiments were conducted on the NERSC Cori supercomputer at Lawrence Berkeley National Laboratory, both on the CPU-only and CPU+GPU compute nodes. The study reveals the value of incorporating these two new ceilings into the roofline model, in addition to the existing memory bandwidth and compute ceilings, in order to ease the identification of performance bottlenecks and to better guide the performance optimization process, particularly in the limit of diminishing strong and weak scaling. We highlight the importance of comparing obtained application roofline points to customized ceilings for the communication and data-access patterns present. In this way, the effects of both throughput and latency can be captured in the model.

includes characterizing which modules have been ported to multi-accelerator nodes, initial performance analyses, the status of software integration, and a current vision of successes, obstacles, and next steps. As such this report contains not only an accurate snapshot of each sub-project's current status, but also represents an unprecedentedly broad account of experiences porting large scientific applications to next-generation HPC architectures.

Maintaining computational load balance is important to the performant behavior of codes which operate under a distributed computing model. This is especially true for GPU architectures, which can suffer from memory oversubscription if improperly load balanced. We present enhancements to traditional load balancing approaches and explicitly target GPU architectures, exploring the resulting performance. A key component of our enhancements is the introduction of several GPU-amenable strategies for assessing compute work. These strategies are implemented and benchmarked to find the most optimal data collection methodology for in-situ assessment of GPU compute work. For the fully kinetic particle-in-cell code WarpX, which supports MPI+CUDA parallelism, we investigate the performance of the improved dynamic load balancing via a strong scaling-based performance model and show that, for a laserion acceleration test problem run with up to 6144 GPUs on Summit, the enhanced dynamic load balancing achieves from 62%-74% (88% when running on 6 GPUs) of the theoretically predicted maximum speedup; for the 96-GPU case, we find that dynamic load balancing improves performance relative to baselines without load balancing (3.8x speedup) and with static load balancing (1.2x speedup). Our results provide important insights into dynamic load balancing and performance assessment, and are particularly relevant in the context of distributed memory applications ran on GPUs.

Metagenomic workflows involve studying uncultured microorganisms directly from the environment. These environmental samples when processed by modern sequencing machines yield large and complex datasets that exceed the capabilities of metagenomic software. The increasing sizes and complexities of datasets make a strong case for exascale-capable metagenome assemblers. However, the underlying algorithmic motifs are not well suited for GPUs. This poses a challenge since the majority of next-generation supercomputers will rely primarily on GPUs for computation. In this paper we present the first of its kind GPU-accelerated implementation of the local assembly approach that is an integral part of a widely used large-scale metagenome assembler, MetaHipMer. Local assembly uses algorithms that induce random memory accesses and non-deterministic workloads, which make GPU offloading a challenging task. Our GPU implementation outperforms the CPU version by about 7x and boosts the performance of MetaHipMer by 42% when running on 64 Summit nodes.

Traditional scientific simulations have for quite some time, dominated the workloads of high-performance computing infrastructures across the world. With recent advancement in data generation capabilities of systems biology equipment, a rise in bioinformatics workloads has been observed. Bioinformatics applications deploy algorithmic motifs that use unique memory access patterns and rely heavily on integer-only computations. These applications place unique requirements on modern programming environments as well as GPU accelerators which are becoming an integral part of next generation of supercomputers. In this paper, we evaluate the performance and code portability of a core bioinformatics kernel that uses dynamic programming method for performing DNA and protein sequence alignments in several bioinformatics software pipelines. Our study evaluates the performance of a GPU accelerated sequence alignment algorithm across multiple vendor GPUs and programming models. We use a highly optimized adaptation of sequence alignment kernel and find the most productive way of porting it across multiple vendor GPUs and then assess its performance portability using Pennycook's method. Methods used in this paper and the insights drawn from those can be extended to a large number of integer-heavy scientific kernels and may aid in future accelerator design and design of programming model requirements.

This Exascale Computing Project (ECP) Milestone Report summarizes the status of all 30 ECP Applications Development (AD) sub-projects at the end of FY19. In August and September of 2019, a comprehensive assessment of AD projects was conducted jointly by the ECP leadership and a team of external subject matter experts. Reviews took place in person over five days-two at the National Renewable Energy Laboratory and three at Argonne National Laboratory and the University of Chicago. The review committees were tasked with evaluating each sub-project's progress in porting their code(s) to current multi-GPU architectures considered precursors to planned exascale machines. This includes characterizing which modules have been ported to multi-accelerator nodes, initial performance analyses, the status of software integration, and a current vision of successes, obstacles, and next steps. As such this report contains not only an accurate snapshot of each sub-project's current status, but also represents an unprecedentedly broad account of experiences porting large scientific applications to next-generation HPC architectures.

The exascale race is at an end with the announcement of the Aurora and Frontier machines. This next generation of supercomputers utilize diverse hardware architectures to achieve their compute performance, providing an added onus on the performance portability of applications. An expanding fragmentation of programming models would provide a compounding optimization challenge were it not for the evolution of performance-portable frameworks, providing unified models for mapping abstract hierarchies of parallelism to diverse architectures. A solution to this challenge is the evolution of performance-portable frameworks, providing unified models for mapping abstract hierarchies of parallelism to diverse architectures. Kokkos is one such performance portable programming model for C++ applications, providing back-end implementations for each major HPC platform. Even with a performance portable framework, restructuring algorithms to expose higher degrees of parallelism is non-trivial. The Spectral Neighbor Analysis Potential (SNAP) is a machine-learned inter-atomic potential utilized in cutting-edge molecular dynamics simulations. Previous implementations of the SNAP calculation showed a downward trend in their performance relative to peak on newer-generation CPUs and low performance on GPUs. In this paper we describe the restructuring and optimization of SNAP as implemented in the Kokkos CUDA backend of the LAMMPS molecular dynamics package, benchmarked on NVIDIA GPUs. We identify novel patterns of hierarchical parallelism, facilitating a minimization of memory access overheads and pushing the implementation into a compute-saturated regime. Our implementation via Kokkos enables recompile-and-run efficiency on upcoming architectures. We find a $\sim$22x time-to-solution improvement relative to an existing implementation as measured on an NVIDIA Tesla V100-16GB for an important benchmark.