HiCCL (Hierarchical Collective Communication Library) addresses the growing complexity and diversity in high-performance network architectures. As GPU systems have envolved into networks of GPUs with different multilevel communication hierarchies, optimizing each collective function for a specific system has become a challenging task. Consequently, many collective libraries struggle to adapt to different hardware and software, especially across systems from different vendors. HiCCL's library design decouples the collective communication logic from network-specific optimizations through a compositional API. The communication logic is composed using multicast, reduction, and fence primitives, which are then factorized for a specified network hieararchy using only point-to-point operations within a level. Finally, striping and pipelining optimizations applied as specified for streamlining the execution. Performance evaluation of HiCCL across four different machines$\unicode{x2014}$two with Nvidia GPUs, one with AMD GPUs, and one with Intel GPUs$\unicode{x2014}$demonstrates an average 17$\times$ higher throughput than the collectives of highly specialized GPU-aware MPI implementations, and competitive throughput with those of vendor-specific libraries (NCCL, RCCL, and OneCCL), while providing portability across all four machines.

DeltaAI is a new NSF funded resource supporting researchers nationwide via the ACCESS and NAIRR Pilot programs. DeltaAI builds upon the success of Delta and supplants Delta as the most powerful GPU resource available via the ACCESS program. DeltaAI leverages the Delta environment sharing storage and infrastructure resources while offering a more scalable platform capable of scaling jobs up to the full node count of the system. DeltaAI offers the latest NVIDIA H100 GPUs as part of the innovative Grace Hopper architecture. This paper describes the full architecture of DeltaAI and our experiences benchmarking applications and supporting research teams during the acceptance process for DeltaAI.

MIRGE is a computational approach for scientific computing based on NumPy-like array computation, but using lazy evaluation to recast computation as data-flow graphs, where nodes represent immutable, multi-dimensional arrays. Evaluation of an array expression is deferred until its value is needed, at which point a pipeline is invoked that transforms high-level array expressions into lower-level intermediate representations (IR) and finally into executable code, through a multi-stage process. Domain-specific transformations, such as metadata-driven optimizations, GPU-parallelization strategies, and loop fusion techniques, improve performance and memory efficiency. MIRGE employs "array contexts" to abstract the interface between array expressions and heterogeneous execution environments (for example, lazy evaluation via OpenCL, or eager evaluation via NumPy or CuPy). The framework thus enables performance portability as well as separation of concerns between application logic, low-level implementation, and optimizations. By enabling scientific expressivity while facilitating performance tuning, MIRGE offers a robust, extensible platform for both computational research and scientific application development. This paper provides an overview of MIRGE. We further describe an application of MIRGE called MIRGE-Com, for supersonic combusting flows in a discontinuous Galerkin finite-element setting. We demonstrate its capabilities as a solver and highlight its performance characteristics on large-scale GPU hardware.

Graphics Processing Units (GPUs) have become a de facto solution for accelerating high-performance computing (HPC) applications. Understanding their memory error behavior is an essential step toward achieving efficient and reliable HPC systems. In this work, we present a large-scale cross-supercomputer study to characterize GPU memory reliability, covering three supercomputers - Delta, Polaris, and Perlmutter - all equipped with NVIDIA A100 GPUs. We examine error logs spanning 67.77 million GPU device-hours across 10,693 GPUs. We compare error rates and mean-time-between-errors (MTBE) and highlight both shared and distinct error characteristics among these three systems. Based on these observations and analyses, we discuss the implications and lessons learned, focusing on the reliable operation of supercomputers, the choice of checkpointing interval, and the comparison of reliability characteristics with those of previous-generation GPUs. Our characterization study provides valuable insights into fault-tolerant HPC system design and operation, enabling more efficient execution of HPC applications.

This manual describes the use of the Portable, Extensible Toolkit for Scientific Computation (PETSc) and the Toolkit for Advanced Optimization (TAO) for the numerical solution of partial differential equations (PDEs) and related problems on high-performance computers.PETSc/TAO is a suite of data structures and routines that provide the building blocks for implementing large-scale application codes on parallel (and serial) computers.PETSc uses the MPI standard for all distributed memory communication.PETSc/TAO includes a large suite of parallel linear solvers, nonlinear solvers, time integrators, and optimizers that may be used in application codes written in Fortran, C, C++, and Python (via petsc4py; see Getting Started ).The library is organized hierarchically, enabling users to employ the abstraction level most appropriate for a particular problem.By using techniques of object-oriented programming, PETSc provides enormous flexibility for users.PETSc is a sophisticated set of software tools; it initially has a steeper learning curve than packages such as MATLAB or a simple subroutine library.In particular, for individuals without some experience programming in C, C++, Python, or Fortran and experience using a debugger such as gdb or lldb, it may require a significant amount of time to take full advantage of the features that enable efficient software use.However, the power of the PETSc design and the algorithms it incorporates makes the efficient implementation of many application codes simpler than "rolling them" yourself.• For many tasks, a package such as MATLAB is often the best tool; PETSc is not intended for the classes of problems for which effective MATLAB code can be written.• Several packages (listed on https://petsc.org/),built on PETSc, may satisfy your needs without requiring directly using PETSc.We recommend reviewing these packages' functionality before starting to code directly with PETSc.• PETSc can be used to provide a "MPI parallel linear solver" in an otherwise sequential or OpenMP parallel code.This approach can provide modest improvements in the application time by utilizing modest numbers of MPI processes.See PCMPI for details on how to utilize the PETSc MPI linear solver server.Since PETSc is under continued development, small changes in usage and calling sequences of routines will occur.PETSc has been supported for twenty-five years; see mailing list information on our website for information on contacting support.PETSc/TAO Users Manual, Release 3.21.0DM: Interfacing Between Solvers and Models/Discretizations details how a user's models and discretizations can easily be interfaced with the solvers by using the DM construct.Additional Information describes a variety of useful information, including profiling, the options database, viewers, error handling, and some details of PETSc design.Visual Studio Code, Eclipse, Emacs, and Vim users may find their development environment's options for searching in the source code are useful for exploring the PETSc source code.Details of this feature are provided in Developer Environments.Note to Fortran Programmers: In most of the manual, the examples and calling sequences are given for the C/C++ family of programming languages.However, Fortran programmers can use all of the functionality of PETSc from Fortran, with only minor differences in the user interface.PETSc for Fortran Users provides a discussion of the differences between using PETSc from Fortran and C, as well as several complete Fortran examples. Note to Python Programmers:To program with PETSc in Python, you need to enable Python bindings (i.e.petsc4py) with the configure option --with-petsc4py=1.See the PETSc installation guide for more details.$ mpiexec -n 8 ./petsc_program_namepetsc_optionsPETSc also provides a script that automatically uses the correct mpiexec for your configuration.$ $PETSC_DIR/lib/petsc/bin/petscmpiexec -n 8 ./petsc_program_namepetsc_options Certain options are supported by all PETSc programs.We list a few particularly useful ones below; a complete list can be obtained by running any PETSc program with the option -help.• -log_view -summarize the program's performance (see Profiling)• -fp_trap -stop on floating-point exceptions; for example divide by zero • -malloc_dump -enable memory tracing; dump list of unfreed memory at conclusion of the run, see Detecting Memory Allocation Problems and Memory Usage, MatCreate(MPI_Comm comm,Mat *A); VecCreate(MPI_Comm comm,Vec *x); KSPCreate(MPI_Comm comm,KSP *ksp); Chapter 2. The Solvers in PETSc/TAO PETSc/TAO Users Manual, Release 3.21.0 DMNETWORK -Creating vectors for networksSee Networks for discussion of creating vectors with DMNETWORK. Common vector functions and operationsOne can examine (print out) a vector with the command VecView(Vec x,PetscViewer v);To print the vector to the screen, one can use the viewer PETSC_VIEWER_STDOUT_WORLD, which ensures that parallel vectors are printed correctly to stdout.To display the vector in an X-window, one can use the default X-windows viewer PETSC_VIEWER_DRAW_WORLD, or one can create a viewer with the routine PetscViewerDrawOpen().A variety of viewers are discussed further in Viewers: Looking at PETSc Objects.To create a new vector of the same format and parallel layout as an existing vector, useVecDuplicate(Vec old,Vec *new);To create several new vectors of the same format as an existing vector, useVecDuplicateVecs(Vec old,PetscInt n,Vec **new);This routine creates an array of pointers to vectors.The two routines are useful because they allow one to write library code that does not depend on the particular format of the vectors being used.Instead, the subroutines can automatically create work vectors based on the specified existing vector.As discussed in Duplicating Multiple Vectors, the Fortran interface for VecDuplicateVecs() differs slightly.When a vector is no longer needed, it should be destroyed with the command VecDestroy(Vec *x);To destroy an array of vectors, use the command VecDestroyVecs(PetscInt n,Vec **vecs);Note that the Fortran interface for VecDestroyVecs() differs slightly, as described in Duplicating Multiple Vectors.It is also possible to create vectors that use an array the user provides rather than having PETSc internally allocate the array space.Such vectors can be created with the routines such as

Modern high-performance computing systems have multiple GPUs and network interface cards (NICs) per node. The resulting network architectures have multilevel hierarchies of subnetworks with different interconnect and software technologies. These systems offer multiple vendor-provided communication capabilities and library implementations (IPC, MPI, NCCL, RCCL, OneCCL) with APIs providing varying levels of performance across the different levels. Understanding this performance is currently difficult because of the wide range of architectures and programming models (CUDA, HIP, OneAPI). We present CommBench, a library with cross-system portability and a high-level API that enables developers to easily build microbenchmarks relevant to their use cases and gain insight into the performance (bandwidth & latency) of multiple implementation libraries on different networks. We demonstrate CommBench with three sets of microbenchmarks that profile the performance of six systems. Our experimental results reveal the effect of multiple NICs on optimizing the bandwidth across nodes and also present the performance characteristics of four available communication libraries within and across nodes of NVIDIA, AMD, and Intel GPU networks.

A burst buffer is a common method to bridge the performance gap between the I/O needs of modern supercomputing applications and the performance of the shared file system on large-scale supercomputers. However, existing I/O sharing methods require resource isolation, offline profiling, or repeated execution that significantly limit the utilization and applicability of these systems. Here we present ThemisIO, a policy-driven I/O sharing framework for a remote-shared burst buffer: a dedicated group of I/O nodes, each with a local storage device. ThemisIO preserves high utilization by implementing opportunity fairness so that it can reallocate unused I/O resources to other applications. ThemisIO accurately and efficiently allocates I/O cycles among applications, purely based on real-time I/O behavior without requiring user-supplied information or offline-profiled application characteristics. ThemisIO supports a variety of fair sharing policies, such as user-fair, size-fair, as well as composite policies, e.g., group-then-user-fair. All these features are enabled by its statistical token design. ThemisIO can alter the execution order of incoming I/O requests based on assigned tokens to precisely balance I/O cycles between applications via time slicing, thereby enforcing processing isolation. Experiments using I/O benchmarks show that ThemisIO sustains 13.5--13.7% higher I/O throughput and 19.5--40.4% lower performance variation than existing algorithms. For real applications, ThemisIO significantly reduces the slowdown by 59.1--99.8% caused by I/O interference.

This manual describes the use of the Portable, Extensible Toolkit for Scientific Computation (PETSc) and the Toolkit for Advanced Optimization (TAO) for the numerical solution of partial differential equations and related problems on high-performance computers. PETSc/TAO is a suite of data structures and routines that provide the building blocks for the implementation of large-scale application codes on parallel (and serial) computers. PETSc uses the MPI standard for all distributed memory communication.

the mechanisms needed within parallel application codes, such as parallel matrix and vector assembly routines. The library is organized hierarchically, enabling users to employ the level of abstraction that is most appropriate for a particular problem. By using techniques of object-oriented programming, PETSc provides enormous flexibility for users. PETSc is a sophisticated set of software tools; as such, for some users it initially has a much steeper learning curve than a simple subroutine library. In particular, for individuals without some computer science background, experience programming in C, C++, python, or Fortran and experience using a debugger such as gdb or dbx, it may require a significant amount of time to take full advantage of the features that enable efficient software use. However, the power of the PETSc design and the algorithms it incorporates may make the efficient implementation of many application codes simpler than "rolling them" yourself. • For many tasks a package such as MATLAB is often the best tool; PETSc is not intended for the classes of problems for which effective MATLAB code can be written. • There are several packages (listed on http://www.mcs.anl.gov/petsc), built on PETSc, that may satisfy your needs without requiring directly using PETSc. We recommend reviewing these packages functionality before using PETSc. • PETSc should not be used to attempt to provide a "parallel linear solver" in an otherwise sequential code. Certainly all parts of a previously sequential code need not be parallelized but the matrix generation portion must be parallelized to expect any kind of reasonable performance. Do not expect to generate your matrix sequentially and then "use PETSc" to solve the linear system in parallel. Since PETSc is under continued development, small changes in usage and calling sequences of routines will occur. PETSc is supported; see http://www.mcs.anl.gov/petsc/miscellaneous/mailinglists.html for information on contacting support. A list of publications and web sites that feature work involving PETSc may be found at http://www.mcs.anl.gov/petsc/publications. We welcome any reports of corrections for this document at petsc-maint@mcs.anl.gov.

Krylov methods are a key way of solving large sparse linear systems of equations but suffer from poor strong scalability on distributed memory machines. This is due to high synchronization costs from large numbers of collective communication calls alongside a low computational workload. Enlarged Krylov methods address this issue by decreasing the total iterations to convergence, an artifact of splitting the initial residual and resulting in operations on block vectors. In this article, we present a performance study of an enlarged Krylov method, Enlarged Conjugate Gradients (ECG), noting the impact of block vectors on parallel performance at scale. Most notably, we observe the increased overhead of point-to-point communication as a result of denser messages in the sparse matrix-block vector multiplication kernel. Additionally, we present models to analyze expected performance of ECG, as well as motivate design decisions. Most importantly, we introduce a new point-to-point communication approach based on node-aware communication techniques that increases efficiency of the method at scale.

Supercomputer architectures are trending toward higher computational throughput due to the inclusion of heterogeneous compute nodes. These multi-GPU nodes increase on-node computational efficiency, while also increasing the amount of data to be communicated and the number of potential data flow paths. In this work, we characterize the performance of irregular point-to-point communication with MPI on heterogeneous compute environments through performance modeling, demonstrating the limitations of standard communication strategies for both device-aware and staging-through-host communication techniques. Presented models suggest staging communicated data through host processes then using node-aware communication strategies for high inter-node message counts. Notably, the models also predict that node-aware communication utilizing all available CPU cores to communicate inter-node data leads to the most performant strategy when communicating with a high number of nodes. Model validation is provided via a case study of irregular point-to-point communication patterns in distributed sparse matrix-vector products. Importantly, we include a discussion on the implications model predictions have on communication strategy design for emerging supercomputer architectures.

In 2014, NASA released a report outlining a future state for aerospace computational fluid dynamics (CFD), the CFD Vision 2030 Study (the Study).1 Developed by experts from industry, government, and academia, the Study provided a forecast of CFD capabilities required for turbulent, transitional, and reacting flow simulations across a broad Mach number regime. In addition, the Study provided an aspirational role for future CFD as part of a routine, efficient, and physics-based aerospace design and development process. This future role of CFD was summarized in the Study as follows: “A single engineer/scientist must be able to conceive, create, analyze, and interpret a large ensemble of related simulations in a time-critical period (e.g., 24 hours), without individually managing each simulation, to a prespecified level of accuracy.”

Despite the critical role that range queries play in analysis and visualization for HPC applications, there has been no comprehensive analysis of indices that are designed to accelerate range queries and the extent to which they are viable in HPC. In this paper we present the first such evaluation, examining 20 open-source C and C++ libraries that support range queries. Contributions of this paper include answering the following questions: which of the implementations are viable in HPC, how do these libraries compare in terms of build time, query time, memory usage, and scalability, what are other trade-offs between these implementations, is there a single overall best solution, and when does a brute force solution offer the best performance? We also share key insights learned during this process that can assist both HPC application scientists and spatial index developers. While we find that there is no single best solution, three libraries, Boost, CGAL and R-tree, offer some of the best performance, scalability, memory overheads, and support for different mesh types. We find several areas where the spatial indices could be substantially improved: better performance when there are a large number of query matches, reduced memory overheads, and better support for GPUs or other accelerators.

High-performance computing scientists are producing unprecedented volumes of data that take a long time to load for analysis. However, many analyses only require loading in the data containing particular features of interest and scientists have many approaches for identifying these features. Therefore, if scientists store information (descriptive metadata) about these identified features, then for subsequent analyses they can use this information to only read in the data containing these features. This can greatly reduce the amount of data that scientists have to read in, thereby accelerating analysis. Despite the potential benefits of descriptive metadata management, no prior work has created a descriptive metadata system that can help scientists working with a wide range of applications and analyses to restrict their reads to data containing features of interest. In this article, we present EMPRESS, the first such solution. EMPRESS offers all of the features needed to help accelerate discovery: It can accelerate analysis by up to 300 ×, supports a wide range of applications and analyses, is high-performing, is highly scalable, and requires minimal storage space. In addition, EMPRESS offers features required for a production-oriented system: scalable metadata consistency techniques, flexible system configurations, fault tolerance as a service, and portability.

In 2014, the CFD Vision 2030 Report proposed a desired status for aerospace CFD by 2030 that included several challenge problems and a Roadmap describing how to reach this status. Since then, the Report has been used to identify research topics and support funded activities. Further, the AIAA CFD Vision 2030 Integration Committee has been established to further progress toward this vision. A detailed review of the progress made on the milestones identified on the Roadmap was completed in 2020 by this committee and was used to assess the technology readiness level (TRL) of the different milestones. An interpretation of the TRL scale based on publications and level of technology usage was developed and used for assessment consistency. With this information, the Roadmap has been adjusted to reflect the present status of key CFD technology items. Emerging and additional key technologies have been incorporated into the Roadmap. With these modifications, an updated Roadmap is provided that reflects present status of technology development and steps necessary to attain the CFD Vision 2030.

While past information technology (IT) advances have transformed society, future advances hold even greater promise. For example, we have only just begun to reap the changes from artificial intelligence (AI), especially machine learning (ML). Underlying IT's impact are the dramatic improvements in computer hardware, which deliver performance that unlock new capabilities. For example, recent successes in AI/ML required the synergy of improved algorithms and hardware architectures (e.g., general-purpose graphics processing units). However, unlike in the 20th Century and early 2000s, tomorrow's performance aspirations must be achieved without continued semiconductor scaling formerly provided by Moore's Law and Dennard Scaling. How will one deliver the next 100x improvement in capability at similar or less cost to enable great value? Can we make the next AI leap without 100x better hardware? This whitepaper argues for a multipronged effort to develop new computing approaches beyond Moore's Law to advance the foundation that computing provides to US industry, education, medicine, science, and government. This impact extends far beyond the IT industry itself, as IT is now central for providing value across society, for example in semi-autonomous vehicles, tele-education, health wearables, viral analysis, and efficient administration. Herein we draw upon considerable visioning work by CRA's Computing Community Consortium (CCC) and the IEEE Rebooting Computing Initiative (IEEE RCI), enabled by thought leader input from industry, academia, and the US government.

The papers in this special section focus on performance portability for advanced architectures. This is a very timely topic as we continue our journey to exascale computers. The leader on the Top 500 list today is the Fujitsu Fugaku computer in Japan, using a modified ARM architecture and 512-bit scalable vector extension (SVE) instructions. In the USA, the National Energy Research Scientific Computing Center (NERSC) has just dedicated Perlmutter, named for the Berkeley Nobel Laureate Saul Perlmutter. Perlmutter, built by Cray/HPE, contains both CPU-only and CPU/GPU nodes. The CPUs are produced by AMD and the GPUs come from NVIDIA. Frontier at Oak Ridge National Laboratory and El Capitan at Lawrence Livermore National Laboratory will contain new AMD GPUs, and Aurora at Argonne National Laboratory will feature new GPUs from Intel. Los Alamos National Laboratory's next-generation machine Crossroads will use Intel CPUs with high-bandwidth memory and AVX-512 vector instructions. Optimizing code for three different GPUs is a daunting challenge for application programmers. Further, there are still many computers that do not feature GPUs, and most computational scientists would not be eager to totally abandon such machines.

The cost of data movement on parallel systems varies greatly with machine architecture, job partition, and nearby jobs. Performance models that accurately capture the cost of data movement provide a tool for analysis, allowing for communication bottlenecks to be pinpointed. Modern heterogeneous architectures yield increased variance in data movement as there are a number of viable paths for inter-GPU communication. In this paper, we present performance models for the various paths of inter-node communication on modern heterogeneous architectures, including the trade-off between GPUDirect communication and copying to CPUs. Furthermore, we present a novel optimization for inter-node communication based on these models, utilizing all available CPU cores per node. Finally, we show associated performance improvements for MPI collective operations.