The HPE Slingshot interconnect is used on numerous supercomputers, including the top two supercomputers on the TOP500. Recently, HPE open-sourced the software stack for Slingshot introducing new opportunities for exploring alternative MPI implementations on HPE’s Cray supercomputers. This work investigates the performance implications of using Open MPI, as opposed to the traditionally bundled Cray MPICH, on systems equipped with Slingshot-11 interconnects. We focus our analysis on Adaptive Mesh Refinement (AMR) applications in this work, as they exhibit a wide variety of communication patterns, including dynamically changing communicating peers. Based on profiling and analysis of these AMR applications, we designed a targeted micro-benchmark to capture key communication patterns in AMR that can benefit from Open MPI on Slingshot-11 systems. We demonstrate that Open MPI can improve the overall execution time of AMR-based scientific applications by up to 11
Nonblocking communication in MPI significantly enhances high-performance computing by minimizing application time spent in MPI communication functions, enabling the overlap of computation and communication. Asynchronous programming models further enhance efficiency and adaptability in load balancing; however, the integration of MPI communication with these models remains insufficient. The MPI Continuations proposal, currently under discussion in the MPI Forum, aims to resolve the latter by introducing asynchronous completion for nonblocking communication. To ensure application developers can adopt this transformative feature, robust support in PMPI-based tools is essential. We have updated an early prototype to implement the latest discussion state from the MPI Forum. This shim library implementation of MPI Continuations is compatible with any MPI implementation. We describe potential pitfalls in implementing continuations, including a significant API issue that could lead to race conditions. We propose restrictions to prevent these issues and strengthen the robustness of the MPI Continuations proposal. Additionally, our analysis highlights the challenges PMPI-based tools will face with the new control flow, as MPI operations may now complete at nearly any point in a program. MPI progress threads and progress engines might become visible, requiring closer attention. Tools need to adapt by intercepting user-provided callback functions to monitor the completion of nonblocking communication effectively. Using the PMPI-based On-the-Fly Critical Path Tool used for performance analysis as an example, we demonstrate a clear path for integrating MPI Continuations into performance analysis tools, enhancing overall performance and adaptability in advanced computing applications.
Parallel programming models such as MPI and OpenSHMEM enable the use of large-scale distributed-memory computers in HPC. However, programmers often miss subtle rules regarding their APIs,such as properly synchronizing local memory accesses with communication and releasing acquired resources. Existing correctness tools aim to detect these issues automatically, but are typically model-specific. We propose the use of model-independent function annotations to avoid this dependency:Contracts allow the specification of generic pre- and postconditions at function declarations. We specify requirements that must be satisfied at each call site to avoid common MPI errors such as resource leaks and local data races. In contrast to traditional checkers, the transparent nature of contracts also allows for maintainability and extensibility of checks by the end user,as well as adapting the specific analyses to their use case. This paper presents a contract language and CoVer, an extensible static verifier to check the use of library-based parallel programming models. It applies data-flow analysis using the LLVM framework to verify these contract annotations. We compare detection accuracy against the static tools PARCOACH and MPI-Checker using RMARaceBench and MPI-BugBench, and compile-time overhead based on the mini-apps LULESH, miniVite, and the PRK Stencil Kernel. CoVer improved the detection accuracy by covering a wide variety of issues, while maintaining comparable overhead.
Sessions is one of the major features introduced in the MPI-4 standard. It offers an alternative to the traditional world communicator model by allowing applications to construct communicators from process sets, thereby eliminating the dependency on MPI COMM WORLD. The Sessions model was proposed as a more scalable solution for exascale systems, where MPI COMM WORLD was viewed as a potential scalability bottleneck. However, supporting Sessions is a significant challenge for established codebases like MPICH due to the deep integration of the world model in traditional MPI implementations. Although MPICH added support for the MPI-4 standard upon its release, it still internally relied on a global world communicator. This approach enabled applications written using the Sessions model to function, but it did not fulfill the full design intent of Sessions, which meant to decouple MPI from MPI COMM WORLD. We describe MPICH's effort to support "true" MPI Sessions, including a major internal refactoring. We describe the architectural changes required to support true Sessions and evaluate the resulting implementation's scalability. Our results demonstrate that true Sessions can offer significant scalability benefits by adopting explicit hierarchical designs.
Modern multi-node systems necessitate parallel programming models (PPMs) like mpi to facilitate execution and communication among multiple processing elements. These single program, multiple data (SPMD) PPMs offer features such as RMA or accelerator support. However, SPMD program tools, such as those for correctness checks or performance optimization, are typically developed for specific PPMs or rely on tool-internal abstractions. To overcome this limitation, the spmd ir was introduced as an intermediate representation (IR) within a multilayer program representation and realized as a dialect in mlir (LLVM). This work extends the spmd ir by incorporating, among others, capabilities for RMA and related completion mechanisms. These enhancements increase compatibility with mpi and shmem, while newly integrating support for nvshmem. By leveraging traits in mlir, spmd ir provides an extensible approach for implementing SPMD program analysis. The applicability of the spmd ir is demonstrated through the use case of static local data race detection. It is implemented in a generalized fashion, covering not only RMA but also non-blocking communication in general, and is independent of specific API calls, offering increased extensibility. Using a comprehensive set of micro-benchmark suites and proxy apps, the spmd ir is evaluated against both static and dynamic tools. Overall, the spmd ir verification distinguishes itself with extensive PPM support and high detection accuracy. Notably, it is the first tool capable of detecting data races across shmem, nvshmem, and their hybrid combinations (with MPI).
The Message Passing Interface (MPI) standard has long been a cornerstone of parallel computing, enabling multi-threaded processes to communicate effectively. However, integrating multi-threading with MPI is not as straightforward as it might seem. MPI must function optimally in a multi-threaded environment, which requires robust thread support. The description provided by MPI about how multiple application threads may interact with MPI forms a contract for guarantees that applications can rely on and that implementations must provide, and vice versa. We find that the definitions regarding multi-threaded behavior provided by MPI are imprecise, which leads to differing interpretations and potential over- or undercommitment of computational resources in order to provide their perceived semantics. We provide an analysis of relevant parts of the MPI standard dealing with multi-threading and provide proposals for definitions of concurrency, conflicts, and race conditions, which we hope will clarify the guarantees that are part of the contract between MPI multi-threaded applications and implementations.
The increasing disparity between computing capabilities and communication bandwidth has become a major bottleneck in High Performance Computing (HPC) applications. To address this challenge, we introduce a framework that leverages early data compression for communication data within the Open MPI library with the use of userfaultfd (uffd) for efficient write detection. By integrating the high-speed LZ4 compression algorithm, the proposed framework minimizes communication overhead by reducing the size of data transmitted among processes while hiding compression overhead behind either pack or communication overhead. Applying our uffd framework onto Livermore Unstructured Lagrangian Explicit Shock Hydrodynamics (LULESH) highlights the potential of the framework in reducing data communication volumes and overall communication latency, paving the way for improved performance in HPC environments.
Message Passing Interface (MPI) has been a well-established technology in the domain of distributed high-performance computing for several decades. However, one of its greatest drawbacks is a rather ancient pure-C interface. It lacks many useful features of modern languages (namely C++), like basic type-checking or support for generic code design. In this paper, we propose a novel abstraction for MPI, which we implemented as an extension of the C++ Noarr library. It follows Noarr paradigms (first-class layout and traversal abstraction) and offers layout-agnostic design of MPI applications. We also implemented a layout-agnostic distributed GEMM kernel as a case study to demonstrate the usability and syntax of the proposed abstraction. We show that the abstraction achieves performance comparable to the state-of-the-art MPI C++ bindings while allowing for a more flexible design of distributed applications.
The increasing complexity of HPC architectures and the growing adoption of irregular scientific algorithms demand efficient support for asynchronous, multithreaded communication. This is most pronounced with Asynchronous Many-Task (AMT) systems. Such communication was not a consideration during the initial MPI design. The MPI community has recently introduced several extensions to address these new requirements. This work evaluates two such extensions, the Virtual Communication Interface (VCI) and the Continuation extensions, in the context of an established AMT runtime, HPX. We begin by using an MPI-level microbenchmark, modeled from HPX's low-level communication mechanism, to measure the peak performance potential of these extensions. We then integrate them into HPX to evaluate their effectiveness in real-world scenarios. Our results show that while these extensions can enhance performance compared to standard MPI, areas for improvement remain. The current continuation proposal limits the maximum multithreaded message rate achievable in the multi-VCI setting. Furthermore, the recommended one-VCI-per-thread mode proves ineffective in real-world scenarios due to the attentiveness problem. These findings underscore the importance of improving intra-VCI threading efficiency to achieve scalable multithreaded communication and fully realize the benefits of recent MPI extensions.
Since the C++ bindings were deleted in 2008, the Message Passing Interface (MPI) community has recently revived efforts in building high-level modern C++ interfaces. Such interfaces are either built to serve specific scientific application needs (with limited coverage to the underlying MPI functionality), or as an exercise in general-purpose programming model building, with the hope that bespoke interfaces can be broadly adopted to construct a variety of distributed-memory scientific applications. However, with the advent of modern C++-based heterogeneous programming models, GPUs and widespread Machine Learning (ML) usage in contemporary scientific computing, the role of prospective community-standardized high-level C++ interfaces to MPI is evolving. The success of such an interface clearly will depend on providing robust abstractions and features adhering to the generic programming principles that underpin the C++ programming language, without compromising on either performance or portability, the core principles upon which MPI was founded. However, there is a tension between idiomatic C++ handling of types and lifetimes and MPI's loose interpretation of object lifetimes/ownership and insistence on maintaining global states. Instead of proposing "yet another" high-level C++ interface to MPI, overlooking or providing partial solutions to work around the key issues concerning the dissonance between MPI semantics and idiomatic C++, this paper focuses on the three fundamental aspects of a high-level interface: type system, object lifetimes, and communication buffers, while also identifying inconsistencies in the MPI specification. Presumptive solutions can be unrefined, and we hope the broader MPI and C++ communities will engage with us in productive exchange of ideas and concerns.
Accelerators have become a corner stone of parallel computing, ranging from scientific computing to artificial intelligence. At the application level, accelerators are controlled by submitting work into a stream, from which the work is executed by the hardware. Vendor-specific communication libraries such as NCCL and RCCL have integrated support for submitting communication operations onto a stream to enable ordering of communication and work on streams. It is safe to assume that stream-based computing will remain relevant for the foreseeable future. MPI has yet to catch up to this reality and prior proposals involved extensions of MPI that would incur significant additions to the API. In this work, we explore alternatives that involve only minor additions to the standard to enable the integration of MPI operations with compute stream. Our additions include i) associating streams with communication objects, ii) blocking streams until completion, and iii) synchronizing streams while progressing MPI operations. Our API is agnostic of the type of stream, reuses existing communication procedures and semantics, and enables integration with graph capturing. We provide a proof-of-concept implementation and show that stream integration of MPI operations can be beneficial.
MPI's low-level interface is prone to errors, leading to bugs that can remain dormant for years. MPI correctness tools can aid in writing correct code but lack a standardized benchmark for comparison. This makes it difficult for users to choose the best tool and difficult for developers to gauge their tools' effectiveness. MPI correctness benchmarks, MPI-CorrBench, the MPI Bugs Initiative, and RMARaceBench have emerged to address this problem. However, comparability is hindered by having separate benchmarks, and none fully reflects real-world MPI usage patterns. Hence, we present MPI-BugBench, a unified MPI correctness benchmark replacing previous efforts. It addresses the shortcomings of its predecessors by providing a single, standardized test harness for assessing tools and incorporates a broader range of real-world MPI usage scenarios. MPI-BugBench is available at https://git- ce.rwth-aachen.de/hpcpublic/mpi-bugbench.
The evolution of parallel computing architectures presents new challenges for developing efficient parallelized codes. The emergence of heterogeneous systems has given rise to multiple programming models, each requiring careful adaptation to maximize performance. In this context, we propose reevaluating memory layout designs for computational tasks within larger nodes by comparing various architectures. To gain insight into the performance discrepancies between shared memory and shared-address space settings, we systematically measure the bandwidth between cores and sockets using different methodologies. Our findings reveal significant differences in performance, suggesting that MPI running inside UNIX processes may not fully utilize its intranode bandwidth potential. In light of our work in the MPC thread-based MPI runtime, which can leverage shared memory to achieve higher performance due to its optimized layout, we advocate for enabling the use of shared memory within the MPI standard.
To effectively utilize modern HPC clusters, inter-node communication and related single program, multiple data (SPMD) parallel programming models such as mpi are inevitable. Current tools and compilers that employ analyses of SPMD models often have the limitation of only supporting one model or implementing the necessary abstraction internally. This makes the analysis and effort for the abstraction neither reusable nor the tool extensible to other models without extensive changes to the tool itself. This work proposes an spmd ir as part of a multi-layer program representation and accompanying compiler passes to explicitly express the results of abstraction and multi-value analysis. The spmd ir makes the executing processes of operations explicit and differentiates between static and dynamic cases. It is implemented as a prototype in the mlir llvm infrastructure and is comprised of the spmd dialect and two compiler passes, supporting mpi, shmem, and nccl, including hybrid cases. To evaluate the proposed IR, verification of collective communication was chosen as a use case. For that, this work reimplements and extends parcoach's static approach on the spmd ir and assesses it by an expanded micro-benchmark suite in mpi, shmem, and nccl. Achieving similar detection accuracy, the evaluation shows that the spmd ir's level of abstraction is strong enough for parcoach's analyses and generic enough for increased extensibility. The prototype also constitutes the first collectives verification of shmem, nccl, and their combinations (with mpi).
This paper explores the use of LLVM IR function and parameter attributes to enhance compiler optimizations for code that uses MPI. As MPI is usually used as a dynamically linked library, the compiler is not able to automatically infer certain function attributes like nofree, which signals that no memory is deallocated in this function. Therefore, we implemented an LLVM compiler pass that annotates the used MPI functions with suitable attributes when compiling the user application. We manually derived applicable attributes based on the semantics described in the MPI standard, so that this approach is applicable to all MPI implementations. We showcase different cases where this additional annotations impact the code generated by the compiler for the MiniApps from the Exascale Proxy Applications Project. The addition of MPI function annotations allows for a variety of compiler optimizations like reducing unnecessary memory accesses, optimizing register usage, and streamlining control flow. The code of our annotation pass is available on GitHub: https:// github.com/AdrSchm/mpi- attributes-pass.
3-dimensional Fast Fourier Transform (3D FFT) parallel computations are an important part of many scientific calculations. For example, 3D FFT is a critical component of molecular dynamics codes when they compute long range electrostatic computations. Parallel distributed 3D FFT computations involve redistributing intermediate data, which constitutes a substantial portion of overall execution time of these operations. There are two primary methods for handling this communication phase: explicitly packing and unpacking data, or using Message Passing Interface (MPI) derived datatypes. Derived datatypes have several advantages in that they are easy to work with and don't require explicit memory pack and unpack operations. As such we propose enhancements for derived datatypes in MPI specifically for 3D FFT calculations which improves upon state of the art methods [8] by using MPI_Type_create_subarray to support arbitrary storage orders. Our method reduces the performance issues associated with MPI derived datatype solutions and benefits from avoiding strided-memory operations in FFT execution. Results show that we can speedup even strong scaled 3DFFT by 1.17X to 1.44X using our method over previous state of the art methods.
The computational power of High-Performance Computing (HPC) systems increases continuously and rapidly. Data-intensive applications are designed to leverage the high computational capacity of HPC resources and typically generate a large amount of data for traditional post-processing data analytics. However, the HPC systems' in-/output (IO) subsystem develops relatively slowly, and the storage capacity is limited. This could lead to limited actual performance and scientific discovery. In-situ techniques are a partial remedy to these problems by reducing or avoiding the data flow through the IO subsystem to/from the storage. However, in current practice, asynchronous in-situ techniques with static resource management often allocate separate computing resources for executing in-situ task(s), which remain idle if no in-situ work is at hand. In the present work, we target improving the efficiency of computing resource usage by launching and releasing necessary additional computing resources for in-situ task(s). Our approach is based on extensions for MPI Sessions that enable the required dynamic resource management. In this paper, we propose a basic and an advanced in-situ techniques with dynamic resource management enabled by MPI Sessions, their implementations on two real-world use cases, and a critical analysis of the experimental results.
GPU-enhanced architectures are now dominant in HPC systems, but message-passing communication involving GPUs with MPI has proven to be both complex and expensive, motivating new approaches that lower such costs. We compare and contrast stream/graph-, kernel-triggered, and GPU-initiated MPI communication abstractions, whose principal purpose is to enhance the performance of communication when GPU kernels create or consume data for transfer through MPI operations. Researchers and practitioners have proposed multiple potential APIs for GPU-involved communication that span various GPU architectures and approaches, including MPI-4 partitioned point-to-point communication, stream communicators, and explicit MPI stream/queue objects. Designs breaking backward compatibility with MPI are duly noted. Some of these strengthen or weaken the semantics of MPI operations. A key contribution of this paper is to promote community convergence toward common abstractions for GPU-involved communication by highlighting the common and differing goals and contributions of existing abstractions. We describe the design space in which these abstractions reside, their implicit or explicit use of stream and other non-MPI abstractions, their relationship to partitioned and persistent operations, and discuss their potential for added performance, how usable these abstractions are, and where functional and/or semantic gaps exist. Finally, we provide a taxonomy for these abstractions, including disambiguation of similar semantic terms, and consider directions for future standardization in MPI-5.
This study focuses on gaining insights into the usage of the Message-Passing Interface (MPI) in a large set of High-Performance Computing (HPC) codes by analyzing MPI function calls and their argument usage patterns. Previous work has focused on analyzing MPI feature usage by statically matching function calls. However, this approach does not reveal common argument-specific call patterns or cross-interactions between MPI functions. In particular, MPI exposes its internal data structures using handles, and users pass these handles to MPI constructor functions, e.g., to create custom communicators. Tracking the relevant MPI arguments of these constructors and cross-referencing them with other MPI calls in a target code can reveal common user interactions. These insights can be used to optimize, e.g., datatype construction at a library level or to extend MPI correctness debugging tools to verify correct construction of these data structures. To that end, we statically analyze codes to extract MPI function calls and their arguments, cross-reference them with other MPI calls, and provide statistics on common argument patterns and cross-use of MPI functions. We believe that these insights can guide further development within the MPI community to ultimately benefit users.
Performance benchmarks are an integral part of the development and evaluation of parallel algorithms, both in distributed applications as well as MPI implementations themselves. The initial step of the benchmark process is to obtain a common timestamp to mark the start of an operation across all involved processes, and the state-of-the-art in many applications and widely used MPI benchmark suites is the use of MPI barriers. In this paper, we show that the synchronization in space provided by an MPI_Barrier is insufficient for proper benchmark results of parallel distributed algorithms, using MPI collective operations as examples. The resulting lack of a global start timestamp for an operation leads to skewed results, with a significant impact of the used barrier algorithm. In order to mitigate these issues, we propose and discuss the implementation of MPIX_Harmonize, which extends the synchronization in space provided by MPI_Barrier with a time synchronization to guarantee a common starting timestamp across all involved processes. By replacing the use of MPI_Barrier with MPIX_Harmonize, benchmark implementors can eliminate skews resulting from barrier algorithms and achieve stable performance benchmark results. We will show that the proper time synchronization can have significant impact on the benchmark results for various implementations of MPI_Allreduce, MPI_Reduce, and MPI_Bcast.
