The HPE Slingshot interconnect is used on numerous supercomputers, including the top two supercomputers on the TOP500. Recently, HPE open-sourced the software stack for Slingshot introducing new opportunities for exploring alternative MPI implementations on HPE’s Cray supercomputers. This work investigates the performance implications of using Open MPI, as opposed to the traditionally bundled Cray MPICH, on systems equipped with Slingshot-11 interconnects. We focus our analysis on Adaptive Mesh Refinement (AMR) applications in this work, as they exhibit a wide variety of communication patterns, including dynamically changing communicating peers. Based on profiling and analysis of these AMR applications, we designed a targeted micro-benchmark to capture key communication patterns in AMR that can benefit from Open MPI on Slingshot-11 systems. We demonstrate that Open MPI can improve the overall execution time of AMR-based scientific applications by up to 11
Nonblocking communication in MPI significantly enhances high-performance computing by minimizing application time spent in MPI communication functions, enabling the overlap of computation and communication. Asynchronous programming models further enhance efficiency and adaptability in load balancing; however, the integration of MPI communication with these models remains insufficient. The MPI Continuations proposal, currently under discussion in the MPI Forum, aims to resolve the latter by introducing asynchronous completion for nonblocking communication. To ensure application developers can adopt this transformative feature, robust support in PMPI-based tools is essential. We have updated an early prototype to implement the latest discussion state from the MPI Forum. This shim library implementation of MPI Continuations is compatible with any MPI implementation. We describe potential pitfalls in implementing continuations, including a significant API issue that could lead to race conditions. We propose restrictions to prevent these issues and strengthen the robustness of the MPI Continuations proposal. Additionally, our analysis highlights the challenges PMPI-based tools will face with the new control flow, as MPI operations may now complete at nearly any point in a program. MPI progress threads and progress engines might become visible, requiring closer attention. Tools need to adapt by intercepting user-provided callback functions to monitor the completion of nonblocking communication effectively. Using the PMPI-based On-the-Fly Critical Path Tool used for performance analysis as an example, we demonstrate a clear path for integrating MPI Continuations into performance analysis tools, enhancing overall performance and adaptability in advanced computing applications.
Parallel programming models such as MPI and OpenSHMEM enable the use of large-scale distributed-memory computers in HPC. However, programmers often miss subtle rules regarding their APIs,such as properly synchronizing local memory accesses with communication and releasing acquired resources. Existing correctness tools aim to detect these issues automatically, but are typically model-specific. We propose the use of model-independent function annotations to avoid this dependency:Contracts allow the specification of generic pre- and postconditions at function declarations. We specify requirements that must be satisfied at each call site to avoid common MPI errors such as resource leaks and local data races. In contrast to traditional checkers, the transparent nature of contracts also allows for maintainability and extensibility of checks by the end user,as well as adapting the specific analyses to their use case. This paper presents a contract language and CoVer, an extensible static verifier to check the use of library-based parallel programming models. It applies data-flow analysis using the LLVM framework to verify these contract annotations. We compare detection accuracy against the static tools PARCOACH and MPI-Checker using RMARaceBench and MPI-BugBench, and compile-time overhead based on the mini-apps LULESH, miniVite, and the PRK Stencil Kernel. CoVer improved the detection accuracy by covering a wide variety of issues, while maintaining comparable overhead.
Sessions is one of the major features introduced in the MPI-4 standard. It offers an alternative to the traditional world communicator model by allowing applications to construct communicators from process sets, thereby eliminating the dependency on MPI COMM WORLD. The Sessions model was proposed as a more scalable solution for exascale systems, where MPI COMM WORLD was viewed as a potential scalability bottleneck. However, supporting Sessions is a significant challenge for established codebases like MPICH due to the deep integration of the world model in traditional MPI implementations. Although MPICH added support for the MPI-4 standard upon its release, it still internally relied on a global world communicator. This approach enabled applications written using the Sessions model to function, but it did not fulfill the full design intent of Sessions, which meant to decouple MPI from MPI COMM WORLD. We describe MPICH's effort to support "true" MPI Sessions, including a major internal refactoring. We describe the architectural changes required to support true Sessions and evaluate the resulting implementation's scalability. Our results demonstrate that true Sessions can offer significant scalability benefits by adopting explicit hierarchical designs.
Modern multi-node systems necessitate parallel programming models (PPMs) like mpi to facilitate execution and communication among multiple processing elements. These single program, multiple data (SPMD) PPMs offer features such as RMA or accelerator support. However, SPMD program tools, such as those for correctness checks or performance optimization, are typically developed for specific PPMs or rely on tool-internal abstractions. To overcome this limitation, the spmd ir was introduced as an intermediate representation (IR) within a multilayer program representation and realized as a dialect in mlir (LLVM). This work extends the spmd ir by incorporating, among others, capabilities for RMA and related completion mechanisms. These enhancements increase compatibility with mpi and shmem, while newly integrating support for nvshmem. By leveraging traits in mlir, spmd ir provides an extensible approach for implementing SPMD program analysis. The applicability of the spmd ir is demonstrated through the use case of static local data race detection. It is implemented in a generalized fashion, covering not only RMA but also non-blocking communication in general, and is independent of specific API calls, offering increased extensibility. Using a comprehensive set of micro-benchmark suites and proxy apps, the spmd ir is evaluated against both static and dynamic tools. Overall, the spmd ir verification distinguishes itself with extensive PPM support and high detection accuracy. Notably, it is the first tool capable of detecting data races across shmem, nvshmem, and their hybrid combinations (with MPI).
