As hardware in high-performance systems evolves, it is critical that systems software adapts to fully leverage new capabilities. High-performance communication middleware plays a key role in abstracting hardware complexity, enabling optimized performance without requiring users to understand low-level network operations. This work investigates the integration of Cerio's Rockport Ethernet Fabric, a multi-path network, into the UCX communication framework. By extending UCX to support multi-path functionality, there can be more effective bandwidth utilization across multiple communication lanes. Through various benchmarks with MPI and application-level evaluation, it can be seen that multi-path support can provide best in class performance without introducing additional latency.

New High Performance Computing (HPC) node architectures such as superchips and Composable Disaggregated Infrastructure have recently emerged as potential directions for the future of supercomputing. However, with new architectures available at different costs, it can be difficult to forecast the correct design decision to maximize science performed per dollar. In this paper, we present a model designed to determine the performance of workloads on different node architectures with a fixed budget. We demonstrate how this model can be applied to narrow design decisions when using a variety of workloads including molecular dynamics applications, machine learning applications, a heat diffusion simulator, and an atomic reactor simulator. Using detailed traces of GPU operations, we justify the components of the model and determine if the model is accurate enough to have decision assisting capability. Specifically, we analyze the results of the model in relation to the GPU utilization and data movement of the workloads and discuss other factors to consider when choosing a node architecture for a datacenter. We find that the model can aid in deciding which node architectures to consider for a supercomputer deployment, using detailed tracing of a variety of representative applications and proxy applications for HPC to demonstrate the utility of the model.

The concept of GPU-aware multi-path communication and heterogeneous computing is pivotal in enhancing HPC cluster performance. Heterogeneous computing utilizes various computing units to boost distributed applications, while multi-path communication frameworks optimize data transfer and reduce latency. This study proposes a collaborative Allreduce collective communication framework that utilizes heterogeneous computing and multi-path communication for multi-GPU systems. Leveraging GPU parallelism and high memory bandwidth, our approach optimizes both data transfer and computing power of the host system. We provide a proof-of-concept implementation and evaluation on a single-node multi-GPU system, comparing its performance to NCCL, UCC, and other algorithms. Our results show that our framework can improve Allreduce performance for large message sizes by up to 1.45x compared to NCCL. Our approach can be extended to various topologies and systems, with more extensive evaluations planned for the future.

Parallel distributed applications running on large-scale high-performance computing systems depend on effective point-to-point and collective communication to meet performance goals. Beginning with version 4.0, the Message Passing Interface (MPI) introduced the partitioned communication API, providing tools for addressing communication bottlenecks raised by hybrid communication models. This API allows individual actors (CPU threads, GPU threads, etc.) to initiate communication on portions of complete buffers, enabling additional communication/computation overlap. Intuitively, the utility of partitioned communication could benefit from network-level support: If there are multiple paths between endpoints, an MPI-aware network could disperse partitions across these paths, avoiding the data serialization entailed by a dependency on a single path. The Cerio Rockport Ethernet Fabric has the ability to expose this capability to communication middleware. In this work we develop this capability to allow for user-level path selection for MPI partitioned communication and explore how this capability impacts point-to-point performance, collective design, and Allreduce efficiency in a Large Language Model task

Efficient synchronization of memory mapping information is increasingly important as systems evolve toward greater resource disaggregation and heterogeneity. When memory is exported between processes, establishing shared mapping often requires costly page table walk and updates, particularly in fault-driven models. To study these costs, we implement an XPMEM-inspired shared-memory driver and evaluate techniques to reduce mapping overhead. Our approach combines parallel batched on-demand pinning, bypassing unnecessary cache-policy lookups in PFN mapping, and dynamic re-registration to expand registered regions without tearing down existing mappings. In our evaluation, these optimizations reduce cold-start memory copy by up to 13.22x over XPMEM in multi-process workloads, with particular benefits for collective communication patterns and rapidly resizing buffers. While developed in a shared-memory context, the results highlight general strategies-avoiding redundant translation work, enabling parallel mapping operations, and preserving mapping state-that can inform the design of memory management in disaggregated systems, including GPU disaggregation and heterogeneous memory environments.

Optimizing GPU-to-GPU communication is a key challenge for improving performance in MPI-based HPC applications, especially when utilizing multiple communication paths. This paper presents a novel performance model for intra-node multi-path GPU communication within the MPI+UCX framework, aimed at determining the optimal configuration for distributing a single Point-to-Point (P2P) communication across multiple paths. By considering factors such as link bandwidth, pipeline overhead, and stream synchronization, the model identifies an efficient path distribution strategy, reducing communication overhead and increasing throughput. Through extensive experiments on various topologies, we demonstrate that our model accurately finds experimentally optimal configurations, achieving significant improvements in performance, with the average of less than 6% error in predicting the optimal configuration for very large messages.

Network traffic in HPC systems can impact application performance by inducing costly re-transmissions or consuming memory bandwidth. Emerging SmartNIC technologies present new opportunities for addressing these issues and optimizing network performance by providing a platform for the intelligent utilization of network resources through machine learning models of network traffic. However, SmartNICs also present challenges for deploying such models insofar as they offer relatively limited computational and memory resources, and these resources must be shared with other services. Based on an analysis of traffic data collected from eight scientific applications and proxies, we explore lightweight approaches to modeling network traffic using dynamic linear regression. Depending on the application, normalized root mean squared error for static regression may be less than 1 %, and dynamic regression can reduce this error by an order of magnitude. We further refine the dynamic approach by adding an additional classifier that categorizes predictions generated by the model as reliable or unreliable, showing that the technique can achieve good precision and recall. Finally, we evaluate the performance of dynamic regression and classification on NVIDIA BlueField-2 and BlueField-3 SmartNICs, demonstrating these computationally lightweight techniques are feasible on contemporary SmartNIC platforms.

Graphics Processing Units have become the dominant type of accelerators for high-performance computing and artificial intelligence. To support these systems, new communication libraries have emerged, such as NCCL, RCCL, and NVSHMEM, providing stream-based semantics and GPU-Initiated Communication. Some of the best performing communication libraries are unfortunately vendor-specific, and may use load-store semantics that have been traditionally underused in the application community. Moreover, the Message Passing Interface (MPI) has yet to define explicit GPU support mechanisms, making it difficult to deploy the message-passing communication model efficiently on GPU-based systems. However, MPI-4.0 introduced MPI Partitioned Point-to-Point communication, which facilitates hybrid-programming models. For example, Partitioned Communication is designed to allow GPUs to trigger data movement through a persistent intra- or inter-node channel. In this work, we extend MPI Partitioned to provide Intra-Kernel GPU-Initiated Communication and Partitioned Collectives, augmenting MPI with techniques used in vendor specific libraries. We evaluate our designs on a NVIDIA GH200 Grace Hopper Superchip testbed, to understand the benefits of GPU-Initiated communication on NVLink and InfiniBand networks. We assess the benefits at the application layer using a Jacobi solver and Partitioned Allreduce with Deep Learning Kernels.

MPI Partitioned communication is designed to allow parts of a message to be sent when ready while multipathing allows messages to travel simultaneously. This presents an opportunity to pair multipathing with partitioned communication to increase efficiency. This paper uses benchmarks to examine the impact of using hardware spraying with partitioned communication. The perceived bandwidth increases drastically from less than 1450 MB/s to over 5650 MB/s. With four or fewer partitions the overhead stays below 21, but with eight or more the overhead is in the hundreds for small messages. Finally, measuring the percentage of early communication shows that there are opportunities to send partitions early when messages stay below 4 MB.

Row-scale Composable Disaggregated Infrastructure (CDI) is a heterogeneous high performance computing (HPC) architecture that relocates the GPUs to a single chassis which CPU nodes can then request compute resources from. This is a distinctly different architecture from rack-scaled CDI as the GPUs are accessed over a network rather than existing in the same PCIe domain as the CPUs. Row-scale CDI expands the benefits and flexibility of rack-scaled CDI, while introducing new challenges. For example, with row-scale CDI, one must account for the effects of "slack", a latency in the CPU-to-GPU communication times due to network delays. This work seeks to assess potential challenges with row-scale CDI to determine which factors are most important to consider when deploying a CDI system. Our strong scaling application analyses reveal that there are two types of HPC workloads that may benefit from row-scale CDI; those that are CPU dominant and periodically call on the GPU to do highly parallel tasks and those that are GPU dominant and primarily rely on the CPU to coordinate work. We perform comparisons between the kernel and data transfer characteristics of each application to a slack proxy application which allowed for the development of a mathematical model to predict the performance penalty different applications can face as a result of slack. To illustrate this we profile two applications using our proposed method and find that they pessimistically would see a less than a 1% performance penalty above the effects of crossing the network in an environment which induced 100 mu s of slack, or a distance of 20 km at the speed of light in a fibre optic network cable. This demonstrates that both row-scale and cluster-scale CDI are viable technologies from an application performance perspective.

3-dimensional Fast Fourier Transform (3D FFT) parallel computations are an important part of many scientific calculations. For example, 3D FFT is a critical component of molecular dynamics codes when they compute long range electrostatic computations. Parallel distributed 3D FFT computations involve redistributing intermediate data, which constitutes a substantial portion of overall execution time of these operations. There are two primary methods for handling this communication phase: explicitly packing and unpacking data, or using Message Passing Interface (MPI) derived datatypes. Derived datatypes have several advantages in that they are easy to work with and don't require explicit memory pack and unpack operations. As such we propose enhancements for derived datatypes in MPI specifically for 3D FFT calculations which improves upon state of the art methods [8] by using MPI_Type_create_subarray to support arbitrary storage orders. Our method reduces the performance issues associated with MPI derived datatype solutions and benefits from avoiding strided-memory operations in FFT execution. Results show that we can speedup even strong scaled 3DFFT by 1.17X to 1.44X using our method over previous state of the art methods.

The emergence of Artificial Intelligence (AI) and Machine Learning (ML) has unleashed groundbreaking tools, expanding the boundaries of what’s achievable. Given the time, energy, and CO2 emissions associated with the rising use of resources for training AI/ML models, researchers and industry alike must optimize this process to reduce its associated financial, ecological, and social impacts. Large-scale AI/ML training requires the computational power of High-Performance Computing Clusters (HPCC) [1] . Underutilization and inefficiencies, resulting from codes’ inability to utilize allocated hardware simultaneously, leave unused resources consuming power [2] , [3] .

Disaggregation is a heterogeneous HPC architecture which relocates the GPUs to a separate chassis, thus leaving homogeneous CPU nodes which can request GPU resources on-demand. This has benefits in scheduling jobs to maximize resource utilization [1] and power savings by eliminating the number of idle devices which are “trapped” in active nodes [2] . Complexities arise from an application perspective as moving the GPUs off-node introduces a latency in the CPU-GPU communication times with messages now needing to traverse the network [3] . Disaggregation also allows for customizing the ratio of CPU and GPU resources to to meet an application’s demands.

MPI neighborhood collectives were introduced in the MPI-3.0 standard to support sparse communication patterns used by many applications. Simultaneously, GPU-Aware MPI communication has become a prominent part of modern systems. With the rise of AMD GPUs and their incorporation into upcoming exascale systems like Frontier, it has become essential to optimize communication libraries for AMD platforms. In this paper, we take advantage of the hardware and networking features of AMD GPUs to design efficient and scalable neighborhood collective operations: allgather and allgatherv. We evaluate the performance of the proposed design for Random Sparse Graph and Moore neighborhood micro-benchmarks as well as an SpMM kernel. The results show that we obtain up to 7.03x speedup for the Random Sparse Graph micro-benchmark, up to 3.82x for the Moore neighborhood micro-benchmark, and up to 2.29x speedup for the SpMM kernel.

Astra, deployed in 2018, was the first petascale supercomputer to utilize processors based on the ARM instruction set. The system was also the first under Sandia's Vanguard program which seeks to provide an evaluation vehicle for novel technologies that with refinement could be utilized in demanding, large-scale HPC environments. In addition to ARM, several other important first-of-a-kind developments were used in the machine, including new approaches to cooling the datacenter and machine. This article documents our experiences building a power measurement and control infrastructure for Astra. While this is often beyond the control of users today, the accurate measurement, cataloging, and evaluation of power, as our experiences show, is critical to the successful deployment of a large-scale platform. While such systems exist in part for other architectures, Astra required new development to support the novel Marvell ThunderX2 processor used in compute nodes. In addition to documenting the measurement of power during system bring up and for subsequent on-going routine use, we present results associated with controlling the power usage of the processor, an area which is becoming of progressively greater interest as data centers and supercomputing sites look to improve compute/energy efficiency and find additional sources for full system optimization.

Modern HPC systems require efficient hybrid programming model to utilize their hardware resources effectively. The Message Passing Interface (MPI) has accommodated next-generation hardware by providing new APIs such as the MPI Partitioned interface. This API provides a user with fine-grain communication without the overhead of traditional MPI point-to-point communication in multi-threaded workloads.To the best of our knowledge, we present the first work on detailed low-level design for an MPI Partitioned implementation. We guide readers through a method to map the MPI Partitioned interface to the InfiniBand Verbs API. Alongside implementation details, we also study the aggregation of user partitions and how we can efficiently send them over the network. We study a brute force approach and using the Partitioned LogGP (PLogGP) model to predict ideal aggregation. We observe that using the PLogGP model provides comparable performance without exhausting computing resources to search the entire solution space. The PLogGP design was further optimized by considering how the partition arrival pattern can be used to dynamically modify our aggregation scheme. We profiled our micro-benchmarks to provide analysis on how and why this additional optimization is beneficial to our results and how we can fine-tune this mechanism. Finally, we evaluated our PLogGP and Timer-based PLogGP designs with a commonly used communication pattern in HPC (communication sweep) to observe the impact when communicating with multiple processes in an application-like scenario at 1024 cores.

The Message Passing Interface (MPI) has been the dominant message passing solution for scientific computing for decades. MPI point‐to‐point communications are highly efficient mechanisms for process‐to‐process communication. However, MPI performance when processes utilize multiple threads is slowed by concurrency protections in the MPI library. MPI's current thread level interface imposes these overheads throughout the library when thread safety is needed. While much work has been done to reduce multithreading overheads in MPI, a solution is needed that reduces the number of messages exchanged in a threaded environment. Partitioned communication is included in the MPI 4.0 standard as an alternative that addresses the challenges of multithreaded communication in MPI today. Partitioned communication reduces overall message volume by creating a buffer‐sharing mechanism between threads such that they can indicate when portions of a communication buffer are available to be sent. Separation of the control and data planes in MPI is enabled by allowing persistent initialization and single occurrence message buffer matching from the indication that the data is ready to be sent. This enables the usage of underlying hardware primitives like triggered operations, where commands (destination, size, etc.) can be set up prior to data buffer readiness and readiness triggered with a simple doorbell/counter later. This approach is useful for future development of MPI operations in environments where traditional networking commands can have performance challenges, like accelerators (GPUs, FPGAs). In this paper, we detail the design and implementation of a layered library (built on top of MPI‐3.1) and an integrated Open MPI solution that supports the new, MPI‐4.0 partitioned communication feature set. The library will enable applications to use currently released MPI implementations and older legacy libraries to provide partitioned communication support while also enabling further exploration of this new communication model in new applications and use cases. We will compare the designs of the library and native Open MPI support, provide performance results and comparisons between the two approaches, and lessons learned from the implementation of partitioned communication in both library and native forms. We find that the native implementation and library have similar performance with a percentage difference under 0.94% in microbenchmarks and performance within 5% for a partitioned communication enabled proxy application.

Scientific workflows have become integral tools in broad scientific computing use cases. Science discovery is increasingly dependent on workflows to orchestrate large and complex scientific experiments that range from execution of a cloud-based data preprocessing pipeline to multi-facility instrument-to-edge-to-HPC computational workflows. Given the changing landscape of scientific computing and the evolving needs of emerging scientific applications, it is paramount that the development of novel scientific workflows and system functionalities seek to increase the efficiency, resilience, and pervasiveness of existing systems and applications. Specifically, the proliferation of machine learning/artificial intelligence (ML/AI) workflows, need for processing large scale datasets produced by instruments at the edge, intensification of near real-time data processing, support for long-term experiment campaigns, and emergence of quantum computing as an adjunct to HPC, have significantly changed the functional and operational requirements of workflow systems. Workflow systems now need to, for example, support data streams from the edge-to-cloud-to-HPC enable the management of many small-sized files, allow data reduction while ensuring high accuracy, orchestrate distributed services (workflows, instruments, data movement, provenance, publication, etc.) across computing and user facilities, among others. Further, to accelerate science, it is also necessary that these systems implement specifications/standards and APIs for seamless (horizontal and vertical) integration between systems and applications, as well as enabling the publication of workflows and their associated products according to the FAIR principles. This document reports on discussions and findings from the 2022 international edition of the Workflows Community Summit that took place on November 29 and 30, 2022.

Traditional point-to-point communication sends data only after the entirety of the data is available. This includes situations where multiple actors (e.g., threads) contribute to the send buffer. As a result, cases where the completion times of these actors are widely distributed may be lost opportunities for optimization because data ready to be sent is waiting to be transmitted. Fine-grained communication exposes these opportunities by allowing buffers to be divided into element s that can then be sent independently (see e.g., Partitioned Communication in Message Passing Interface v4.0). While some research has been directed at exploring the utility of such ‘early-bird’ transmission, the overall search space for finding the best performing actor completion timings and element counts is large. In this work, we present an abstract model of fine-grained communication based on the LogGP model and a complementary benchmark. We use the model to explore actor completion timing scenarios and identify trends in communication behavior based on factors such as overall message size and delay between actor completions. We evaluate the benchmarks on three systems utilizing distinct network technologies and show that: (i) smaller numbers of element s are able to exploit most of the benefit of early-bird communication, (ii) performance benefit will depend non-trivially on application behavior, and (iii) benefits are highly network-dependent.