With the recent explosive growth in worldwide data and data processing demands, the need to support a large volume of transactions on shared data is increasing in both high performance computing and datacenter processing. A recent innovation in server architectures is the use of disaggregated memory based on the Compute eXpress Link (CXL) interconnect protocol. This memory architecture is increasing in popularity as it allows for dynamic demand-sensitive resizing of aggregated memory, support for heterogeneous memory types, and sharing of data amongst supported processors and devices, including computational accelerators. However, while this new memory architecture alleviates many concerns in datacenter architectures, the data integrity when using memory based transactions over CXL faces many challenges.To solve for these challenges, we describe a novel solution for providing ACID (Atomicity, Consistency, Isolation, Durability) transactions in a CXL-based disaggregated memory architecture. We call this solution Transactional CXL or TCXL. TCXL requires no changes to the existing processor microarchitectures and is implemented in a software library with a back-end controller that can be embedded in a CXL controller or as a stand alone CXL Device or implemented on a host.The transactions support persistent memory durable transactions and in-memory volatile transactions which can be in a pooled memory expansion for a single processor or shared amongst multiple processors. TCXL also supports processor based Hardware Transactional Memory (HTM) based transactions both on processor and over CXL. We evaluate TCXL by extending a CXL simulator and executing micro-benchmarks. In addition to gaining the benefits of using CXL, we show TCXL outperforms other approaches.
Reliable and uninterrupted operation is crucial in supercomputers, especially during failures or inconsistencies i.e., anomalies. In this paper, we present a federated adaptive Digital Twin (DT) framework, with a focus on enhancing anomaly detection - a critical aspect of modern data center management. Our DT continuously monitors key metrics, detects anomalies powered by AI, and dynamically adjusts its monitoring parameters to ensure optimal performance. Using a dashboard, our system provides real-time alarms and detailed visualizations of detected anomalies, along with real-time visualization and forecast for selected metrics. Through a series of experiments, we validate the effectiveness of our approach in maintaining operational reliability and promptly identifying potential anomalies within the data center.
We explore the development of a performance-portable CPU/GPU ecosystem to integrate two of the US Department of Energy’s (DOE’s) largest scientific instruments, the Oak Ridge Leadership Computing facility and the Spallation Neutron Source (SNS), both of which are housed at Oak Ridge National Laboratory. We select a relevant data reduction workflow use-case to obtain the differential scattering cross-section from data collected by SNS’s CORELLI and TOPAZ instruments. We compare the current CPU-only production implementation using the Garnet Python multiprocess package based on the Mantid C++ framework against our proposed CPU/GPU implementation that uses the LLVM-based, just-in-time Julia scientific language and the JACC.jl performance-portable package. Two proxy apps were developed: (i) an app for extracting relevant Mantid kernels (MDNorm) in C++ and (ii) the Julia MiniVATES.jl miniapp. We present performance results for NVIDIA A100 and AMD MI100 GPUs and AMD EPYC 7513 and 7662 CPUs. The results provide insights for future generations of data reduction software that can embrace performance portability for an integrated research infrastructure across DOE’s experimental and computational facilities.
In this paper, we present GPU-optimizations for an ice-sheet modeling code known as MPAS-Albany Land Ice (MALI). MALI is a C++ template code that leverages the Kokkos programming model for portability and the Trilinos library for data structures, nonlinear and linear solvers and optimization packages for ice-sheet simulations. Performance of the most expensive kernel is assessed via the Roofline model to highlight the potential for code improvement according to the underlying GPU architecture. We perform a collection of optimizations consisting of loop fusions, loop optimizations and local accumulation to productively and portably attain an overall speedup of 3× in either NVIDIA and AMD GPU. We analyze the performance gains using a time-oriented performance portability model based on time per invocation and GPU data movement. Results show an increment between 20% and 50% on the performance portability metric by improving data locality on the GPU kernels of a Stokes solver and highlights the importance of optimizing GPU-ported scientific applications to maximize memory bandwidth and minimize data movement on modern supercomputers.
To increase the dependability and portability of scientific data analysis workflows (DAWs), recent work has proposed contract-driven design of DAWs, providing verifiable expectations and obligations to ensure that tasks run in a proper environment and produce correct results. However, the specification of suitable contracts is still left to the discretion of DAW developers, imposing labor-intensive manual work which likely hampers the widespread adoption of contracts in scientific practice. We report about work-in-progress of developing a pipeline empowered by Large Language Models for automatically generating code contracts from logical workflow descriptions. We instantiate this pipeline within the workflow system Nextflow, and evaluate its contract generation capabilities in an experiment using real-world Nextflow modules. Our findings indicate that we generate a substantial amount of contracts serving as starting point for DAW developers. Our approach demonstrates potential in assisting domain scientists with contract-driven design of DAWs, laying the groundwork for its future adoption.
Exascale applications are being increasingly written in modern languages such as Python, Julia, C++, and Rust. The Message-Passing Interface (MPI), the de facto standard for parallel computing, only defines interfaces for C and Fortran, languages that are very different from these modern languages, often containing more complex types and representations incompatible with MPI. The existing derived datatype interface is widely used for older applications, but fails to work efficiently for types containing multiple pointers, requiring application-specific initialization, or serialization. Applications written in these languages can still use MPI, but at the cost of complicated address manipulation or high overhead. This work proposes a new datatype interface for MPI giving more control to the application over buffer packing and the wire representation. We built a prototype for this interface, demonstrating it with Rust, Python, and C++, highlighting key concerns of each language and showing the improvements provided.
Graph Neural Networks (GNNs) have been successfully used in a variety of challenging application areas, including Electronic Design Automation and molecular property prediction, among others. However, training GNN models is time-consuming as it incurs a high volume of irregular data accessing due to its graph-structured input data; such a challenge is further exacerbated in real-world applications as they often involve training GNN models on large-scale graphs with over billions of edges. While several GNN accelerators have been proposed, most of them cannot scale to billion-scale graphs due to the limitation of memory capacity. To this end, we propose xBS-GNN, a novel accelerator optimized for billion-scale GNN training. xBS-GNN exploits the multi-level memory hierarchy on state-of-the-art FPGA-based systems to enable billion-scale GNN training. To achieve high training throughput, xBS-GNN jointly exploits several optimizations, including (1) a novel data placement policy optimized for GNN training, along with (2) a vertex-renaming technique and memory-efficient lookup table design for fast data retrieval, and (3) a feature quantization mechanism to reduce memory traffic. We evaluate xBS-GNN with a three-layer GCN model on three large datasets. xBS-GNN achieves up to 8.39× speedup over a widely-used GPU baseline and up to 5.13× speedup over a state-of-the-art GNN training accelerator. xBS-GNN also demonstrates high scalability on multi-FPGA platforms.
We present DeLiBA-K, an improved version of the Development of Linux Block I/O Accelerators (DeLiBA) framework. DeLiBA-K operates at the Linux kernel level, bypassing the user-space interactions of DeLiBA-1 and -2 to interact with the block and network I/O kernel stack directly. Another critical feature of DeLiBA-K is implementing and benchmarking the modern io_uring Asynchronous I/O (AIO) API within a 16nm AMD Alveo U280 FPGA I/O framework. This allows for better parallelism and reduced latency in I/O operations. Our results show significant performance gains, up to a 3.2x improvement in I/O operations per second (IOPS) and 3.45x increased throughput for synthetic workloads. Real-world applications see a 30% reduction in execution time for data-intensive tasks. DeLiBA-K has been successfully tested in an industrial environment using real workloads, demonstrating its effectiveness in large-scale enterprise environments.
The performance of large-scale graph analytics is limited by the capacity and performance of the memory subsystem on the platforms on which they execute. In this paper, we first discuss the limitations of existing approaches to scaling graph processing, and describe how they can be addressed via the use of disaggregated solutions with near-data processing (NDP) capabilities. Using observations from experimental analysis of the trade-offs for different types of graphs and analytics kernels, we identify the systems-level mechanisms that will be required by future graph analytics frameworks for disaggregated NDP architectures.
OpenMP® is a widely used API in high-performance computing that enables parallelization on the host as well as offload work to an accelerator, such as a GPU. The OpenMP specification defines an OpenMP Tool Interface (OMPT), which allows a third-party tool be notified about OpenMP runtime events. Ensuring that the runtime correctly reports such events is thus important. We propose a unit testing framework for testing OpenMP implementations, like the ROCm™ compiler. It offers a simple-to-use framework that allows a tester to check for OMPT events in addition to the regular unit testing code. It also facilitates writing concise tests while bridging the semantic gap between the unit under test and the OMPT-event testing. Our experimental results show that for the ROCm compiler, ompTest provides coverage similar to its existing test cases, with better readability, and a compile and runtime speedup close to 2.5× for the ROCm OMPT test suite.
We present JACC (Julia for Accelerators), the first high-level, and performance-portable model for the just-in-time and LLVM-based Julia language. JACC provides a unified and lightweight front end across different back ends available in Julia, enabling the same Julia code to run efficiently on many HPC CPU and GPU targets. We evaluated the performance of JACC for common HPC kernels as well as for the most computationally demanding kernels used in applications, HPCCG, a supercomputing benchmark test for sparse domains, and HARVEY, a blood flow simulator to assist in the diagnosis and treatment of patients suffering from vascular diseases. We carried out the performance analysis on the most advanced US DOE supercomputers: Aurora, Frontier, and Perlmutter. Overall, we show that JACC has a negligible overhead versus vendor-specific solutions, reporting GPU speedups with no extra cost to programmability.
The limiting factor in the application of high-accuracy quantum molecular simulations to large systems has been the associated high computational costs in terms of both compute power and memory. In this paper we explore the use of various BLAS precision modes (BF16, TF32, and Complex 3M) in DCMESH (divide-and-conquer Maxwell-Ehrenfest-surface hopping), a framework utilized for the study of light-matter interaction. On a single stack of the Intel® Data Center GPU Max Series 1550, we are able to achieve a speedup of 1.35x while retaining accuracy in key output parameters such as the number of excited electrons, current density, and kinetic energy. For large problem sizes, we observe speed-ups of up to 3.91x for individual BLAS calls. Switching between BLAS precision modes requires no source code changes (only environment variables), and so the approach we demonstrate here could be readily applied to other High Performance Computing (HPC) workloads that spend a significant amount of time in BLAS calls.
Power management and energy efficiency are critical research areas for exascale computing and beyond, necessitating reliable telemetry and control for distributed systems. Despite this need, existing approaches present several limitations precluding their adoption in production. These limitations include, but are not limited to, lack of portability due to vendor-specific and closed-source solutions, lack of support for non-MPI applications, and lack of user-level customization.We present a job-level power management framework based on Flux. We introduce flux-power-monitor and demonstrate its effectiveness on the Lassen (IBM Power AC922) and Tioga (HPE Cray EX235A) systems with a low average overhead of 0.4%. We also present flux-power-manager, where we discuss a proportional sharing policy and introduce a hierarchical FFT-based dynamic power management algorithm (FPP). We demonstrate that FPP reduces energy by 1% compared to proportional sharing, and by 20% compared to the default IBM static power capping policy.
Disk failure data provides valuable insights for preventing failures, enhancing storage robustness, guiding system design and deployment, and ensuring reliable operations at data centers. This paper introduces two disk failure datasets collected from large-scale HPC production environments over the past five years, comprising over 5,000 failure records from more than 40,000 disks. We analyzed these datasets across multiple dimensions, including temporal, spatial, and relational trends, and performed a comprehensive reliability assessment. Our analysis yielded numerous observations and insights that influence various operational aspects of HPC storage systems. We believe this study offers a holistic understanding of disk failure trends likely to interest the HPC storage community.
Scientific productivity can be enhanced through workflow management tools, relieving large High Performance Computing (HPC) system users from the tedious tasks of scheduling and designing the complex computational execution of scientific applications. This paper presents a study on the usage of ensemble workflow tools to accelerate science using the Summit and Frontier supercomputing systems. The research aims to connect science domain simulations using Oak Ridge Leadership Computing Facility (OLCF) supercomputing platforms with ensemble workflow methods in order to accelerate HPC-enabled discovery and boost scientific impact. We present the coupling, porting and optimization of Radical-Cybertools on three applications: Chroma, NAMD and LAMMPS. The tools augment traditional HPC monolithic runs with a pilot scheduler. Lessons-learned are discussed for physics, biology and materials science applications. We discuss intrinsic limitations of coupling and porting ensemble workflow tools to applications that run on large HPC systems. The origins of technical challenges and their solutions developed during the implementation process are discussed. Data management strategies, OLCF’s policies for ensembles, and natively supported workflow tools are also summarized. 1
The OMFIT CAKE workflow for plasma state reconstruction has been automated on the DIII-D National Fusion Facility to run on a combination of computational resources at DIII-D and the National Energy Research Computing (NERSC) Center, utilizing the emerging DOE Integrated Research Infrastructure (IRI). The reconstruction of the plasma state is vital for understanding what occurred in the DIII-D machine during the pulse. This understanding allows informed decisions to be made on how to change the configuration for the next pulse. The initial reconstruction workflow was performed on DIII-D resources for a benchmark case in 62 minutes. The wall-clock time for the benchmark case was reduced to 11 minutes by running on the Perlmutter system at NERSC, which opens the possibility to influence decisions between DIII-D pulses during an experiment. The reconstruction results can be used as inputs for other modeling analyses; the determination of the classification of the microturbulence modes is given as an example model analysis.
In modern HPC systems, performance measurements are often disturbed by noise. Because repeating measurements to increase confidence in their results is costly, alternative noise-resilient techniques are desirable. Therefore, we implement a logical clock, which does not rely on real-time measurements, in Score-P. We explore several methods to model computational work with the clock increment, counting OpenMP loop iterations, LLVM basic blocks/statements, or hardware counters. We demonstrate the strengths and weaknesses of using logical time stamps in a trace analysis workflow with Score-P and Scalasca, by evaluating the performance problems we can find in three MPI+OpenMP mini-apps. By design, logical measurements reliably show algorithmic issues, such as load imbalance, but cannot capture external aspects of program execution, for example memory contention. In summary, logical-time based measurements are a specialized but valuable addition to the performance analyst’s toolbox.
Interconnects have always played a cornerstone role In HPC. Since the Inception of the Top500 ranking, Interconnect statistics have been predominantly dominated by two competing technologies: InfiniBand and Ethernet. However, even if Ethernet is very popular due to versatility and cost-effectiveness, InfiniBand used to provide higher bandwidth and continues to feature lower latency. Industry seeks for a further evolution of the Ethernet standards to enable fast and low-latency interconnect for emerging AI workloads by offering competitive, open-standard solutions. This paper analyzes the early results obtained from two systems relying on an HPC Ethernet interconnect, one relying on 100G and the other on 200G Ethernet. Preliminary findings indicate that the Ethernet-based networks exhibit competitive performance, closely aligning with InfiniBand, especially for large message exchanges.
This paper reviews recent enhancements to the Linux kernel that impact network throughput, and their potential impact on Data Transfer Node (DTN) performance. In particular, we explore the benefits of MSG_ZEROCOPY and BIG TCP in controlled testbed environments at AmLight and ESnet. We compare performance on three different Linux kernel versions, on Intel vs AMD processors, and over multiple round trip times. Our results indicates that MSG_ZEROCOPY, in conjunction with packet pacing, provides up to 35% improvement in throughput, and that Linux 6.8 provides an increase in throughput of up to 38% on WAN and 30% on LAN compared to the 5.15 kernel. We conclude with recommendations for both host benchmarking and production-ready DTN configurations.
Hybrid MPI + X models, combining the Message Passing Interface (MPI) with node-level parallel programming models, increase complexity and introduce additional correctness issues. This work addresses the challenges of detecting data races in hybrid CUDA-aware MPI applications due to the asynchronous and non-blocking nature of CUDA and MPI APIs. We introduce CuSan, an LLVM compiler extension, and runtime that tracks CUDA-specific concurrency, synchronization, and memory access semantics. We integrate CuSan with MUST, a dynamic MPI correctness tool, and ThreadSanitizer (TSan), a thread-level data race detector. MUST with TSan can already detect concurrency issues for multi-threaded MPI codes. Together with CuSan, these tools allow for comprehensive correctness checking of concurrency issues in CUDA-aware MPI applications. Our evaluation of two mini-apps reveals runtime overhead of CuSan ranging from 6× to 36×, depending on the amount of memory tracked by TSan, compared to the uninstrumented version. Memory overhead consistently remains under 1.8×. CuSan is available at https://github.com/tudasc/cusan.
