This paper presents the design and implementation of a multi-GPU concurrent queue system using NVIDIA's NVSHMEM. The Bellman-Ford algorithm is used as a case study to evaluate the performance of the proposed concurrent FIFO queue, with this multi-GPU implementation being the first known instance of its kind. Experimental results demonstrate that the multi-GPU queue implementation achieves a maximum speedup of 3.92x and an average speedup of 3.04x over the singleGPU baseline on four NVIDIA A100 GPUs. When applied to the Bellman-Ford Single-Source Shortest Path (SSSP) algorithm, the multi-GPU system achieves a maximum speedup of 3.03x and an average speedup of 2.65x compared to the single-GPU implementation, tested on 10 graphs of different kinds taken from the SuiteSparse Matrix Collection.
We present a solution to unify various High Performance Computing (HPC) resources and enable environmentally sustainable (“green”), automated job assignment for independent HPC systems. The goal is to increase utilization of carbon-neutral computing resources and reduce the time to solution with simple job management using a job specification file and command line interface modelled after established batch systems. Our workload-agnostic approach is motivated by a comprehensive overview of existing software, which reveals the lack of a solution for executing arbitrary jobs for multiple users across multiple targets with low technical requirements and complexity. The proposed solution is transparent to HPC administrators through SSH-based instrumentation and to users with regard to the utilized batch system. We demonstrate the capabilities of our prototype in a case study of Molecular Dynamics (MD) simulations. The individual jobs are allocated over four different target systems and the scheduling policy takes into account the availability of resources which can be powered exclusively by Renewable Energy (RE). This results in a reduction of Carbon Emissions (CEs) by at least 9% compared to running all jobs on a conventional HPC system. In the future, we plan to extend our scheduling policy to exploit job execution timing using grid carbon intensity over time and investigate additional simulation parameters to minimize the economical and ecological impact of HPC jobs.
Asymmetric multicore processors such as Intel Arrow Lake, integrate performance (P) and efficiency (E) cores within the same die, offering new opportunities for flexible execution. However, these architectures also introduce additional challenges for energy-efficient operation, as balancing power and performance across such heterogeneous cores of these architectures is nontrivial. In this work, we first derive nearoptimal energy-efficient regions for three classes of bottleneckoriented microbenchmarks. We then investigate to what extent microbenchmark-derived insights translate to real applications. For this purpose, we characterize applications using the Roofline Model and additional performance metrics, to identify their compute-, bandwidth-, or latency-bound behavior. Based on this classification, we derive simple guidelines for selecting core type, operating frequency, and core number. We evaluate these guidelines using representative workloads from SPEC CPU 2017 and GAP graph benchmark suite. Our results show that microbenchmark-based insights can provide practical guidance for energy-efficient execution while also highlighting limitations when applied to more complex applications.
The exponential growth of data-intensive workloads in high-performance computing has turned traditional shared parallel file systems into a performance bottleneck. To address this, ad-hoc parallel file systems have emerged as a highperformance alternative, taking advantage of local node resources to provide ephemeral, low-latency storage. However, their temporary nature introduces a critical reliability challenge, since any node failure or job termination results in the total loss of the ad-hoc data. In addition, traditional checkpointing is ineffective here, as it usually focuses on application memory and does not account for potential data loss within the ad-hoc file system itself. This work addresses this limitation by presenting the integration of a transparent checkpointing workflow with the Expand Ad-Hoc parallel file system. The proposed solution employs the MPI-Agnostic Network-Agnostic (MANA) framework, providing a unified solution that saves both application memory and ad-hoc storage data. By leveraging the fault tolerance of Expand AdHoc to store checkpoint images, the system ensures data integrity against node failures. Additionally, an intelligent mechanism selectively captures only modified files, effectively reducing I/O redundancy and overhead for read-only files. This solution was evaluated on the C3 supercomputer using the IOR benchmark and the Quantum ESPRESSO, a real-world scientific application. The experimental results demonstrate that our integrated solution reduces checkpointing time by 44% to 83% compared to traditional backend storage systems, as illustrated in these experiments using Lustre. In scenarios with frequent checkpointing intervals, the implementation of Expand Ad-Hoc results in a substantial reduction in total execution time, demonstrating its effectiveness as a high-performance, transparent resilience solution for large-scale computing.
The Nvidia Grace Hopper Superchip has seen a wide adoption across both HPC sites and AI data centers. This hybrid architecture places the Grace CPU and Hopper GPU on the same board, with a shared and adjustable power budget. We present an end-to-end analysis of its energy efficiency, from a verification of power sensor accuracy using microbenchmarks, an investigation of power knob effects to an energy-efficiency and TCO investigation for selected Large Language Model (LLM) inference use cases. Our study shows that the internal power sensors provide precise readings when compared to an external power meter, unlike contemporary x86 processors. Some power budget configurations are not enforced by the system, the power draw can exceed the configured limit. This behavior is not documented by Nvidia. We analyzed the energy efficiency of LLM inference for prefill and decoding-dominated workloads, with the latter having emerged as the driving factor. Increasing the module power limit while maintaining a constant GPU power did not affect the system's energy efficiency. According to our TCO analysis, no cost savings can be expected from reduced power limits for output lengths of up to 2048 tokens for our LLM setup.
