Executing task graphs on multi-GPU systems presents challenges typically managed by CPU-side runtimes, which handle memory management, track dependencies, and balance load. However, the interplay of runtime components, CPUdriven kernel initialization, and dynamic task graph construction creates significant overhead. For static graphs, recent advancements have enabled GPU-side execution, demonstrating substantial performance gains in single-GPU scenarios. However, multi-GPU execution still lags behind in both usability and performance. In particular, no GPU-side solution exists for executing task graphs on multiple nodes. In this work, we introduce Mustard, a multi-GPU execution model that shifts execution of static task graphs entirely to the devices, drastically reducing overhead. Mustard offers a clean solution for executing CUDA graphs across multiple GPUs on multiple nodes without requiring modifications to GPU kernel code or the adoption of new runtime mechanisms or APIs. By transforming the task graph, Mustard enables precise tracking of task dependencies and load balancing directly on the GPU, eliminating the need for host CPU involvement. We evaluate our approach using generated graphs, as well as LU and Cholesky decomposition graphs. In a multi-node scenario with 64 GPUs, Mustard achieves an average 5.83x speedup over the linear algebra library SLATE. On a single node, compared to the best-performing baseline, Mustard delivers an average 1.66x speedup for LU and 1.29x for Cholesky.
The advent of cutting-edge AI applications has emphasized the importance of reducing inference latency. Consequently, efficient model-parallel execution on multiple GPUs represents a key challenge in achieving high performance through the partitioning of the target neural network. Nevertheless, in recent complex deep learning models, as the size of parameters continues to increase and overall inference latency is no longer solely dominated by kernel execution, performance improvements using multiple GPUs cannot be achieved by simply exploiting model parallelism without considering data transfer parallelism and the system topology. To address this challenge, this paper proposes SortingHat, which generates an efficient schedule of target neural network models on multi-GPU systems to minimize inference latency. Initially, SortingHat partitions a target model into multiple submodels based on dominator analysis to find the best solution within a reasonable time. Subsequently, SortingHat finds the best schedule for each submodel using Mixed Integer Linear Programming, taking system topology into account to exploit both model parallelism and data transfer parallelism. Once the schedules of all submodels are found, they are merged and executed on the ready queue-based executor. Evaluations on diverse multi-GPU environments with various large language models show that SortingHat achieves an average speedup of 2.28x and up to 2.96x over the single GPU on TVM baseline.
Pathological program behavior may cause a non-uniform access distribution in set associative caches, leading to an increase in conflict misses. To address this challenge, prior works profile the program patterns and propose different index functions to avoid these conflict misses [11, 18]. However, as we analyze the prior work on set-associative cache indexing, we identify two major issues. First, there is no single index function that is guaranteed to perform well for every application. Second, advanced indexing schemes typically have sophisticated implementation and prohibitively long computation latency. In this paper, we propose Duelhash, a dynamic N-way indexing framework for set associative caches, which provides an effective dueling mechanism for multiple index functions at runtime with a simple and efficient hardware implementation. At runtime, the performance of the index functions are evaluated periodically, and the best performer is applied to the cache. To evaluate the performance of Duelhash, we conduct a case study on a 16-way set-associative LLC using a diverse set of benchmarks, including SPEC 2006, SPEC 2017, PARSEC 3.0, CVP and GAP. Our empirical results show that without prefetching, Duelhash provides an IPC speed up of 2.8% (with the highest being 23%) over the conventional power-of-two modulo (Default) index, compared to a 1.6% speed up of a commercialized indexing scheme (Xorhash). When pattern-based prefetchers are turned on in the L1 data and L2 caches, Duelhash can provide up to 5.8% single-core speedup over Default. Duelhash also provides a 6.2% MPKI reduction over Default for non-uniform applications, the highest among all schemes tested in the study. The reduced LLC miss rate results in an uncore energy saving of 4.9% (without prefetching) or 4.1% (with prefetching). Compared to the state-of-the-art cache indexing scheme, Duelhash offers a comparable performance in most applications while using 93% less power.
Modern GPU-based lossy compressors face significant challenges due to the imbalance in development between compression throughput and compression ratio. Most compressors tend to prioritize one metric while neglecting the other, leading to either redundant compressed files or inefficient kernel designs that hinder end-to-end throughput. This work introduces Aatrox, a generic single-kernel error-bounded lossy compressor specifically designed for GPUs, targeting applications that demand high performance and high compression ratio at the same time, such as large-scale quantum circuit simulations and large language model training. In particular, Aatrox features a novel hierarchical data blocking strategy, large-block delta encoding, and dual-level delta decoding, achieving substantial end-to-end throughput and an optimized compression ratio. Experiments on NVIDIA A100 GPU using nine real-world scientific datasets show that Aatrox achieves higher compression ratios while preserving high data quality, with an average throughput of 388.3 GB/s for compression and 718.0 GB/s for decompression. These results represent approximately 1.2x speedup compared to the throughput of existing pure-GPU compressors and 250x that of CPU-GPU hybrid compressors.
Computing a maximal independent set (MIS) of a graph is an important problem in many scientific applications. Several parallel algorithms exist to perform this computation quickly. Though the state-of-the-art GPU implementation is very efficient, it cannot process graphs that do not fit in the global memory of a single GPU. We propose MG-MIS, a multi-GPU algorithm that addresses this problem. It distributes the computation across the GPUs in a compute node and uses novel techniques to minimize inter-GPU communication. Our results show that, for graphs that require more than 32 GB memory, MG-MIS outperforms the state-of-the-art singleGPU code with UVM by a geometric mean of 17.73x on a system with 4 V100 GPUs, each with 32 GB global memory. For another set of graphs that require more than 12 GB memory, MG-MIS outperforms the same single-GPU code by 22.88x on a system with 2 RTX 3080 GPUs, each with a global memory of 12 GB. On average, the size of the MIS computed by MG-MIS is 2.6% smaller than that produced by the state-of-the-art single-GPU code.
