Collaborative Bandwidth-Efficient Intra-Node Allreduce

The concept of GPU-aware multi-path communication and heterogeneous computing is pivotal in enhancing HPC cluster performance. Heterogeneous computing utilizes various computing units to boost distributed applications, while multi-path communication frameworks optimize data transfer and reduce latency. This study proposes a collaborative Allreduce collective communication framework that utilizes heterogeneous computing and multi-path communication for multi-GPU systems. Leveraging GPU parallelism and high memory bandwidth, our approach optimizes both data transfer and computing power of the host system. We provide a proof-of-concept implementation and evaluation on a single-node multi-GPU system, comparing its performance to NCCL, UCC, and other algorithms. Our results show that our framework can improve Allreduce performance for large message sizes by up to 1.45x compared to NCCL. Our approach can be extended to various topologies and systems, with more extensive evaluations planned for the future.