Design and Implementation of Distributed Computing Framework Based on GPU

As the scale of big data continues to expand, traditional single-machine processing methods have been replaced by distributed computing frameworks. With the limited potential for further CPU performance improvements, GPU-accelerated computing has emerged as a trend, particularly for single-instruction multiple-data (SIMD) and computationally intensive tasks. However, with the clustering of GPUs, traditional scheduling algorithms face limitations such as resource waste and inefficient allocation. To meet the diverse needs of big data applications, it is necessary to optimize the task scheduling and architecture of computing frameworks to enhance processing efficiency and adaptability. Addressing these challenges, this paper proposes a novel distributed computing framework, GDSF, based on a thorough examination of existing computing frameworks and task scheduling algorithms, combined with the computational characteristics of GPUs and the mechanism for multi-task GPU sharing. This framework incorporates various task scheduling strategies to ensure effective scheduling for both dependent and independent task models, achieving an optimal match between tasks and GPU resources. Experiments conducted on a real heterogeneous CPU-GPU cluster using diverse GPU workloads demonstrate that the performance of the GDSF computing framework and its scheduling algorithms meets the expected research objectives, thus validating the feasibility and effectiveness of the proposed design.