Design of a Distributed Parallel Computing Engine for High-Throughput Data Processing of HEPS

HEPS is a fourth-generation synchrotron light source, and the experiments conducted at HEPS will transition to high-throughput, multi-modal, ultra-fast frequency, and cross-scale formats. The annual data flux generated by these experiments is anticipated to enter the 'Exa-scale' era. Given the substantial volume of high-throughput experimental data, a single computing node struggles to meet the computational demands for data analysis. Consequently, it is essential to develop a high-performance, robust, and user-friendly distributed parallel computing engine to enhance the performance of data processing software. Due to significant variations in data rates across different beamline stations, supporting heterogeneous resources (such as GPUs) in a flexible and fine-grained manner presents a challenge. To optimize computational efficiency and handle extensive datasets, we have developed a distributed parallel computing engine that can leverage scalable, heterogeneous computing resources to deliver HEPS' data analytical services across various scales. Experiment shows that our distributed computing engine significantly enhances the efficiency of HEPS' data processing.