XkitS:A computational storage framework for high energy physics based on EOS storage system

EPJ Web of Conferences(2024)

引用 0|浏览2
暂无评分
摘要
Large-scale high-energy physics experiments generate scientific data at the scale of petabytes or even exabytes, requiring high-performance data IO for processing. However, in large computing centers, computing and storage devices are typically separated. Large-scale data transfer has become a bottleneck for some data-intensive computing tasks, such as data encoding and decoding, compression, sorting, etc. The time spent on data transfer can account for 50% of the entire computing task. The larger the amount of data accessed, the more significant this cost becomes. One attractive solution to address this problem is to offload a portion of data processing to the storage layer. However, modifying traditional storage systems to support computation offloading is often cumbersome and requires a broad understanding of their internal principles. Therefore, we have designed a flexible software framework called XkitS, which builds a computable storage system by extending the existing storage system EOS. This framework is deployed on the EOS FTS storage server and offloads computational tasks by invoking the computing capabilities (CPU, FPGA, etc.) on FTS. Currently, it has been tested and applied in the data processing of the Large High Altitude Air Shower Observatory (LHAASO), and the results show that the time spent on data decoding using the computable storage technology is half of that using the original method.
更多
查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要