Reducing the Impact of I/O Contention in Numerical Weather Prediction Workflows at Scale Using DAOS
arxiv(2024)
摘要
Operational Numerical Weather Prediction (NWP) workflows are highly
data-intensive. Data volumes have increased by many orders of magnitude over
the last 40 years, and are expected to continue to do so, especially given the
upcoming adoption of Machine Learning in forecast processes. Parallel
POSIX-compliant file systems have been the dominant paradigm in data storage
and exchange in HPC workflows for many years. This paper presents ECMWF's move
beyond the POSIX paradigm, implementing a backend for their storage library to
support DAOS – a novel high-performance object store designed for massively
distributed Non-Volatile Memory. This system is demonstrated to be able to
outperform the highly mature and optimised POSIX backend when used under high
load and contention, as per typical forecast workflow I/O patterns. This work
constitutes a significant step forward, beyond the performance constraints
imposed by POSIX semantics.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要