From Static to Malleable: Improving Flexibility and Compatibility in Burst Buffer File Systems.

Numerous burst buffer file systems have been developed in recent years in the context of high-performance computing (HPC). These file systems aim to combat I/O bottlenecks and cross-application interference that arise in parallel file systems due to competing and uncoordinated I/O patterns. While burst buffer file systems have demonstrated linear scaling for metadata and data workloads, their configurations are currently static and cannot adapt to changing requirements from applications or the HPC system. User space file system implementations have emerged as a promising solution since they can be customized to meet specific application I/O requirements. However, developing file systems in user space comes with significant challenges, such as the lack of mature I/O interfaces that can hurt application compatibility. This paper explores the challenges that have shaped the design of GekkoFS, an exemplary user space burst buffer file system, and presents an overhauled file system architecture to improve application compatibility and provide a foundation for additional malleability techniques, molding the file system to the application’s requirements. We also evaluate this new architecture and show its performance overhead that remains well above the storage capability of commonly used node-local storage devices. Overall, the paper demonstrates the potential benefits of this architecture for user space file system implementations and presents a path forward for their continued development.