Origami: Efficient ML-Driven Metadata Load Balancing for Distributed File Systems.

Modern distributed file systems (DFSs) rely on metadata server clusters to manage large-scale files and achieve scalability. However, the hierarchical namespace structure and dynamic user workloads pose severe challenges for efficient metadata partitioning and load balancing. Existing approaches primarily focus on identifying and redistributing hot metadata to address imbalances. While these load-balancing strategies offer potential benefits, they often reduce metadata locality, ultimately failing to improve the end-to-end job completion time-a key metric prioritized by users. Although recent research reveals that learning-based approaches are effective in predicting hotspots, they have been shown to be less effective in improving metadata performance. We revisit metadata load balancing strategies and propose a learning-based metadata load balance framework Origami, which focuses on minimizing end-to-end job completion time rather than equalizing loads. Origami first decomposes the overhead of metadata operations and assesses the impact of migration decisions on user requests, allowing us to compute the benefits of migration decisions for job completion time when future requests are known. Subsequently, Origami propose the Meta-OPT algorithm to determine near-optimal migration decisions. Finally, we implemented OrigamiFS, on which we collected statistical data to train and validate ML-models capable of predicting migration benefits. By predicting the benefits of migration decisions and employing Meta-OPT to quickly explore nearly optimal migration decisions, Origami makes a better trade-off between load balancing and namespace locality. Our evaluation shows that compared to state-of-the-art methods, Origami increases aggregated metadata throughput by 1.12-2.51x across three real-world workloads, and enhances end-to-end throughput by 1.11-2.02x.