Scaling Distributed File Systems In Resource-Harvesting Datacenters
2017 USENIX ANNUAL TECHNICAL CONFERENCE (USENIX ATC '17)(2017)
摘要
Datacenters can use distributed file systems to store data for batch processing on the same servers that run latency critical services. Taking advantage of this storage capacity involves minimizing interference with the co-located services, while implementing user-friendly, efficient, and scalable file system access. Unfortunately, current systems fail one or more of these requirements, and must be manually partitioned across independent subclusters. Thus, in this paper, we introduce techniques for automatically and transparently scaling such file systems to entire resource-harvesting datacenters. We create a layer of software in front of the existing metadata managers, assign servers to subclusters to minimize interference and data movement, and smartly migrate data across subclusters in the background. We implement our techniques in HDFS, and evaluate them using simulation of 10 production datacenters and a real 4k-server deployment. Our results show that our techniques produce high file access performance, and high data durability and availability, while migrating a limited amount of data. We recently deployed our system onto 30k servers in Bing's datacenters, and discuss lessons from this deployment.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络