Mapreduce Optimization Using Regulated Dynamic Prioritization
ACM SIGMETRICS Performance Evaluation Review(2009)
摘要
We present a system for allocating resources in shared data and compute clusters that improves MapReduce job scheduling in three ways. First, the system uses regulated and user-assigned priorities to offer different service levels to jobs and users over time. Second, the system dynamically adjusts resource allocations to fit the requirements of different job stages. Finally, the system automatically detects and eliminates bottlenecks within a job. We show experimentally using real applications that users can optimize not only job execution time but also the cost-benefit ratio or prioritization efficiency of a job using these three strategies. Our approach relies on a proportional share mechanism that continuously allocates virtual machine resources. Our experimental results show a 11 - 31% improvement in completion time and 4 - 187% improvement in prioritization efficiency for different classes of MapReduce jobs. We further show that delay intolerant users gain even more from our system.
更多查看译文
关键词
MapReduce,Proportional Share,Resource Allocation,Workflow Optimization,DISC
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络