Cost-Efficient Distributed MapReduce Job Scheduling across Cloud Federation

This paper proposes a fully distributed scheduling algorithm to process MapReduce data-intensive applications across geo-distributed clusters in federated clouds. The proposed algorithm, called FedSCD, takes advantage of data locality while reducing both VM cost and data transfer cost (between clusters) subject to Deadline constraint. This work is compared to conventional partially distributed scheduling algorithms in federated multi-cloud environments. Performance evaluation proves that the proposed algorithm FedSCD can reduce the MapReduce job cost by an average of 40% and ensure optimal resource allocation.