Performance Prediction for Distributed Graph Computing

Performance estimation for executing graph algorithms on the distributed systems, especially with sacrificing the accuracy of results to improve the runtime performance, is a pre-requisite for optimizing system parameters to achieve an applicable tradeoff between the runtime and the inaccuracy of results. This paper presents an experimental approach that predicts the runtime and the inaccuracy of conducting graph algorithms on the BSP-based distributed graph computing systems to optimize system parameters by using an artificial neural network (ANN) model. It samples different scales of subgraphs from the complete input graphs and executes the underlying algorithm on each subgraph to capture its characteristics. Then it essentially predicts the performance of executing the underlying algorithm on the complete graph by learning the scalability that how the runtime and the inaccuracy of results vary with different scales of graphs with an ANN network, which is trained off-line based on the captured characteristics of the subgraphs. We conducted all experimental studies on the Amazon EC2 Cloud. The experimental results demonstrate that the prediction approach can effectively predict the runtime with a relative error rate under 8% averagely and the inaccuracy of results with a relative error rate under 25% averagely compared to the actual performance results.