Graph-Optimizer: Towards Predictable Large-Scale Graph Processing Workloads

Graph and hardware-specific optimisations lead to orders of magnitude improvements in performance, energy, and cost over conventional graph processing methods. Typical big data platforms, such as Apache MapReduce and Apache Spark, rely on generic primitives, exhibiting poor performance and high financial and environmental costs. Even optimised basic graph operations (BGOs) lack the tools to combine them towards real-world applications. Furthermore, graph topology and dynamics (i.e., changing the number and content of vertices and edges) lead to high variability in computational needs. Primitive predictive models demonstrate they can enable algorithm selection and advanced auto-scaling techniques to ensure better performance, but no such models exist for energy consumption. In this work, we present the Graph-Optimizer tool. Graph-Optimizer uses optimised BGOs and composition rules to capture and model the workload. It combines the workload model with hardware and infrastructure models, predicting performance and energy consumption. Combined with design space exploration, such predictions select codesigned workload implementations to fit a requested performance objective and guarantee their performance bounds during execution.