A Traffic and Resource Aware Online Storm Scheduler.

Streaming applications have become widespread with the advent of big data and IoT. They are latency-sensitive applications that aim to process vast amounts of data in near real time. They are usually modelled as directed graphs and their deployment and orchestration in a cluster of nodes are managed by distributed stream processing systems. These systems are responsible for placing the graph components within the cluster nodes, which determines the application’s communication overhead and ultimately affects performance metrics such as latency and throughput. This work presents an adaptive, heuristic-based, scheduling algorithm for distributed stream processing systems that aims to minimize the latency and maximize the throughput of streaming applications deployed in heterogeneous clusters, while considering the resource constraints of the available nodes. The proposed approach uses a graph partitioning algorithm and real time traffic data monitored from a deployed application to derive a near-optimal operator placement plan that minimizes inter-node communication and balances the overall communication load distribution. We evaluated our approach using three micro-benchmark and two practical applications, and the results demonstrate that our scheduler outperforms the default scheduler of a popular stream processing system and a state-of-the-art algorithm, improving throughput by up to 106% and reducing complete latency by up to 58% for most applications.