SnailTrail : Online Bottleneck Detection for your Dataflow †

Motivation Understanding the performance of large-scale data processing applications is hard. In distributed dataflow systems, the computation of several parallel processes is interleaved with data and control communication and execution dependencies typically span multiple system components. In such an environment, bottleneck detection is cumbersome and currently relies heavily on humans. After decades of systems research, the state-of-the-art in performance analysis still relies on offline trace processing, thus it is only suitable for batch computations and post-mortem reports. This work presents SnailTrail, a novel framework for online performance analysis in modern dataflow engines that can identify bottlenecks in real-time and make automated optimization possible at runtime.