ForkTail: a black-box fork-join tail latency prediction model for user-facing datacenter workloads.

The workflows of the predominant user-facing datacenter services, including web searching and social networking, are underlaid by various Fork-Join structures. Due to the lack of understanding the performance of Fork-Join structures in general, today's datacenters often resort to resource overprovisioning, operating under low resource utilization, to meet stringent tail-latency service level objectives (SLOs) for such services. Hence, to achieve high resource utilization, while meeting stringent tail-latency SLOs, it is of paramount importance to be able to accurately predict the tail latency for a broad range of Fork-Join structures of practical interests. In this paper, we propose ForkTail, a black-box Fork-Join tail latency prediction model that covers a wide range of Fork-Join structures. In ForkTail, all Fork nodes are treated as black boxes, admitting both homogeneous and inhomogeneous cases, and different requests in the request flow are allowed to spawn different numbers of tasks forked to different numbers of Fork nodes. On the basis of the central limit theorem for queuing models under heavy load, we are able to arrive at a highly computational effective, empirical expression for the tail latency as a function of the means and variances of the task response times. Since this expression can be applied to request sub-flows at any granularities, it can be used for tail-latency prediction for services in a consolidated environment, where different services and applications may share the same datacenter cluster resources. Our extensive testing results based on model-based and trace-driven simulations, as well as a real-world case study in a cloud environment demonstrate that the expression can consistently predict the tail latency within 20% and 15% prediction errors at 80% and 90% load levels, respectively. Moreover, our sensitivity analysis demonstrates that such errors can be well compensated for with no more than 5% and 3% resource overprovisioning at these two load levels, respectively. This, together with its extremely low computational complexity, makes ForkTail a viable tool for both offline and online job scheduling and resource provisioning for user-facing datacenter applications.