We approach the task of network congestion control in datacenters using Reinforcement Learning (RL). Successful congestion control algorithms can dramatically improve latency and overall network throughput. Until today, no such learning-based algorithms have shown practical potential in this domain. Evidently, the most popular recent deployments rely on rule-based heuristics that are tested on a predetermined set of benchmarks. Consequently, these heuristics do not generalize well to newly-seen scenarios. Contrarily, we devise an RL-based algorithm with the aim of generalizing to different configurations of real-world datacenter networks. We overcome challenges such as partial-observability, non-stationarity, and multi-objectiveness. We further propose a policy gradient algorithm that leverages the analytical structure of the reward function to approximate its derivative and improve stability. We show that this scheme outperforms alternative popular RL approaches, and generalizes to scenarios that were not seen during training. Our experiments, conducted on a realistic simulator that emulates communication networks' behavior, exhibit improved performance concurrently on the multiple considered metrics compared to the popular algorithms deployed today in real datacenters. Our algorithm is being productized to replace heuristics in some of the largest datacenters in the world.

Networking devices deployed in ultra-scale data centers must run perfectly and in real-time. The networking device performance is tuned using the device configuration registers. The optimal configuration is derived from the network topology and traffic patterns. As a result, it is not possible to specify a single configuration that fits all scenarios, and manual tuning is required in order to optimize the devices' performance. Such tuning slows down data center deployments and consumes massive resources. Moreover, as traffic patterns change, the original tuning becomes obsolete and causes degraded performance. This necessitates expensive retuning, which in some cases is infeasible. In this work, we present ZTT: a continuously running Genetic Algorithm that can be used for online, automatic tuning of the networking device parameters. ZTT is adaptive, fast to respond and have low computational costs required for running on a networking device. We test ZTT in a diversity of real-world traffic scenarios and show that it is able to obtain a significant performance boost over static configurations suggested by experts. We also demonstrate that ZTT is able to outperform alternative search algorithms like Simulated Annealing and Recursive Random Search even when those are adapted to better match the task at hand.