Tail latency dominates the performance of synchronous pretraining jobs when running at very large scales. We describe a three-pronged approach: (1) a new RDMA-based transport protocol, MRC, sprays across many paths and actively load-balances between them, eliminating the issue of flow collisions (2) the use of multi-plane Clos topologies to get the benefits of high switch radix and redundancy, allowing training clusters well over 100K GPUs to be built as two-tier topologies while increasing physical redundancy, and (3) the use of static source-routing using SRv6 to allow MRC the freedom to bypass failures by itself. We describe our experiences running MRC and static SRv6 routing in production in OpenAI and Microsoft's largest training clusters, where it has been used to train the latest frontier models. We demonstrate how MRC allows AI training jobs to ride out many network failures that previously would have interrupted training.

With the advancement of data center intelligence, more flow states are being managed by programmable switches and DPUs to enhance efficiency and reduce overhead. However, in practical deployment, they face the challenge of achieving high availability. This is primarily due to the large flow tables in smart hardware, which require significant hardware and network overhead for synchronization between devices. On the other hand, the flow states often change, making rapid synchronization with the hardware another challenge. In this paper, we design a synchronization system, FHA, between smart network devices, utilizing both the control plane and the data plane to solve the challenge of rapid synchronization. Additionally, we have designed the RangeSync algorithm, which greatly reduces the overhead of synchronizing flow tables between systems. We implemented a preliminary prototype on the AMD Pensando DPU. In our evaluation, our data plane synchronization achieved a latency of 6 mu s. For the control plane channel, compared to the baseline, synchronization resulted in an 83.3% reduction in latency and a 66.7% decrease in overhead when synchronizing 12 million flow states.

Given the wide adoption of disaggregated storage in public clouds, networking is the key to enabling high performance and high reliability in a cloud storage service. In Azure, we choose Remote Direct Memory Access (RDMA) as our transport and aim to enable it for both storage frontend traffic (between compute virtual machines and storage clusters) and backend traffic (within a storage cluster) to fully realize its benefits. As compute and storage clusters may be located in different datacenters within an Azure region, we need to support RDMA at regional scale. This work presents our experience in deploying intra-region RDMA to support storage workloads in Azure. The high complexity and heterogeneity of our infrastructure bring a series of new challenges, such as the problem of interoperability between different types of RDMA network interface cards. We have made several changes to our network infrastructure to address these challenges. Today, around 70% of traffic in Azure is RDMA and intra-region RDMA is supported in all Azure public regions. RDMA helps us achieve significant disk I/O performance improvements and CPU core savings.

SONiC (Software for Open Networking in the Cloud) is instantiated in a chassis-based networking switch device to enable control plane functionality for the line cards and backplane. The SONiC platform may be configured with a routing table and BGP (border gateway protocol) to provide routing capabilities for the application-specific integrated circuits (ASICs) operating on each respective line card. Ethernet ports are utilized within the chassis to enable the utilization of standardized networking protocols, such as protocols on the data link layer (layer 2) within the OSI (Open Systems Interconnection) model. The implementation of SONiC and standardized networking techniques creates a simplified and more proficient routing system in the chassis framework.

Programmable ASIC technology enables the switching data plane to rapidly support emergent technologies such as VNF offloading, custom tunneling and in-band telemetry. We propose a new approach for a "hybrid mode" of ASIC programmability, which maintains a discrete legacy hardware pipeline and control functions (e.g. routing, bridging) while providing a way to extend it. This places requirements on the switching hardware, programming language, data plane APIs and the network OS in order to achieve this goal. In this paper we present two hardware agnostic hybrid mode applications using a Mellanox programmable switch ASIC, P4-16 programming language, SAI flexible APIs and the SONIC Open Network OS and Linux TC. Also applications based on the Onyx OS and Spectrum SDK is discussed as a hardware specific example.

Major cloud providers offer networks of virtual machines with private IP addresses as a service on the cloud. To isolate the address space of different customers, customers are required to tunnel their traffic to a Virtual Private Network (VPN) gateway, which is typically a middlebox inside the cloud that internally tunnels each packet to the correct destination. To improve performance, an increasing number of enterprises connect directly to the cloud provider's network at the edge, to a device we call the provider's edge (PE). PE is a chokepoint for customer's traffic to the cloud, and therefore a natural candidate for implementing network functions concerning customers' virtual networks, including the VPN gateway, to avoid a detour to middleboxes inside the cloud.At the scale of today's cloud providers, VPN gateways need to maintain information for around a million internal tunnels. We argue that no single commodity device can handle these many tunnels while providing a high enough port density to connect to hundreds of cloud customers at the edge. Thus, in this paper, we propose a hybrid architecture for the PE, consisting of a commodity switch, connected to a commodity server which uses Data-Plane Development Kit (DPDK) for fast packet processing. This architecture enables a variety of network functions at the edge by offering the benefits of both hardware and software data planes. We implement a scalable VPN gateway on our proposed PE and show that it matches the scale requirements of today's cloud providers while processing packets close to line rate.

Network reliability is critical for large clouds and online service providers like Microsoft. Our network is large, heterogeneous, complex and undergoes constant churns. In such an environment even small issues triggered by device failures, buggy device software, configuration errors, unproven management tools and unavoidable human errors can quickly cause large outages. A promising way to minimize such network outages is to proactively validate all network operations in a high-fidelity network emulator, before they are carried out in production. To this end, we present CrystalNet, a cloud-scale, high-fidelity network emulator. It runs real network device firmwares in a network of containers and virtual machines, loaded with production configurations. Network engineers can use the same management tools and methods to interact with the emulated network as they do with a production network. CrystalNet can handle heterogeneous device firmwares and can scale to emulate thousands of network devices in a matter of minutes. To reduce resource consumption, it carefully selects a boundary of emulations, while ensuring correctness of propagation of network changes. Microsoft's network engineers use CrystalNet on a daily basis to test planned network operations. Our experience shows that CrystalNet enables operators to detect many issues that could trigger significant outages.

Network reliability is critical for large clouds and online service providers like Microsoft. Our network is large, heterogeneous, complex and undergoes constant churns. In such an environment even small issues triggered by device failures, buggy device software, configuration errors, unproven management tools and unavoidable human errors can quickly cause large outages. A promising way to minimize such network outages is to proactively validate all network operations in a high-fidelity network emulator, before they are carried out in production. To this end, we present CrystalNet, a cloud-scale, high-fidelity network emulator. It runs real network device firmwares in a network of containers and virtual machines, loaded with production configurations. Network engineers can use the same management tools and methods to interact with the emulated network as they do with a production network. CrystalNet can handle heterogeneous device firmwares and can scale to emulate thousands of network devices in a matter of minutes. To reduce resource consumption, it carefully selects a boundary of emulations, while ensuring correctness of propagation of network changes. Microsoft's network engineers use CrystalNet on a daily basis to test planned network operations. Our experience shows that CrystalNet enables operators to detect many issues that could trigger significant outages.

Debugging faults in complex networks often requires capturing and analyzing traffic at the packet level. In this task, datacenter networks (DCNs) present unique challenges with their scale, traffic volume, and diversity of faults. To troubleshoot faults in a timely manner, DCN administrators must a) identify affected packets inside large volume of traffic; b) track them across multiple network components; c) analyze traffic traces for fault patterns; and d) test or confirm potential causes. To our knowledge, no tool today can achieve both the specificity and scale required for this task. We present Everflow, a packet-level network telemetry system for large DCNs. Everflow traces specific packets by implementing a powerful packet filter on top of "match and mirror" functionality of commodity switches. It shuffles captured packets to multiple analysis servers using load balancers built on switch ASICs, and it sends "guided probes" to test or confirm potential faults. We present experiments that demonstrate Everflow's scalability, and share experiences of troubleshooting network faults gathered from running it for over 6 months in Microsoft's DCNs.

Load balancing is a foundational function of datacenter infrastructures and is critical to the performance of online services hosted in datacenters. As the demand for cloud services grows, expensive and hard-to-scale dedicated hardware load balancers are being replaced with software load balancers that scale using a distributed data plane that runs on commodity servers. Software load balancers offer low cost, high availability and high flexibility, but suffer high latency and low capacity per load balancer, making them less than ideal for applications that demand either high throughput, or low latency or both. In this paper, we present DUET, which offers all the benefits of software load balancer, along with low latency and high availability - at next to no cost. We do this by exploiting a hitherto overlooked resource in the data center networks - the switches themselves. We show how to embed the load balancing functionality into existing hardware switches, thereby achieving organic scalability at no extra cost. For flexibility and high availability, DUET seamlessly integrates the switch-based load balancer with a small deployment of software load balancer. We enumerate and solve several architectural and algorithmic challenges involved in building such a hybrid load balancer. We evaluate DUET using a prototype implementation, as well as extensive simulations driven by traces from our production data centers. Our evaluation shows that DUET provides 10x more capacity than a software load balancer, at a fraction of a cost, while reducing latency by a factor of 10 or more, and is able to quickly adapt to network dynamics including failures.

Clos-based networks including Fat-tree and VL2 are being built in data centers, but existing per-flow based routing causes low network utilization and long latency tail. In this paper, by studying the structural properties of Fat-tree and VL2, we propose a per-packet round-robin based routing algorithm called Digit-Reversal Bouncing (DRB). DRB achieves perfect packet interleaving. Our analysis and simulations show that, compared with random-based load-balancing algorithms, DRB results in smaller and bounded queues even when traffic load approaches 100%, and it uses smaller re-sequencing buffer for absorbing out-of-order packet arrivals. Our implementation demonstrates that our design can be readily implemented with commodity switches. Experiments on our testbed, a Fat-tree with 54 servers, confirm our analysis and simulations, and further show that our design handles network failures in 1-2 seconds and has the desirable graceful performance degradation property.

Reliable Group Data Delivery (RGDD) is a pervasive traffic pattern in data centers. In an RGDD group, a sender needs to reliably deliver a copy of data to all the receivers. Existing solutions either do not scale due to the large number of RGDD groups (e.g., IP multicast) or cannot efficiently use network bandwidth (e.g., end-host overlays). Motivated by recent advances on data center network topology designs (multiple edge-disjoint Steiner trees for RGDD) and innovations on network devices (practical in-network packet caching), we propose Datacast for RGDD. Datacast explores two design spaces: 1) Datacast uses multiple edge-disjoint Steiner trees for data delivery acceleration. 2) Datacast leverages in-network packet caching and introduces a simple soft-state based congestion control algorithm to address the scalability and efficiency issues of RGDD. Our analysis reveals that Datacast congestion control works well with small cache sizes (e.g., 125KB) and causes few duplicate data transmissions (e.g., 1.19%). Both simulations and experiments confirm our theoretical analysis. We also use experiments to compare the performance of Datacast and BitTorrent. In a BCube(4, 1) with 1Gbps links, we use both Datacast and BitTorrent to transmit 4GB data. The link stress of Datacast is 1.01, while it is 1.39 for BitTorrent. By using two Steiner trees, Datacast finishes the transmission in 16.9s, while BitTorrent uses 52s.

Commodity switches are becoming increasingly important as they are the basic building blocks for the enterprise and data center networks. With the availability of all-in-one switching ASICs, these switches almost universally adopt single switching ASIC design. However, such design also brings two major limitations, i.e, limited forwarding table for flow-based forwarding scheme such as Openflow and shallow buffer for bursty traffic pattern. In this paper, we propose to use CPU in the switches to handle not only control plane but also data plane traffic. We show that this design can provide large forwarding table for flow-based forwarding scheme and deep packet buffer for bursty traffic. We build such a prototype switch on ServerSwitch platform. In our evaluation, we show that our prototype can achieve over 90% traffic offloading ratio, absorb large traffic bursts without a single packet drop, and can be easily programmed to detect and defend low-rate burst attacks.

There have been some serious concerns about the TCP performance in data center networks, including the long completion time of short TCP flows in competition with long TCP flows, and the congestion due to TCP incast. In this paper, we show that a properly tuned instant queue length based Explicit Congestion Notification (ECN) at the intermediate switches can alleviate both problems. Compared with previous work, our approach is appealing as it can be supported on current commodity switches with a simple parameter setting and it does not need any modification on ECN protocol at the end servers. Furthermore, we have observed a dilemma in which a higher ECN threshold leads to higher throughput for long flows whereas a lower threshold leads to more senders on incast under buffer pressure. We address this problem with a switch modification only scheme - dequeue marking, for further tuning the instant queue length based ECN to achieve optimal incast performance and long flow throughput with a single threshold value. Our experimental study demonstrates that dequeue marking is effective for increasing the maximum incast senders close to the performance limit of ECN, achieving a gain anywhere from 16% to 140%.