Hardware transports such as RoCE deliver high performance with minimal host CPU, but are best suited to special-purpose deployments that limit their use, e.g., backend networks or Ethernet with Priority Flow Control (PFC). We introduce Falcon, the first hardware transport that supports multiple Upper Layer Protocols (ULPs) and heterogeneous application workloads in general-purpose Ethernet datacenter environments (with losses and without special switch support). Key design elements include: delay-based congestion control with multipath load balancing; a layered design with a simple request-response transaction interface for multi-ULP support; hardware-based retransmissions and error-handling for scalability; and a programmable engine for flexibility. The first Falcon hardware implementation delivers a peak performance of 200 Gbps, 120 Mops/sec, with near-optimal operation completion times that are up to 8x lower than CX-7 RoCE under network congestion, and up to 65% higher goodput under lossy conditions.

Datacenter workloads have evolved from the data intensive, loosely-coupled workloads of the past decade to more tightly coupled ones, wherein ultra-low latency communication is essential for resource disaggregation over the network and to enable emerging programming models. We introduce Aquila , an experimental datacenter network fabric built with ultra-low latency support as a first-class de-sign goal, while also supporting traditional datacenter traffic. Aquila uses a new Layer 2 cell-based protocol, GNet, an integrated switch, and a custom ASIC with low-latency Remote Memory Access (RMA) capabilities co-designed with GNet. We demonstrate that Aquila is able to achieve under 40 µ s tail fabric Round Trip Time (RTT) for IP traffic and sub-10 µ s RMA execution time across hundreds of host machines, even in the presence of background throughput-oriented IP traffic. This translates to more than 5x reduction in tail latency for a production quality key-value store running on a prototype Aquila network.

Remote Direct Memory Access (RDMA) plays a key role in supporting performance-hungry datacenter applications. However, existing RDMA technologies are ill-suited to multi-tenant datacenters, where applications run at massive scales, tenants require isolation and security, and the workload mix changes over time. Our experiences seeking to operationalize RDMA at scale indicate that these ills are rooted in standard RDMA's basic design attributes: connectionorientedness and complex policies baked into hardware. We describe a new approach to remote memory access -- One-Shot RMA (1RMA) -- suited to the constraints imposed by our multi-tenant datacenter settings. The 1RMA NIC is connection-free and fixed-function; it treats each RMA operation independently, assisting software by offering fine-grained delay measurements and fast failure notifications. 1RMA software provides operation pacing, congestion control, failure recovery, and inter-operation ordering, when needed. The NIC, deployed in our production datacenters, supports encryption at line rate (100Gbps and 100M ops/sec) with minimal performance/availability disruption for encryption key rotation.

Clock synchronization is critical for many datacenter applications such as distributed transactional databases, consistent snapshots, and network telemetry. As applications have increasing performance requirements and datacenter networks get into ultra-low latency, we need submicrosecond-level bound on time-uncertainty to reduce transaction delay and enable new network management applications (e.g., measuring one-way delay for congestion control). The state-of-the-art clock synchronization solutions focus on improving clock precision but may incur significant time-uncertainty bound due to the presence of failures. This significantly affects applications because in large-scale datacenters, temperature-related, link, device, and domain failures are common. We present Sundial, a fault-tolerant clock synchronization system for datacenters that achieves ∼100ns time-uncertainty bound under various types of failures. Sundial provides fast failure detection based on frequent synchronization messages in hardware. Sundial enables fast failure recovery using a novel graphbased algorithm to precompute a backup plan that is generic to failures. Through experiments in a >500-machine testbed and large-scale simulations, we show that Sundial can achieve ∼100ns time-uncertainty bound under different types of failures, which is more than two orders of magnitude lower than the state-of-the-art solutions. We also demonstrate the benefit of Sundial on applications such as Spanner and Swift congestion control.

Light Peak is a new optical interconnect technology developed by Intel and targeted at connecting computing and consumer electronic devices. It provides bandwidth of 10Gbps and higher using optical fiber technology, but at extremely low cost. This paper investigates whether Light Peak technology can be leveraged to form the basis of a low-cost, high performance, scalable network. This paper presents our feasibility analysis, supported by a prototype network we constructed. The prototype consisted of a number of Light Peak PCI-Express cards, each with one host interface, an integrated switch, and transceiver pair with four optical ports. With very small forwarding delays, Light Peak supports direct networks with topologies that have interesting traffic characteristics suitable for small-scale clusters (containers with 1,000-10,000 servers).

Network processors employ a multithreaded, chip-multiprocessing architecture to effectively hide memory latency and deliver high performance for packet processing applications. In such a parallel paradigm, when multiple threads modify a shared variable in the external memory, the threads should be properly synchronized such that the accesses to the shared variable are protected by critical sections. Therefore, in order to efficiently harness the performance potential of network processors, it is critical to hide the memory latency and synchronization latency in multi-threading and multiprocessing. In this paper, we present a novel program transformation used in the Intelreg Auto-partitioning C Compiler for IXP, which perform optimal placement of memory access instructions and synchronization instructions for effective latency hiding. Experimental results show that the transformation provides impressive speedup (up-to to 8.5x) and scalability (up- to 72 threads) of the performance for the real-world network application (a 10Gbps Ethernet Core/Metro Router).

Network processors (NPs) are an emerging field of programmable processors that are optimized to implement data plane packet processing networking functions. Unlike the general-purpose CPUs that rely heavily on caching for improving performance, the lack of locality in packet processing and need for high-performance I/O have forced designers to come up with innovative architectures that can hide memory latency while still processing packets at high data rates. Most of these NPs use some type of multiprocessing in combination with a hierarchy of memory types to achieve high performance. In addition, to keep up with packets arriving at high data rates over multiple incoming media interfaces, an NP must perform fast I/O and memory operations such as packet storage, table lookup, and extraction of fields in packet headers. We describe an architecture that uses a combination of distributed memory architecture and one or more multithreaded processors to achieve the necessary performance. We describe the challenges in programming such a processor including the issues related to consistency and maintaining packet ordering. We also present a programming model for generic network applications that uses software pipelines. We then demonstrate the use of the programming model in implementing two applications, namely, mapping traffic management algorithms onto a multithreaded architecture and an implementation of a media gateway based on voice-over-AAL2.

Publisher Summary Next generation access and edge equipment requires flexible programming, high performance, low power consumption, and small real estate. In context to this, Intel has developed a next-generation network processor, the IXP2400, which is optimized to meet these requirements. The Intel IXP2400 network processor delivers a new level of intelligence and performance for access and edge applications, enabling the realization of quality of service (QoS), enforcement of service-level agreements (SLAs), and traffic engineering at OC-48/2 .5 Gbps and 4 Gbps data rates. The flexible media interface allows a variety of media devices, ranging from OC3 to OC48 speeds to be connected without logic to the IXP2400 for easier design and lower system cost. These capabilities essentially allow OEMs and service providers to offer differentiated and tiered services to their customers while efficiently managing their network resources and bandwidth. The performance and flexibility of the IXP2400 makes it desirable for a wide variety of high-performance applications such as multiservice switches, DSLAMs (DSL access multiplexers), CMTS (cable modem termination system) equipment, 2.5G and 3G wireless infrastructure, and layer 4-7 switches.

Next generation access and edge equipment requires flexible programming, high performance, low power consumption, and small real estate. In context to this, Intel has developed a next-generation network processor, the IXP2400, which is optimized to meet these requirements. The Intel IXP2400 network processor delivers a new level of intelligence and performance for access and edge applications, enabling the realization of quality of service (QoS), enforcement of service-level agreements (SLAs), and traffic engineering at OC-48/2 .5 Gbps and 4 Gbps data rates. The flexible media interface allows a variety of media devices, ranging from OC3 to OC48 speeds to be connected without logic to the IXP2400 for easier design and lower system cost. These capabilities essentially allow OEMs and service providers to offer differentiated and tiered services to their customers while efficiently managing their network resources and bandwidth. The performance and flexibility of the IXP2400 makes it desirable for a wide variety of high-performance applications such as multiservice switches, DSLAMs (DSL access multiplexers), CMTS (cable modem termination system) equipment, 2.5G and 3G wireless infrastructure, and layer 4-7 switches.