AI supercomputer network performance has become a blocker to training larger AI models. Analytically modeling such networks is virtually impossible, and the cost and time of building prototypes is prohibitive, making simulation the only real option. Simulating an 8,192-node AI cluster for one second, however, using existing network simulators requires over a week for a single run. In this paper, we propose Miniature, an FPGA-based network simulator that simulates network components using specialized circuits, achieving both fast simulation and efficient resource utilization. Miniature further provides an efficient multiplexing architecture to automatically fit large AI supercomputer networks on given number of FPGA(s). We implement a Miniature prototype running AI traffic. Our measurements demonstrate that Miniature can simulate a 65,536 node AI-cluster 4332 x faster than state-of-the-art softwarebased network simulators on a single FPGA.

Cloud providers offer storage-optimized VMs equipped with locally attached storage to meet the high performance requirements of cloud users. However, current cloud providers cannot enable live migration for storage-optimized VMs due to the high resource overheads. Moreover, resources should be permanently provisioned for live migration as on-demand provisioning needs to de-allocate resources from either VMs or the hypervisor, thus violating SLA. We propose a storage live migration acceleration system on SmartNICs. Our design achieves minimal resource overhead and SLA violations by proposing (1) a SmartNIC-managed live migration architecture and (2) an efficient consistency algorithm. We implement a basic prototype on an FPGA-based SmartNIC. Preliminary results show that we can migrate storage-optimized VMs with no host resource usage and minimal performance interference to RocksDB running inside the VM. This project is part of the Terminus Project [28].

Overlay processors on FPGAs enable i) software programmability through sequential code calling library functions, ii) high performance by converting the library calls to invocations of corresponding accelerators, and iii) faster deployment than reprogramming the FPGA. Traditionally, overlays have been hand-written in RTL and programmed through handwritten assembly. We present the Primate framework, which automatically generates overlays from applications written in annotated C++. We evaluated Primate on Whippersnapper (Dang et al. 2017) P4 benchmarks. Primate Overlay latencies are 0.06x - 0.15x compared to PISCES (Shahbaz et al. 2016), a high-performance CPU solution, and 0.25x - 2.3x compared to solutions generated by P4FPGA (Wang et al. 2017), a P4 HLS compiler on FPGA.

With the increasing probability of software errors being caused by hardware transient faults, hardware engineers require better methods to evaluate the impact of faults and determine how to harden their products against them. This paper presents novel techniques to accurately and efficiently simulate faults on processors by dynamically switching, with little designer effort, a simulation from RTL-level-or-lower to instruction-level. When combined with known techniques, the proposed techniques make fault modeling orders-of-magnitude faster than RTL simulation while achieving the same level of accuracy, enabling fast identification of the portions of the design that would benefit most from hardening, and providing statistical guarantees of a design's resistance to faults.

Conventional wisdom says cloud-scale deployment (where every server has an FPGA) is not practical due to power, performance, and cost. This paper describes when it makes sense to deploy FPGAs in the cloud and some system aspects of doing so. Though this paper is written with an FPGA focus, many of the cloud infrastructure observations are applicable to non-FPGAs as well.

Una primera tarjeta de interfaz de red en linea, NIC (104), para indexar los flujos de red (106), comprendiendo la primera NIC en linea (104): un primer controlador de acceso a los medios, MAC (150); un segundo MAC (152); hardware de procesamiento (124) configurado para proporcionar la transmision de paso (154, 156) de los paquetes de los flujos de red (106) mediante la transmision de los paquetes del primer MAC (150) recibidos por el segundo MAC (152) y mediante la transmision de los paquetes del segundo MAC (152) recibidos por el primer MAC (150); un primer modulo (130) configurado para implementar un protocolo de transporte ligero, LTP; y un segundo modulo (132) configurado para comunicarse con una segunda NIC en linea arbitraria por medio del primer modulo (130) especificando una direccion de red correspondiente a la segunda NIC en linea para permitir que el primer modulo (130) establezca una conexion LTP (108) con puntos finales en la primera NIC en linea (104) y en la segunda NIC en linea, en donde la primera NIC en linea (104) se conecta a un primer anfitrion y a la red de datos, y la segunda NIC en linea se conecta a un segundo anfitrion y a la red de datos, en donde el hardware de procesamiento se configura para proporcionar conectividad de red entre el primer anfitrion y el segundo anfitrion, realizando una transmision de paso de los paquetes recibidos que se ha determinado que no son paquetes LTP entre NIC, y proporcionar conectividad LTP entre NIC entre la primera NIC en linea (104) y la segunda NIC en linea, y en donde los paquetes recibidos por la primera NIC en linea que se determina que son paquetes LTP entre IC se consumen por la primera NIC en linea (104) y no se reenvian al primer o segundo anfitrion mediante la primera NIC en linea, y en donde los paquetes LTP entre NIC se originan por la primera o segunda NIC en linea.

”Soft” GPUs are overlays that implement GPGPU-like data parallel processor architectures in FPGA logic to make FPGAs as software-programmable as ”hard” GPGPUs. Unlike hard GPUs, soft GPU architectures can be specialized to further improve efficiency by leveraging FPGA’s flexibility. Prior work has shown the software programmability potential for soft GPUs but only studied general-purpose soft GPUs with minor specializations (e.g., FPGU, FlexGrip, MIAOW, and SCRATCH) or only domain-optimized for a particular application domain (e.g., PDL-FGPU for the persistent deep learning domain.) This paper proposes a soft GPU development framework to automate the creation of soft GPU instances with aggressive application-domain optimizations (i.e., domain-optimized GPUs, or DOGPUs) that consists of a baseline general soft GPU architecture ”template” with an improved architecture over prior general purpose soft GPUs, along with a customizable partition that enables a custom datapath (macro unit) to be inserted to optimize for a target application domain. Unlike the prior PDL-FGPU which targets the persistent deep learning domain, the proposed framework can be used to target optimization for any application domain. Our evaluation on a set of data parallel workloads shows that (i) the proposed general soft GPU architecture offers average speedup of 1.8x versus the best prior soft GPUs we know of (i.e., FGPU, PDL-FGPU), (ii) DO-GPUs with domain-optimizations provide an average of 218x speedup over general soft GPUs, (iii) the proposed framework enabled building six new domain-optimized soft GPU instances in a matter of days, and (iv) enables quick GPU-like development effort (hours), where code is concise (low 100s of lines) and can be compiled in seconds without FPGA EDA tools in the loop, assuming an appropriate soft DO-GPU bitstream for the application domain is already built.

Domain-specific accelerators achieve high performance and efficiency at the cost of reduced adaptivity to functionality changes, i.e., flexibility. Balancing efficiency and flexibility is important for accelerator designing. There is not, however, a commonly accepted metric for flexibility. We propose FlexScore as a flexibility metric based on the relationship between flexibility and goodness metrics, such as performance. FlexScore does not incorporate domain-specific or hardware-specific knowledge and, thus, is applicable to general accelerator design. We present FlexScores of three well-known DNN accelerators and observe that there is 45 percent FlexScore difference among them and show how architectural changes could improve FlexScore by up to 21 percent, demonstrating the usefulness of FlexScore in evaluating, comparing, and trading-off different architectures.

Overlay architectures are a good way to enable fast development and debug on FPGAs at the expense of potentially limited performance compared to fully customized FPGA designs. When used in concert with hand-tuned FPGA solutions, performant overlay architectures can improve time-to-solution and thus overall productivity of FPGA solutions. This work tunes and specializes FGPU, an open source OpenCL-programmable GPU overlay for FPGAs. We demonstrate that our persistent deep learning (PDL)-FGPU architecture maintains the ease-of-programming and generality of GPU programming while achieving high performance from specialization for the persistent deep learning domain. We also propose an easy method to specialize for other domains. PDL-FGPU includes new instructions, along with micro-architecture and compiler enhancements. We evaluate both the FGPU baseline and the proposed PDL-FGPU on a modern high-end Intel Stratix 10 2800 FPGA in simulation running persistent DL applications (RNN, GRU, LSTM), and non-DL applications to demonstrate generality. PDL-FGPU requires 1.4–3× more ALMs, 4.4–6.4× more M20ks, and 1–9.5× more DSPs than baseline, but improves performance by 56–693× for PDL applications with an average 23.1% degradation on non-PDL applications. We integrated the PDL-FGPU overlay into Intel OPAE to measure real-world performance/power and demonstrate that PDL-FGPU is only 4.0–10.4× slower than the Nvidia V100.

As semiconductor technology continues to scale to smaller dimensions and as power-consumption of processor chips continues to increase in importance, there is significant uncertainty about the reliability of future large-scale computers. This uncertainty impacts algorithm design and application development, as well as decisions about system operation policy and provisioning decisions. The Veracity endeavored to answer pertinent questions regarding system reliability and resilience with the goal of enabling reliable science on future leadership machines while keeping overprovisioning low.

Un metodo para reconfigurar parcialmente un componente de aceleracion de hardware programado con un rol (1303, 1304) y una interfaz (1306) de red, el rol enlazado a uno o mas de: un rol de flujo descendente en el componente (1301) de aceleracion vecino del flujo descendente y un rol de flujo ascendente en el componente (1301) de aceleracion vecino del flujo ascendente para componer un grafico, el metodo que comprende: detectar una razon (1321) para cambiar el rol durante la monitorizacion del componente de aceleracion para comportamientos incorrectos; detener (1322) el rol (1303, 1304) que incluye instrucciones al menos de: el rol del flujo descendente y el rol del flujo ascendente para parar la recepcion de datos desde el rol; reconfigurar parcialmente el componente (1301) de aceleracion de hardware mediante la escritura de una imagen (1312) para el rol (1303, 1304) desde una ubicacion de almacen de imagenes al componente (1301) de aceleracion; mantener la interfaz (1306) de red como operativa durante la reconfiguracion parcial del componente (1301) de aceleracion para permitir que un segundo rol programado en el componente de aceleracion intercambie comunicacion de red con uno o mas otros roles compuestos en otro grafico en otros componentes de aceleracion; y activar (1326) el rol (1303, 1304) en uno de los componentes (1301) de aceleracion del flujo ascendente o del flujo descendente despues de que la reconfiguracion parcial del componente (1301) de aceleracion se complete que incluye notificar a al menos uno de: el rol del flujo descendente y el rol del flujo ascendente de que el rol esta operativo.

FPGAs have been deployed at massive scale in data centers. Using currently available communication architectures, however, it is difficult for FPGAs to access and utilize the various heterogenous resources available in data centers (DRAM, CPU, GPU,...). In this paper, we present Direct Universal Access (DUA), a communication architecture that provides uniform access for FPGA to these data center resources. Without being limited by machine boundaries, DUA provides global names and a common interface for communicating across various resources, the underlying network automatically routing traffic and managing resource multiplexing. Our benchmarks show that DUA provides simple and fair-share resource access with small logic area overhead (< 10%) and negligible latency (< 0.2 mu s). We also build two practical multi-FPGA applications-deep crossing and regular expression matching-on top of DUA to demonstrate its usability and efficiency.

A smart NIC (Network Interface Card) is provided with features to enable the smart NIC to operate as an in-line NIC between a host's NIC and a network. The smart NIC provides pass-through transmission of network flows for the host. Packets sent to and from the host pass through the smart NIC. As a pass-through point, the smart NIC is able to accelerate the performance of the pass-through network flows by analyzing packets, inserting packets, dropping packets, inserting or recognizing congestion information, and so forth. In addition, the smart NIC provides a lightweight transport protocol (LTP) module that enables it to establish connections with other smart NICs. The LTP connections allow the smart NICs to exchange data without passing network traffic through their respective hosts.

Web search engines deploy large-scale selection services on CPUs to identify a set of web pages that match user queries. An FPGA-based accelerator can exploit various levels of parallelism and provide a lower latency, higher throughput, more energy-efficient solution than commodity CPUs. However, maintaining such a customized accelerator in a commercial search engine is challenging because selection services are changed often. This article presents our design for FlexSaaS (Flexible Selection as a Service), an FPGA-based accelerator for web search selection. To address efficiency and flexibility challenges, FlexSaaS abstracts computing models and separates memory access from computation. Specifically, FlexSaaS (i) contains a reconfigurable number of matching processors that can handle various possible query plans, (ii) decouples index stream reading from matching computation to fetch and decode index files, and (iii) includes a universal memory accessor that hides the complex memory hierarchy and reduces host data access latency. Evaluated on FPGAs in the selection service of a commercial web search--the Bing web search engine—FlexSaaS can be evolved quickly to adapt to new updates. Compared to the software baseline, FlexSaaS on Arria 10 reduces average latency by 30% and increases throughput by 1.5×.

Power has become a fundamental limit to silicon performance. Most research has focused on reducing transistor switching to constrain power (dark silicon.) Specialized accelerators have been proposed since they implement functionality with fewer transistor switches than general purpose cores. Increasing efficiency requirements lead to more specialization and, therefore, more accelerators that potentially leads to longer distances to get to all the accelerators. Communication, however, consumes energy, and therefore needs to be minimized as well (dark wires.) This paper examines the balance between compute and communication specialization in the context of hard logic (e.g., ASIC) that is highly efficient but static versus soft logic (e.g., FPGA) that is less efficient but allows computation to be moved to reduce communication distances. Our experimental results show using soft accelerators consumes 0.6x-2.1x total power compared to using hard accelerators when communication costs are taken into account.

Web search engines deploy large-scale selection services on CPUs to identify a set of web pages that match user queries. An FPGA-based accelerator can exploit various levels of parallelism and provide a lower latency, higher throughput, more energy-efficient solution than commodity CPUs. However, maintaining such a customized accelerator in a commercial search engine is challenging because selection services are changed often. This article presents our design for FlexSaaS (Flexible Selection as a Service), an FPGA-based accelerator for web search selection. To address efficiency and flexibility challenges, FlexSaaS abstracts computing models and separates memory access from computation. Specifically, FlexSaaS (i) contains a reconfigurable number of matching processors that can handle various possible query plans, (ii) decouples index stream reading from matching computation to fetch and decode index files, and (iii) includes a universal memory accessor that hides the complex memory hierarchy and reduces host data access latency. Evaluated on FPGAs in the selection service of a commercial web search--the Bing web search engine—FlexSaaS can be evolved quickly to adapt to new updates. Compared to the software baseline, FlexSaaS on Arria 10 reduces average latency by 30% and increases throughput by 1.5×.

Provides an abstract for each of the invited presentations and may include a brief professional biography of each presenter. The complete presentations were not made available for publication as part of the conference proceedings.

To meet the computational demands required of deep learning, cloud operators are turning toward specialized hardware for improved efficiency and performance. Project Brainwave, Microsofts principal infrastructure for AI serving in real time, accelerates deep neural network (DNN) inferencing in major services such as Bings intelligent search features and Azure. Exploiting distributed model parallelism and pinning over low-latency hardware microservices, Project Brainwave serves state-of-the-art, pre-trained DNN models with high efficiencies at low batch sizes. A high-performance, precision-adaptable FPGA soft processor is at the heart of the system, achieving up to 39.5 teraflops (Tflops) of effective performance at Batch 1 on a state-of-the-art Intel Stratix 10 FPGA.