This paper introduces two extensions to the popular PyTorch machine learning framework, TorchDynamo and TorchInductor, which implement the torch.compile feature released in PyTorch 2. TorchDynamo is a Python-level just-in-time (JIT) compiler that enables graph compilation in PyTorch programs without sacrificing the flexibility of Python. It achieves this by dynamically modifying Python bytecode before execution and extracting sequences of PyTorch operations into an FX graph, which is then JIT compiled using one of many extensible backends. TorchInductor is the default compiler backend for TorchDynamo, which translates PyTorch programs into OpenAI's Triton for GPUs and C++ for CPUs. Results show that TorchDynamo is able to capture graphs more robustly than prior approaches while adding minimal overhead, and TorchInductor is able to provide a 2.27x inference and 1.41x training geometric mean speedup on an NVIDIA A100 GPU across 180+ real-world models, which outperforms six other compilers. These extensions provide a new way to apply optimizations through compilers in eager mode frameworks like PyTorch.

AI is a foundational technology at Meta: we use AI to identify relevant and interesting content our users delight at interacting with, to translate content to transcend language barriers, and to keep our communities safe by identifying inappropriate content, such as bullying, domestic violence and terrorism in images, videos, and text. The need for ever higher quality models intersects with the imperative of keeping AI growth sustainable. While academic research has emphasized an exponential resource growth of models to deliver quality, we must bend the curve to ensure sustainable growth and minimize environmental impact. As we are looking for ever higher quality, larger scale models to deliver on our mission to connect users and build safe communities, AI accelerators provide the foundation for scaling up quality, while keeping power consumption manageable and sustainable, and delivering on our sustainability commitments.

In this paper, we provide a deep dive into the deployment of inference accelerators at Facebook. Many of our ML workloads have unique characteristics, such as sparse memory accesses, large model sizes, as well as high compute, memory and network bandwidth requirements. We co-designed a high-performance, energy-efficient inference accelerator platform based on these requirements. We describe the inference accelerator platform ecosystem we developed and deployed at Facebook: both hardware, through Open Compute Platform (OCP), and software framework and tooling, through Pytorch/Caffe2/Glow. A characteristic of this ecosystem from the start is its openness to enable a variety of AI accelerators from different vendors. This platform, with six low-power accelerator cards alongside a single-socket host CPU, allows us to serve models of high complexity that cannot be easily or efficiently run on CPUs. We describe various performance optimizations, at both platform and accelerator level, which enables this platform to serve production traffic at Facebook. We also share deployment challenges, lessons learned during performance optimization, as well as provide guidance for future inference hardware co-design.

This paper explores the environmental impact of the super-linear growth trends for AI from a holistic perspective, spanning Data, Algorithms, and System Hardware. We characterize the carbon footprint of AI computing by examining the model development cycle across industry-scale machine learning use cases and, at the same time, considering the life cycle of system hardware. Taking a step further, we capture the operational and manufacturing carbon footprint of AI computing and present an end-to-end analysis for what and how hardware-software design and at-scale optimization can help reduce the overall carbon footprint of AI. Based on the industry experience and lessons learned, we share the key challenges and chart out important development directions across the many dimensions of AI. We hope the key messages and insights presented in this paper can inspire the community to advance the field of AI in an environmentally-responsible manner.

The IBM POWER9 processor is the latest Reduced Instruction Set Computer microprocessor from IBM. POWER9 employs a new modular core microarchitecture to counter the technology trend of decreasing frequency and increasing power density from generation to generation. The new POWER9 design enables a family of processors optimized for a broad range of server applications. The new microarchitecture is closely coupled with a rich set of new instructions geared toward data-centric applications. In this paper, we describe the POWER9 core microarchitecture innovations, its new instructions and features, and the exploitation of this new design for computing in the cognitive era.

The IBM POWER9 architecture offers a substantial set of novel and performance-improvement features that are made available to both scale-up and scale-out applications via system software. These features provide significant performance improvements for cognitive, cloud, and virtualization workloads, many of which use dynamic scripting languages. In this paper, we describe some of the key features.

We describe the design and implementation of the new, little-endian Linux on Power operating environment, with a particular emphasis on its application interfaces. The new environment was designed to simplify porting of applications from other processor architectures to Power. It uses little-endian data formats to reduce the need for code adaptation and debugging during porting, and to simplify the use of commodity off-the-shelf system components, such as I/O adapters and accelerators. We took advantage of introducing a new operating environment to optimize the platform application binary interface (ABI) for new programming patterns and paradigms, resulting in enhanced out-of-the-box performance. The new ABI responds to increasing componentization of applications by reducing the overhead involved in function calls with an emphasis on accelerating calls to short functions as well as calls to runtime-resolved functions, i.e., functions in shared libraries, function pointers, virtual methods, and interface functions.

Training deep learning models often occupies entire compute clusters, built solely for this purpose, for days or even weeks at a time. There exists a large body of work on approaches for improving training performance, ranging from novel algorithms to full custom hardware accelerators. Offering compute capabilities of multiple teraflops (trillion floating point operations per second), graphics processing units (GPUs) have established themselves as a de-facto standard for accelerating deep learning network training. As systems with up to 16 GPUs-each GPU consuming up to 300 W-become available, efficient usage of these resources becomes imperative. We conduct a detailed analysis o f deep learning workloads to characterize their efficiency in making use of GPU acceleration. We found that many deep learning workloads consume only a fraction of GPU resources, and we demonstrate how sharing GPU resources can improve throughput by a factor of 3, effectively turning a 4-GPU commodity cloud system into a high-end 12-GPU supercomputer. Using Watson workloads from three major areas that incorporate deep learning technology-i.e., language classification, visual recognition, and speech recognition-we document the effectiveness and scalability of our approach. We are working toward enabling GPU virtualization not only to reduce cost, but also to accelerate new breakthroughs in deep learning by increasing compute capacity without making further hardware investments.

In this paper, we describe the history and development of the IBM POWER® vector-scalar architecture, as well as how the design goals of hardware efficiency and software interoperability are achieved by integrating existing floating-point and vector functions into a new unified architecture and function unit. The vector-scalar instructions were defined with an emphasis on out-of-the-box performance and consumability, while accelerating a broad set of enterprise server workloads. Vector-scalar instructions were first introduced in the IBM POWER7® architecture to accelerate high-performance computing applications. With the introduction of the POWER8® processor, the vector-scalar architecture expanded to accelerate a diverse set of enterprise workloads including unstructured text and string processing, business analytics, in-memory databases, big data, and stream coding. We conclude this paper with a description of workload performance and application acceleration to demonstrate the effectiveness of the new vector-scalar architecture.

This article consists of a collection of slides from the author's conference presentation. POWER8 delivers advanced virtualization for CPU and I/O. POWER8 takes the next step in exploiting system accelerators. Collaborative Innovation based on Open Standards.