The workloads that run on the cloud are getting increasingly diverse and require customization for better performance and energy efficiency. On the other hand, general purpose and high performance Xeon® processors are ubiquitous in cloud and data centers. Barriers to Dennards scaling and emergence of the dark silicon issues are forcing architects to look into adding application-specific accelerators to squeeze as much energy efficiency and performance as possible. In such a scenario FPGAs provide an optimal middle ground not only with performance and energy efficiency but also as a highly customizable hardware accelerator. As Intel® couples high performance Xeon® processors with highly customizable and energy efficient FPGA, it provides unique challenges and opportunities for mapping cloud scale workloads on these heterogeneous architectures. In this paper, the challenges of enabling a popular big data framework called Apache Spark for the first generation Intel® Xeon® with integrated FPGA are presented. The FPGA acceleration capability of a big data application on the integrated platform through Intel® Data Analytics and Acceleration Libraries are exposed. A novel load balancing scheme for integrated FPGA is presented that allows programmers to view an FPGA accelerator as general purpose compute resource. In doing so, the complexity of programming the FPGA is completely abstracted from the programmer. A fully functional end-to-end inference application with FPGA acceleration, measuring speedup of $1.5 \mathrm{x} /$ socket as seen from Apache Spark when compared to the optimized Xeon® implementation is shown.

Remote procedure calls are the workhorse of distributed systems. However, as software engineering trends, such as micro-services and serverless computing, push applications towards ever finer-grained decompositions, the overhead of RPC-based communication is becoming too great to bear. In this paper, we argue that point solutions that attempt to optimize one aspect of RPC logic are unlikely to mitigate these ballooning communication costs. Rather, we need a dramatic reappraisal of how we provide communication. Towards this end, we propose to emulate message-passing RPCs by sharing message payloads and metadata on CXL 3.0-backed far memory. We provide initial evidence of feasibility and analyze the expected benefits.

Resource sharing between multiple workloads has become a prominent practice among cloud service providers, motivated by demand for improved resource utilization and reduced cost of ownership. Effective resource sharing, however, remains an open challenge due to the adverse effects that resource contention can have on high-priority, user-facing workloads with strict Quality of Service (QoS) requirements. Although recent approaches have demonstrated promising results, those works remain largely impractical in public cloud environments since workloads are not known in advance and may only run for a brief period, thus prohibiting offline learning and significantly hindering online learning. In this paper, we propose RAPID, a novel framework for fast, fully-online resource allocation policy learning in highly dynamic operating environments. RAPID leverages lightweight QoS predictions, enabled by domain-knowledge-inspired techniques for sample efficiency and bias reduction, to decouple control from conventional feedback sources and guide policy learning at a rate orders of magnitude faster than prior work. Evaluation on a real-world server platform with representative cloud workloads confirms that RAPID can learn stable resource allocation policies in minutes, as compared with hours in prior state-of-the-art, while improving QoS by 9.0x and increasing best-effort workload performance by 19-43%.

Over the last 25 years, the use of caches has advanced significantly in mainstream microprocessors to address the memory wall challenge. As we transformed microprocessors from single-core to multicore to manycore, innovations in the architecture, design, and management of on-die cache hierarchy were critical to enabling scaling in performance and efficiency. In addition, at the system level, as input/output (I/O) devices (e.g., networking) and accelerators (domain-specific) started to interact with general-purpose cores across shared memory, advancements in caching became important as a way of minimizing data movement and enabling faster communication. In this article, we cover some of the major advancements in cache research and development that have improved the performance and efficiency of microprocessor servers over the last 25 years. We will reflect upon several techniques including shared and distributed last-level caches (including data placement and coherence), cache Quality of Service (addressing interference between workloads), direct cache access (placing I/O data directly into CPU caches), and extending caching to off-die accelerators (CXL.cache). We will also outline potential future directions for cache research and development over the next 25 years.

Microservices are becoming the defining paradigm of cloud applications, which raises urgent challenges for efficient datacenter management. Guaranteeing end-to-end Service Level Agreement (SLA) while optimizing resource allocation is critical to both cloud service providers and users. However, one application may contain hundreds of microservices, which constitute an enormous search space that is unfeasible to explore exhaustively. Thus, we propose RAMBO, an SLA-aware framework for microservices that leverages multi-objective Bayesian Optimization (BO) to allocate resources and meet performance/cost goals. Experiments conducted on a real microservice workload demonstrate that RAMBO can correctly characterize each microservice and efficiently discover Pareto-optimal solutions. We envision that the proposed methodology and results will benefit future resource planning, cluster orchestration, and job scheduling.

Artificial intelligence and machine learning are experiencing widespread adoption in industry and academia. This has been driven by rapid advances in the applications and accuracy of AI through increasingly complex algorithms and models; this, in turn, has spurred research into specialized hardware AI accelerators. Given the rapid pace of advances, it is easy to forget that they are often developed and evaluated in a vacuum without considering the full application environment. This paper emphasizes the need for a holistic, end-to-end analysis of AI workloads and reveals the "AI tax." We deploy and characterize Face Recognition in an edge data center. The application is an AI-centric edge video analytics application built using popular open source infrastructure and ML tools. Despite using state-of-the-art AI and ML algorithms, the application relies heavily on pre-and post-processing code. As AI-centric applications benefit from the acceleration promised by accelerators, we find they impose stresses on the hardware and software infrastructure: storage and network bandwidth become major bottlenecks with increasing AI acceleration. By specializing for AI applications, we show that a purpose-built edge data center can be designed for the stresses of accelerated AI at 15% lower TCO than one derived from homogeneous servers and infrastructure.

In this paper we present the results of our studies focused on enabling predictable performance for functions executing in scalable, multi-tenant Function-as-a-Service environments. We start by analyzing QoS and performance requirements and use cases from the point of view of End-Users, Developers and Infrastructure Owners. Then we take a closer look at functions’ resource utilization patterns and investigate functions’ sensitivity to those resources. We specifically focus on the CPU microarchitecture resources as they have significant impact on functions’ overall performance. As part of our studies we have conducted experiments to research the effect of co-locating different functions on the compute nodes. We discuss the results and provide an overview of how we have further modified the scheduling logic of our containers orchestrator (Kubernetes), and how that impacted functions’ execution times and performance variation. We have specifically leveraged the low-level telemetry data, mostly exposed by the Intel® Resource Director Technology (Intel® RDT) [1]. Finally, we provide an overview of our future studies, which will be centered around node-level resource allocations, further improving a function’s performance, and conclude with key takeaways.

Artificial intelligence and machine learning are experiencing widespread adoption in the industry, academia, and even public consciousness. This has been driven by the rapid advances in the applications and accuracy of AI through increasingly complex algorithms and models; this, in turn, has spurred research into developing specialized hardware AI accelerators. The rapid pace of the advances makes it easy to miss the forest for the trees: they are often developed and evaluated in a vacuum without considering the full application environment in which they must eventually operate. In this paper, we deploy and characterize Face Recognition, an AI-centric edge video analytics application built using open source and widely adopted infrastructure and ML tools. We evaluate its holistic, end-to-end behavior in a production-size edge data center and reveal the "AI tax" for all the processing that is involved. Even though the application is built around state-of-the-art AI and ML algorithms, it relies heavily on pre-and post-processing code which must be executed on a general-purpose CPU. As AI-centric applications start to reap the acceleration promised by so many accelerators, we find they impose stresses on the underlying software infrastructure and the data center's capabilities: storage and network bandwidth become major bottlenecks with increasing AI acceleration. By not having to serve a wide variety of applications, we show that a purpose-built edge data center can be designed to accommodate the stresses of accelerated AI at 15% lower TCO than one derived from homogeneous servers and infrastructure. We also discuss how our conclusions generalize beyond Face Recognition as many AI-centric applications at the edge rely upon the same underlying software and hardware infrastructure.

We are witnessing an explosion of AI based use cases driving the computer industry, and especially datacenter and server architectures. As Intel faces fierce competition in this emerging technology space, it is important that architecture definitions and directions are driven with data from proper tools and methodologies, and insights are drawn from end-to-end holistic analysis at the datacenter levels. In this paper, we introduce DeepSim, a cluster-level behavioral simulation model for deep learning. DeepSim, which is based on the Intel CoFluent simulation framework, uses timed behavioral models to simulate complex interworking between compute nodes, networking, and storage at the datacenter level, providing a realistic performance model of a real-world image recognition applications based on the popular Deep Learning Framework Caffe. The end-to-end simulation data from DeepSim provides insight which can be used for architecture analysis driving future datacenter architecture directions. DeepSim enables scalable system design, deployment, and capacity planning through accurate performance insights. Results from preliminary scaling studies (e.g. node scaling and network scaling) and what-if analyses (e.g., Xeon with HBM and Xeon Phi with dual OPA) are presented in this paper. The simulation results are correlated well with empirical measurements, achieving an accuracy of 95%.

Artificial intelligence and machine learning are experiencing widespread adoption in industry and academia. This has been driven by rapid advances in the applications and accuracy of AI through increasingly complex algorithms and models; this, in turn, has spurred research into specialized hardware AI accelerators. Given the rapid pace of advances, it is easy to forget that they are often developed and evaluated in a vacuum without considering the full application environment. This article emphasizes the need for a holistic, end-to-end analysis of artificial intelligence (AI) workloads and reveals the “AI tax.” We deploy and characterize Face Recognition in an edge data center. The application is an AI-centric edge video analytics application built using popular open source infrastructure and machine learning (ML) tools. Despite using state-of-the-art AI and ML algorithms, the application relies heavily on pre- and post-processing code. As AI-centric applications benefit from the acceleration promised by accelerators, we find they impose stresses on the hardware and software infrastructure: storage and network bandwidth become major bottlenecks with increasing AI acceleration. By specializing for AI applications, we show that a purpose-built edge data center can be designed for the stresses of accelerated AI at 15% lower TCO than one derived from homogeneous servers and infrastructure.

This is an educational white paper on transfer learning, showcasing how existing deep learning models can be easily and flexibly customized to solve new problems. One of the biggest challenges with deep learning is the large number of labeled data points that are required to train the deep learning models them to sufficient accuracy. For example, the ImageNet* [20] database for image recognition consists of over 14 million hand-labeled images. While the number of possible applications of deep learning systems in vision tasks, text processing, speech-to-text translation and many other domains is enormous, very few potential users of deep learning systems have sufficient training data to create models from scratch. A common concern among teams considering the use of deep learning to solve business problems is the need for training data: “Doesn’t deep learning need millions of samples and months of training to get good results?” One powerful solution is transfer learning, in which part of an existing deep learning model is re-optimized on a small data set to solve a related, but new, problem. In fact, one of the great attractions of transfer learning is that, unlike most traditional approaches to machine learning, we can take models trained on one (perhaps very large) dataset and modify them quickly and easily to work well on a new problem (where perhaps we have only a very small dataset). Transfer learning methods are not only parsimonious in their training data requirements, but they run efficiently on the same Intel® Xeon® CPUbased systems that are widely used for other analytics workloads, including machine learning and deep learning inference. The abundance of readily available CPU capacity in current datacenters, in conjunction with transfer learning, makes CPU-based systems the preferred choice for deep learning training and inference.

Quality of service (QoS) management is widely employed to provide differentiable performance to programs with distinctive priorities on conventional chip multi-processor (CMP) platforms. Recently, heterogeneous architecture integrating diverse processor cores on the same silicon has been proposed to better serve various application domains and it is expected to be an important design paradigm of future processors. Therefore, the QoS management on emerging heterogeneous systems will be of great significance. On the other hand, parallel applications are becoming increasingly important in modern computing community in order to explore the benefit of thread-level parallelism on CMPs. However, considering the diverse characteristics of thread synchronization, data sharing, and parallelization pattern, governing the execution of multiple parallel programs with different performance requirements becomes a complicated yet significant problem. In this paper, we study QoS management for parallel applications running on heterogeneous CMP systems. We comprehensively assess a series of task-to-core mapping policies on a real heterogeneous hardware (QuickIA) by characterizing their impacts on performance of individual applications. Our evaluation results show that the proposed QoS policies are effective to improve the performance of programs with highest priority while striking good tradeoff with system fairness.

As programs for microprocessor architectures, network-on-chip (NoC) traffic patterns are essential tools for NoC performance assessment and design exploration. The fidelity of NoC traffic patterns has profound influence on NoC studies. In this paper, we present a systematic traffic modeling and generation methodology and a traffic suite for efficient evaluation of NoC-based many-core systems. The publicly released MCSL (multi-constraint system-level) traffic suite includes a set of realistic traffic patterns for real-world applications and covers popular NoC architectures. It captures both the communication behaviors in NoCs and the temporal dependencies among them. The MCSL traffic suite can be easily incorporated into existing NoC simulators and significantly improve NoC simulation accuracy. The proposed methodology uses formal computational models to capture both communication and computation requirements of applications. It optimizes application memory requirements, mapping, and scheduling to maximize overall system performance and utilization before extracting traffic patterns through cycle level simulations. Experiment results show that the MCSL traffic suite can be used to study NoC characteristics more accurately than traditional random traffic patterns.

Over the last decade, addressing quality of service (QoS) in multi-core server platforms has been growing research topic. QoS techniques have been proposed to address the shared resource contention between co-running applications or virtual machines in servers and thereby provide better isolation, performance determinism and potentially improve overall throughput. One of the most important shared resources is cache space. Most proposals for addressing shared cache contention are based on simulations and analysis and no commercial platforms were available that integrated such techniques and provided a practical solution. In this paper, we will present the first set of shared cache QoS techniques designed and implemented in state-of-the-art commercial servers (the Intel® Xeon® processor E5-2600 v3 product family). We will describe two key technologies: (i) Cache Monitoring Technology (CMT) to enable monitoring of shared cache usage by different applications and (ii) Cache Allocation Technology (CAT) which enables redistribution of shared cache space between applications to address contention. This is the first paper to describing these techniques as they moved from concept to reality, starting from early research to product implementation. We will also present case studies highlighting the value of these techniques using example scenarios of multi-programmed workloads, virtualized platforms in datacenters and communications platforms. Finally, we will describe initial software infrastructure and enabling for industry practitioners and researchers to take advantage of these technologies for their QoS needs.

The Network-on-chip (NoC) based multiprocessor system-on-chip (MPSoCs) is becoming a promising architecture to meet modern applications' ever-increasing demands for computing capability under limited power budget. NoC traffic patterns are essential tools for NoC performance assessment and architecture design exploration. In this paper, we present a systematic NoC traffic modeling and generation methodology and a set of realistic NoC traffic patterns called MCSL, which are generated through the methodology. The proposed methodology can faithfully capture both the communication behaviors of real applications in NoCs and the temporal dependencies among them. And it optimizes application memory requirements, mapping and scheduling to maximize overall system performance and utilization before extracting traffic patterns through cycle-level simulations. Extensive experiments are conducted to verify the effectiveness of the methodology, and evaluate the performance of the generated traffic patterns. The results show that the MCSL traffic patterns can be used to study NoC characteristics more accurately than traditional random traffic patterns.