Fast and programmable network devices are now readily available, both in the form of programmable switches and smart network-interface cards. Going forward, we envision that these devices will be widely deployed in the networks of cloud providers (e.g., AWS, Azure, and GCP) and exposed as a programmable surface for cloud customers—similar to how cloud customers can today rent CPUs, GPUs, FPGAs, and ML accelerators. Making this vision a reality requires us to develop a mechanism to share the resources of a programmable network device across multiple cloud tenants. In other words, we need to provide multitenancy on these devices. In this position paper, we design compile and run-time approaches to multitenancy. We present preliminary results showing that our design provides both efficient utilization of the resources of a programmable network device and isolation of tenant programs from each other.
The plethora of parameters and nuanced configuration options that govern complex, large-scale caching systems restrict their designers and operators. We analyze cache warmup times that can arise in failure handling, load balancing, and cache partitioning of large-scale distributed memory and storage systems. Through simulation on traces from production CDN and storage systems, we derive rules of thumb formulas for designers and operators to use when reasoning about caches.
The emerging next generation of cloud services like Granular and Serverless computing are pushing the boundaries of the current cloud infrastructure. In order to meet the performance objectives, researchers are now leveraging low-level microarchitectural resources in clouds. At the same time these resources are also a major source of security problems that can compromise the confidentiality and integrity of sensitive data in multi-tenant shared cloud infrastructures. The core of the problem is the lack of isolation due to the unsupervised sharing of microarchitectural resources across different performance and security boundaries. In this paper, we introduce Stratus clouds that treat the isolation on microarchitectural elements as the key design principle when allocating cloud resources. This isolation improves both performance and security, but at the cost of reducing resource utilization. Stratus captures this trade-off using a novel abstraction that we call isolation credit , and show how it can help both providers and tenants when allocating microarchitectural resources using Stratus's declarative interface. We conclude by discussing the challenges of realizing Stratus clouds today.
RDMA is increasingly popular for low-latency communication in datacenters, marking a major change in how we build distributed systems. Unfortunately, as we pursue significant system re-designs inspired by new technology, we have not given equal thought to the consequences for system security. This paper investigates security issues introduced to datacenter systems by switching to RDMA and challenges in building secure RDMA systems. These challenges include changes in RPC reliability guarantees and unauditable data-accesses. We show how RDMA's design makes it challenging to build secure storage systems by analyzing recent research systems; then we outline several directions for solutions and future research, with the goal of securing RDMA datacenter systems while they are still in the research and prototype stages.
Online experimentation is an agile software development practice, which plays a central role in enabling rapid innovation. It helps shorten code delivery cycles, which is critical for companies to survive in a competitive software-driven market. Recent advances in cloud computing, including the maturity of container-based technologies and cloud infrastructure, as well as the advent of service meshes, have created an opportunity to broaden the scope of online experimentation and further increase developers' agility. In this paper, we propose a novel formulation for online experimentation of cloud applications which generalizes traditional approaches applied to web and mobile applications by incorporating the unique challenges posed by cloud environments. To enable practitioners to apply our formulation, we develop and present JACKPOT, a system for online cloud experimentation in the presence of multiple interacting microservices. We discuss an initial prototype of JACKPOT along with a preliminary evaluation of this prototype based on experiments on a public container cloud.
Many services replicate data for fault-tolerant storage of the data and high-availability of the service. When deployed in the cloud, the replication performed by these services provides the desired high-availability but does not provide significant additional fault-tolerance for the data. This is because cloud deployments use fault-tolerant storage services instead of the simple local disks that many replicated data services were designed to use. Because the cloud storage services already provide fault-tolerance for the data, the extra replicas create unnecessary cost in running the service. However, replication is still needed for high-availability of the service itself. In this paper, we explore types of replicated data services and how they can be mapped onto various classes of cloud storage. We then propose a general architectural pattern that can be used to: (1) limit additional storage resulting in monetary cost saving, (2) while keeping the same performance for the service, and (3) maintaining the same high-availability of the services and the durability guarantees for the data. We prototype our approach in two popular open-source replicated data services, Kafka and Cassandra, and show that with relatively little modification these systems can be deployed for a fraction of the storage cost without affecting the availability guarantees, durability guarantees, or performance.
Burstable storage is a public cloud feature that enhances cloud storage volumes with credits that can be used to boost performance temporarily. These credits can be exchanged for increased storage throughput, for a short period of time, and are replenished over time. We examine how burstable storage can be leveraged to reduce cost and/or improve performance for three use cases with different data-longevity requirements: traditional persistent storage, caching, and ephemeral storage. Although cloud storage volumes are typically priced by capacity, we find that each AWS gp2 volume starts with the same number of burst credits. Exploiting that fact, we find that aggressive interchanging of large numbers of small short-term volumes can increase IOPS by up to 100× at a cost increase of only 10-40%. Compared to an AWS io1 volume provisioned for the same performance, such interchanging reduces cost by 97.5%.
To achieve higher utilisation, cloud providers offer VMs with GPUs as lower-cost transient cloud resources. Transient VMs can be revoked at short notice and vary in their availability. This poses challenges to distributed machine learning (ML) jobs, which perform long-running stateful computation in which many workers maintain and synchronise model replicas. With transient VMs, existing systems either require a fixed number of reserved VMs or degrade performance when recovering from revoked transient VMs. We believe that future distributed ML systems must be designed from the ground up for transient cloud resources. This paper describes SPOTNIK, a system for training ML models that features a more adaptive design to accommodate transient VMs: (i) SPOTNIK uses an adaptive implementation of the allreduce collective communication operation. As workers on transient VMs are revoked, SPOTNIK updates its membership and uses the all-reduce ring to recover; and (ii) SPOTNIK supports the adaptation of the synchronisation strategy between workers. This allows a training job to switch between different strategies in response to the revocation of transient VMs. Our experiments show that, after VM revocation, SPOTNIK recovers training within 300 ms for ResNet/ImageNet.
Stream processing as a platform-as-a-service (PaaS) offering is used at LinkedIn to host thousands of business-critical applications. This requires service owners to manage applications' resource sizing and tuning. Unfortunately, applications have diverged from their conventional model of a directed acyclic graph (DAG) of operators and incorporate multiple other functionalities, which presents numerous challenges for sizing. We present a controller that dynamically controls applications' resource sizing while accounting for diverse functionalities, load variations, and service dependencies, to maximize cluster utilization and minimize cost. We discuss the challenges and opportunities in designing such a controller.
Understanding the GPU utilization of Deep Learning (DL) workloads is important for enhancing resource-efficiency and cost-benefit decision making for DL frameworks in the cloud. Current approaches to determine DL workload GPU utilization rely on online profiling within isolated GPU devices, and must be performed for every unique DL workload submission resulting in resource under-utilization and reduced service availability. In this paper, we propose a prediction engine to proactively determine the GPU utilization of heterogeneous DL workloads without the need for in-depth or isolated online profiling. We demonstrate that it is possible to predict DL workload GPU utilization via extracting information from its model computation graph. Our experiments show that the prediction engine achieves an RMSLE of 0.154, and can be exploited by DL schedulers to achieve up to 61.5% improvement to GPU cluster utilization.
The more we know about the resource usage patterns of workloads, the better we can allocate resources. Here we present a methodology to discover resource usage behaviors of containers training Deep Learning (DL) models. From monitoring, we can observe repeating patterns and similitude of resource usage among containers training different DL models. The repeating patterns observed can be leveraged by the scheduler or the resource autoscaler to reduce resource fragmentation and overall resource utilization in a dedicated DL cluster. Specifically, our approach combines Conditional Restricted Boltzmann Machines (CRBMs) and clustering techniques to discover common sequences of behaviors (phases) of containers running the DL training workloads in clusters providing IBM Deep Learning Services. By studying the resource usage pattern at each phase and the typical sequences of phases among different containers, we discover a reduced set of prototypical executions representing the majority of executions. We use statistical information from each phase to refine resource provisioning by dynamically tuning the amount of resource each container requires at each phase. Evaluation of our method shows that by leveraging typical resource usage patterns, we can auto-scale containers to reduce CPU and Memory allocation by 30% compared to statistics based reactive policies, which is close to having a-priori knowledge of resource usage while fulfilling resource demand over 95% of the time.
Serverless computing is a new paradigm that promises to free cloud users from the burden of having to provision and manage resources. However, the degree to which serverless computing will replace provisioned servers remains an open question. To address this, we develop an economic model that aims to quantify the value of serverless to providers and customers. A simple model of incentives for rational providers and customers allows us to see, in broad strokes, when and why serverless technologies are worth pursuing. By characterizing the conditions under which mutually beneficial economic incentives exist, our model suggests that many classes of customers can already benefit from switching to a serverless model and taking advantage of autoscaling at today's price points. Our model also helps characterize technical research directions that would be likely to have impact in the market.
Resource allocation for serverless query processing is a challenge. Unfortunately, prior approaches have treated queries as black boxes, thereby missing significant resource optimization opportunities. In this paper, we propose a plan-aware resource allocation approach where the resources are adaptively allocated based on the runtime characteristics of the query plan. We show the savings opportunity from such an allocation scheme over production SCOPE workloads at Microsoft. We present our current implementation of a greedy version that periodically estimates the peak resource for the remaining of the query as the query execution progresses. Our experimental evaluation shows that such an implementation could already save more than 8% resource usage over one of our production virtual clusters. We conclude by opening the discussion on various strategies for plan-aware resource allocation and their implications on the cloud computing stack.
Machine learning (ML) based prediction models, and especially deep neural networks (DNNs) are increasingly being served in the cloud in order to provide fast and accurate inferences. However, existing service ML serving systems have trouble dealing with fluctuating workloads and either drop requests or significantly expand hardware resources in response to load spikes. In this paper, we introduce Model-Switching, a new approach to dealing with fluctuating workloads when serving DNN models. Motivated by the observation that endusers of ML primarily care about the accuracy of responses that are returned within the deadline (which we refer to as effective accuracy), we propose to switch from complex and highly accurate DNN models to simpler but less accurate models in the presence of load spikes. We show that the flexibility introduced by enabling online model switching provides higher effective accuracy in the presence of fluctuating workloads compared to serving using any single model. We implement Model-Switching within Clipper, a state-of-art DNN model serving system, and demonstrate its advantages over baseline approaches.
We presented a novel platform dedicated to stream processing that improved resource efficiency by sharing resources among applications. The platform utilized latency-aware schedulers to handle stream applications with heterogeneous SLAs and workloads. We implemented the prototype in Spark Structured Streaming and evaluated the platform with pseudo IoT services. The result showed that our platform outperformed default Spark Structured Streaming while reducing the necessary CPU cores by 36%. We further compared the adaptability of the schedulers and found that one of the schedulers reduced the SLA violations by 90% compared to the default FAIR when the platform was overloaded.
Multitenancy is the foundation of modern cloud computing: a single datacenter machine must run code from multiple customers. To safely expose such a machine to untrusted tenants, datacenters have traditionally leveraged virtualization [4, 9]. In this approach, privileged hypervisor software (provided by the datacenter operator) mediates tenant access to raw physical resources like RAM and IO devices. A hypervisor isolates tenants from each other, and isolates the hypervisor from tenants. However, tenants are not isolated from the datacenter operator; the operator’s hypervisor can arbitrarily manipulate tenant state, and the operator herself can physically inspect or modify the contents of server RAM. Intel’s SGX-enabled processors [13] stop these attacks. Using hardware-enforced memory partitioning, SGX prevents a hypervisor from accessing secure tenant pages. SGX hardware also transparently encrypts and HMACs cache lines during eviction to RAM; thus, a datacenter operator with physical control of a machine cannot see cleartext tenant RAM, or undetectably tamper with the encrypted RAM that is visible. SGX is the foundation for a variety of software-level runtimes that isolate datacenter tenants from privileged management software [3, 5, 45, 58]. Unfortunately, SGX-based approaches have three important limitations. • SGX gives a tenant the illusion of ISA-level isolation. However, tenants cohabitate at the microarchitectural level, resulting in side channel vulnerabilities that leak information from ostensibly secure computations (§2.1). • SGX can cryptographically vouch for the initializationtime integrity of a secure computation. However, SGX has no way to attest a computation’s dynamic (i.e., post-load) integrity. Both initial and post-load integrity are important (§2.2). Clients do not wish to exchange data with “secure” server-side code that was initialized correctly, but subsequently corrupted by a network attacker (e.g., via a ROP exploit [7, 43, 49] or type confusion [8, 30, 31, 35]). • SGX is implemented using a combination of hardwired circuits and updatable microcode [13]. Unfortunately, Intel hides many implementation details that are necessary to fully understand SGX’s security properties (§2.3). In response to high-profile security bugs involving SGX [10–12, 39, 46, 52], Intel has released a variety of microcode patches, many of which were incorrect or incomplete and required subsequent revision [21,23,34]. These episodes demonstrate that security researchers need more visibility into the silicon and the microcode that implement hardware-based isolation mechanisms. Motivated by these problems, we propose a new isolation approach for datacenter multitenancy. As with SGX, we leverage trusted hardware to isolate tenants from each other and from the datacenter operator. However, our approach differs from SGX in three crucial ways. • First, our trusted hardware strongly isolates each tenant’s ISA-level state at the microarchitectural level, removing side channels involving other tenants or the hypervisor. • Second, we allow a tenant to explicitly bind application code to monitor code that dynamically enforces runtime security invariants like control flow integrity. The monitor code runs in parallel with application code; however, the monitor runs on a different CPU pipeline (managed by trusted hardware) that receives a read-only stream of the register state from the application-level pipeline. With the exception of this register mirroring, the two pipelines are isolated at the microarchitectural level. This design prevents side channel leakages of monitor state to application code that might be under attack. Microarchitectural partitioning of an application and its monitor also eliminates more direct attacks that could occur if monitor state were located in the same address space as the application to protect. • Third, our new trusted hardware uses an open microcode format, and exposes a software-readable description of microarchitectural-level hardware details. This approach allows tenants to independently verify the security properties of a server’s hardware. This design also allows tenants to customize monitor code to fully exploit the microarchitectural affordances provided by a particular datacenter server. In Section 2, we provide more background on SGX and related technologies. We then sketch a preliminary design for our new isolation hardware (§3). We conclude by describing some open research challenges (§4).
How can Minecraft-like games become scalable cloud services? Hundreds of Minecraft-like games, that is, games acting as modifiable virtual environments (MVEs), are currently played by over 100 million players, but surprisingly they do not scale and are frequently not published as cloud services. We envision a new architecture for large-scale MVEs, supporting much larger numbers of concurrent users by scaling up and out using serverless technology. In our vision, developers focus on the game (business) logic, while cloud providers manage resource management and scheduling (RMS) and guarantee non-functional properties. We provide a definition for MVEs, model their services and deployments, present a vision for large-scale MVEs architected as serverless systems, and suggest concrete steps towards realizing this vision.
Cloud gaming in recent years has gained growing success in delivering games-as-a-service by leveraging cloud resources. Existing cloud gaming frameworks deploy the entire game engine within Virtual Machines (VMs) due to the tight-coupling of game engine subsystems (graphics, physics, AI). The effectiveness of such an approach is heavily dependant on the cloud VM providing consistently high levels of performance, availability, and reliability. However this assumption is difficult to guarantee due to QoS degradation within, and outside of, the cloud - from system failure, network connectivity, to consumer datacaps - all of which may result in game service outage. We present a cloud gaming framework that creates a distributed game engine via loose-coupling the graphical renderer from the game engine, allowing for its execution across cloud VMs and client devices dynamically. Our framework allows games to operate during performance degradation and cloud service failure, enabling game developers to exploit heterogeneous graphical APIs unrestricted from Operating System and hardware constraints. Our initial experiments show that our framework improves game frame rates by up to 33% via frame interlacing between cloud and client systems.
