Consensus, widely regarded as the most fundamental primitive in distributed systems, lies at the core of countless services that require coordination among remote processes. Datacenter services typically achieve consensus through long-established, quorum-based algorithms such as Paxos and Raft, including recent re-adaptations for kernel bypass datapaths (e.g. smartNIC/RDMA-based consensus). While these optimizations can reduce latency to the mu s-scale, they remain constrained by inherent message complexity, namely the need for acknowledgments from majority quorums to tolerate faults and arbitrary message delays. Our approach takes a step further from bare acceleration of classical primitives, focusing instead on leveraging FPGA-smartNIC and priority-queue reservation to achieve synchronous remote interactions in practice. We use synchrony to devise a novel, efficient quorum-less consensus protocol which we use to build Nano-consensus: a novel hardware consensus engine. Nano-consensus operates at network line rate and can reach consensus in 1.03 mu s for single-packet instances, delivering 3.82x latency and 4.8x improvements over the state of the art. We demonstrate how Nano-consensus can be integrated into distributed applications to boost both performance and consistency.
S3-compatible object storage has become ubiquitous, used by an ever-expanding range of applications. Workload traces show that many of these applications treat object storage like a traditional file system, with many small reads and writes, yet object storage implementations have not kept up. Optimized for bulk storage, these systems cannot efficiently exploit modern SSDs, requiring large hardware installations to achieve operation rates typical of local file systems on modest machines. ZStore is a highly-efficient object store designed for modern hardware, providing strong consistency (per-key linearizability) via a novel architecture which replicates data over independent per-device shared logs, using NVMe-over-Fabrics as its backend storage protocol. Based on a 3-node symmetric active-active cell, ZStore performs small reads and writes with minimal I/O amplification (beyond replication factor) while supporting object sizes up to the S3 maximum of 5 TB and optional erasure coding for objects larger than 128 KB. ZStore guarantees single-key linearizability using a two-phase coordination mechanism, tracking in-flight writes so that reads of stable data can be handled on a single gateway, with a heavier-weight multi-node read protocol used only when interfering writes are detected. Our evaluation shows ZStore achieving nearly an order of magnitude improvement in IOPS over widely-used systems (MinIO and Ceph) when evaluated on comparable hardware.
Machine Learning (ML) emerged as a powerful tool for predicting task execution times across the variety of VM types offered by Infrastructure-as-a-Service (IaaS) clouds. However, training ML models to ensure accurate predictions can often become uneconomical for users due to the high costs-in terms of both time and money-for collecting samples, especially when an IaaS cloud offers a wide choice of VM types. This paper investigates a ML model boot-strapping technique that leverages analytical modeling to reduce the cost of collecting training samples while maintaining robust performance predictions. Complementarily, the technique can be used to improve the accuracy of ML models in the case of limited availability of training samples. Experimental results highlighted the potential of the proposed technique with various workloads and with a large set of VM types, paving the way for more cost-effective ML-based performance prediction in IaaS clouds.
Synchronous consensus protocols are often considered impractical due to their reliance on strict timing assumptions. In this paper, we revisit this perception with a focus on public cloud environments, where infrastructure has evolved to offer increasingly predictable network behavior. We report on an extensive empirical study across major cloud providers and regions to evaluate whether the timing guarantees required by synchronous protocols hold in practice. Our measurements show that bounded communication delays are not only common but also stable across a variety of configurations when messages are small (i.e., <= 4 KB). Motivated by these observations, we explore the design space of robust synchronous consensus protocols suited for deployment in public clouds. We introduce SyncPaxos, a synchronous version of the celebrated Paxos protocol, designed for partially synchronous environments, and several variants that benefit from the characteristics of public cloud environments. We analyze their performance and resilience to timing violations and identify the conditions under which they remain safe and performant. Our findings suggest that synchrony is not a theoretical relic but a viable and efficient foundation for building resilient distributed systems in today's cloud infrastructure.
Small deep learning models deployed at edge servers suffer from decreasing accuracy due to data drift and hence require continual learning, which leads to resource competition with inference execution, decreasing accuracy and latency service-level-objective (SLO) fulfillment. Previous methods fail to maximize both accuracy and SLO fulfillment. To address this problem, we propose a Checkpointed Inference-based system for accurate and SLO-guaranteed data drift-resilient model Serving (CIS). CIS incorporates checkpointed inference - maximizing accuracy by continuously switching to intermediately retrained models during inference. However, model switching introduces significant inference latency overhead. To mitigate this, first, CIS proposes a lightweight allocation and placement scheduler to minimize switching time. Second, during inference execution, CIS reorders retraining samples to reduce switching frequency. Third, it temporarily reallocates GPU space from retraining tasks to inference tasks to address request queuing issue caused by the switching, with minimal impact on accuracy. Trace-driven experiments show that CIS achieves up to 25.1% higher accuracy without affecting SLO fulfillment, and requires 4x lower GPU cost compared to the existing method in achieving similar or higher accuracy.
Datacenter growth is constrained by power grids because the constant power required by datacenters (DCs) is difficult to balance with variable solar and wind generation. DC load flexibility is the key to solving these grid problems, but it conflicts with the stable capacity needed for compute efficiency. We propose "power Middlebox", a new system architecture, to bridge the gap. Middlebox decouples datacenter capacity and grid load, reconciling the conflict between DC capacity needs and grid load flexibility requirements. It provides each with the freedom to meet their divergent objectives. We define the Middlebox system architecture, frame its objectives, and explore designs (type and quantity of energy resources, extent of decoupling, management) in varied power grid settings. Evaluation shows that Middlebox unlocks 460% or 170% datacenter growth with grid reliability or decarbonization constraints in a wind-dominated grid. We study decoupling in growing DC scenarios. Decoupling reconciles the conflict between grid and datacenter needs, enabling constant DC power capacity on 99.9% of days, for a cost equal to 37% of a DC's annual power bill (or 3-5% of DC TCO). Future technologies could reduce Middlebox cost by up to 70%. Further, workload flexibility can be exploited to reduce cost. These results are robust across growth scenarios and grid types. Overall, the results show that Middlebox can be deployed in small to large datacenters economically with today's technology.
Modern cloud applications are refactored into microservices, which are deployed as containers across multiple servers. An end-user request often triggers several remote procedure calls (RPCs) between these microservices. RPC latency anomalies caused by packetprocessing delays (bottlenecks) in the host network stack are common. Bottlenecks at a few network components can compound across services, causing SLA violations for many requests. Diagnosing RPC latency anomalies is challenging because many host-level components can contribute to the delay. Identifying the bottleneck component is a crucial first step. However, it often takes significant manual effort and expertise to find the bottleneck component due to the lack of visibility on per-component processing time. In this paper, we present PerfMon, a lightweight system designed to monitor the performance of components in the host stack that automatically identifies the bottleneck component. We develop PerfMon using eBPF technology and evaluate it on a Kubernetes-managed cluster of bare metal servers. Our evaluation demonstrates that PerfMon introduces minimal monitoring overheads while accurately identifying the bottleneck components.
Serverless workflows introduce unique challenges for modern cluster schedulers, as they consist of highly concurrent and ephemeral functions with unpredictable execution patterns. Through analysis of workloads derived from production serverless trace characteristics, we observe that existing OS-level schedulers, such as Linux CFS, lack workflow-level awareness and make scheduling decisions solely at the function level, which can result in bottlenecks within workflows and prolongedWorkflow Completion Times (WCTs). We presentMetis, a non-clairvoyant, workflow-aware OS scheduler designed specifically for serverless workflow workloads, which aims to reduce average WCTs by treating workflows as first-class scheduling entities. Metis implements Workflow-Aware Least-Attained Service (WLAS), a non-clairvoyant scheduling algorithm that leverages workflow-level virtual clocks and critical path estimation to reduce WCTs and ensure fairness. By utilizing eBPF to hook into existing OS primitives, Metis achieves practical deployment with minimal kernel modifications. Extensive synthetic trace-driven simulations demonstrate that Metis reduces average WCTs by 31.3% and the 95th percentile by 14.3% compared to state-of-the-art function-centric scheduling approaches. Real-system experiments across diverse workflow patterns show improvements ranging from 47.2% to 58.2% in average end-to-end latency and the 99th percentile by up to 73.9% over baseline schedulers.
Rising computational demands have made cloud datacenters' water footprint a critical concern. We demonstrate how different water footprint accounting methodologies - incorporating operational, manufacturing, and decommissioning water consumption, impact measurements and highlight the need for methodology standardization for water-aware operations. Our analysis reveals opportunities for water-aware scheduling in datacenters by considering regional water variations and lifecycle impacts.
GPU multiplexing is a widely-adopted strategy in GPU clusters for improving overall throughput and lowering the total cost of ownership. To mitigate inter-task interference in compute power and memory bandwidth on multiplexed GPUs, existing techniques divide a GPU into instances with limited predefined rigid configurations. Low utilization arises from the mismatch between heterogeneous burstiness and immutable resource configurations: 1) bursty inference traffic forces the scheduler to launch underfilled batches that cannot saturate the instance; 2) bursty kernel resource utilization leads to bubbles in compute power and memory bandwidth. This paper presents ZipBatch, a multi-tenant model serving system with elastic fine-grained GPU resource provisioning. First, ZipBatch launches dynamic batches with right-sized resource reservations to reduce macroscopic model-level over-provisioning while decreasing queuing time. Second, by observing that kernels exhibit imbalanced and inconsistent utilization of resources, we develop a prediction-based dual-dimensional regulator that allocates right-sized compute power and memory bandwidth in real time to suppress microscopic resource underutilization. Compared with state-of-the-art solutions, our experimental results show that ZipBatch achieves up to 60.84% reduction in average latency while increasing total throughput by 71.27%.
Federated Learning (FL) is a rising distributed learning paradigm that facilitates multiple devices to jointly train a shared model. Given the presence of heterogeneous devices with distinct data distributions, it is critical to select an optimal subset of devices for engagement in the collaborative training process. However, the dynamic nature of FL, encompassing aspects like dynamic device availability and inherent training dynamics, significantly complicates participant selection, and current systems routinely fall short in adapting effectively to such dynamic environments. This paper introduces FedDance, an efficient and intelligent participant selection framework tailored to tackle the dynamic nature in FL. FedDance employs a lightweight stochastic prediction method to anticipate the dynamic availability of each device. To address the diminishing marginal returns from frequently selected devices due to training dynamics, especially in later training stages, FedDance explicitly quantifies the marginal return of each local training process over time. Additionally, FedDance incorporates a lightweight model to characterize data heterogeneity, enhancing the advantages of managing device and training dynamics. Extensive experiments conducted on four A100 GPUs demonstrate that FedDance can substantially outperform the state-of-the-art systems in final model accuracy and convergence speed, while also providing a considerable advantage in computational overhead.
The increasing demand for ssds coupled with scaling difficulties has left manufacturers scrambling for newer ssd interfaces which promise better performance and durability. While these interfaces reduce the rigidity of traditional abstractions, they require application or system-level changes that can impact the stability, security, and portability of systems. To make mattersworse, such changes are rendered futile with the introduction of next-generation interfaces. It is therefore no surprise that such interfaces have seen limited adoption, leaving behind a graveyard of experimental interfaces ranging from open-channel ssds to stream ssds. Our solution, Valet, leverages userspace shim layers to add placement hints for application data, delivering up to 2-4x write throughput over filesystems and comparable or better performance than application-specific solutions, with up to 6x lower tail latency. Valet generates dynamic placement hints, remapping application data to modern ssds with zero modifications to the application, the filesystem, or the kernel. We demonstrate performance, efficiency, and multi-tenancy benefits of Valet across a set of widely-used applications: RocksDB, MongoDB, and CacheLib, presenting a solution that combines the performance of application-specific solutions with wide applicability to log-structured data-intensive applications.
Multi-agent reinforcement learning (MARL) has emerged as a promising approach for tasks requiring multiple agents for cooperation or competition, such as scientific simulation, multi-robot collaboration, and traffic control. Serverless computing, with its dynamic and flexible resource allocation, has demonstrated potential for improving training efficiency and cost-efficiency in RL workloads. However, existing serverless RL training systems focus primarily on single-agent scenarios, overlooking the unique characteristics and inherent complexities of MARL-such as dynamic interagent relationships and heterogeneous policy requirements across agents-leaving inefficient and even infeasible support to diverse and complex MARL algorithms. This paper introduces MARLess, the first serverless MARL framework designed to support general MARL algorithms. MARLess decomposes MARL algorithms into serverless functions. It further integrates a dynamic learner sharing mechanism that exploits agent similarities to reduce model update costs and employs actor scaling tailored to MARL tasks, minimizing unnecessary sampling costs based on the data requirements of agents' models. This design optimizes both training efficiency and costs without harming the training quality. Experiments on AWS EC2 testbeds show that MARLess outperforms SOTA MARL baselines with up to 1.27x faster training and 68% cost reduction. Large-scale evaluations on a 15-node cluster with a total of 1,920 vCPUs demonstrate MARLess's scalability and consistent performance under increasing workloads. For a real-world scientific application-turbulent flow simulation, MARLess achieves a 34% cost reduction and 1.1x speedup.
Recent advances in the Linux kernel, such as eXpress Data Path (XDP) and AF_XDP sockets, enable high-speed packet processing for software NFs while preserving access to kernel features. However, the default AF_XDP implementation in the Linux kernel does not permit easy and performant NF chaining, e.g., zero-copy transfer of packets across NFs co-located on the same host. While prior work has proposed solutions for optimized NF chaining in the context of kernel bypass frameworks like DPDK that operate entirely in userspace, such solutions do not extend easily to AF_XDP, because the AF_XDP datapath is fragmented across the kernel driver and userspace. This paper introduces FLASH, a low-overhead inkernel chaining mechanism for AF_XDP sockets. FLASH enables zero-copy packet transfers for FLASH-native NFs, and single-copy packet transfers for legacy AF_XDP NFs. Further, via integration with K8s, FLASH supports the deployment of unprivileged containerized NFs on cloud platforms. Our work contributes several novel modifications to the AF_XDP datapath in the kernel to implement optimized NF chaining, and provides userspace libraries/APIs to easily build NFs that leverage FLASH. Our evaluations show that FLASH matches the performance of userspace DPDK-based NF chaining frameworks, while outperforming the best available AF_XDP-based alternatives by up to 2.5x in throughput, and achieving the lowest latency among all NF chaining frameworks.
Modern GPU clusters in computing centers face significant challenges in resource utilization due to fragmentation caused by GPU-sharing mechanisms, job diversity and asynchronous job lifecycles. Existing methods fail to address GPU fragmentation in dynamic scheduling scenarios under GPU sharing. To tackle this issue, this paper proposes DRR, a defragmentation scheduler with deep reinforcement learning (DRL) and rescheduling to mitigate GPU fragmentation. DRR employs a DRL agent trained via imitation learning from heuristic algorithms to overcome cold-start issues, and further enhanced by multi-scale policy optimization for balanced exploration and exploitation to reduce GPU fragmentation. Additionally, the rescheduling strategy in DRR further optimizes GPU utilization by relocating the running jobs. Evaluations conducted on the physical Kubernetes-based testbed and large-scale simulated clusters demonstrate that DRR reduces the average GPU fragmentation rate by 50% compared to state-of-the-art methods, while maintaining Quality of Service (QoS) and ensuring fairness for users.
Modern cloud platforms, such as Kubernetes, provide a service-oriented resource abstraction for explicit Quality of Service (QoS) provisioning, which guarantees resource reservations and supports resource elasticity. In regards to CPU resources, tenants can explicitly specify a guaranteed base demand and an upper bound for resources. Thus, it is desirable that GPU resource provisioning should be analogous to mature cloud-native CPU resource abstraction and can support familiar QoS classes, such as Guaranteed, Burstable, and BestEffort. Although existing research efforts have primarily focused on maximizing GPU utilization by exploiting profiling, capabilities for guaranteed reservation, precise throttling, and elastic bursting are essential to support cloud-native GPU provisioning. It is challenging to provide such features due to the GPU's asynchronous and non-preemptive characteristics. To address this issue, we introduce a spatio-temporal provisioning framework that ensures both resource guarantee and elasticity for GPUs. We define a resource model to control both spatial and temporal dimensions of GPU resources and support familiar QoS classes. Our framework features a per-container Agent for transparent resource accounting and local quota enforcement, and the central Multi-Tenant Arbitrator for global, partition-aware fair scheduling. Performance evaluation demonstrates that our framework can provide accurate GPU resource guarantee and manages dynamic mixed-QoS workloads by honoring reserved GPU resources during contention while allowing tenants to elastically burst into idle capacity up to their limit.
Cloud environments continue to experience substantial memory wastage due to inefficient resource sharing and workload variability. Emerging Compute Express Link (CXL) technology offers fabricattached memory that can expand memory capacity despite added access latency, but existing VM allocation strategies in the Cloud have critical limitations. The static partitioning of local DRAM and CXL memory underutilize capacity, and cannot adapt to dynamic demands. Conversely, Host-managed tiering (software-managed placement) relies on page-table scans or sampling, which incur high CPU overhead. To overcome these challenges, we introduce TierVM, a tiered memory-management design targeting multi-tiered memory hierarchies. TierVM decouples coarse-grained memory-capacity management, handled by the host, from fine-grained, workload-aware memory optimizations, managed by the guest. This design simplifies memory-management complexity, effectively mitigates intraand inter-VM memory-tier contention, and enhances overall resource utilization. TierVM introduces a lightweight "soft hot-plug" mechanism - dyno-NUMA - that updates NUMA memory map in the Guest OS. From our evaluation, dyno-NUMA reconfigures memory up to 90x faster than Linux hot-plug and avoids the failure cases that plague traditional hot-plug. Prototype results show that TierVM can maintain application performance under dynamic demands, and a trace-based data center simulation shows that TierVM can significantly improve memory utilization in the Cloud.
While security concerns of data at rest and in transit have been addressed over the years using standard cryptographic measures, those surrounding data in use have garnered significant attention in recent times. In response, various trusted execution environments (TEEs) have been proposed and are on offer from leading public cloud providers. With development and re-programming efforts, availability, threat models, pricing, performance, etc., differing between various TEEs themselves and also with viable alternatives such as software solutions like partially homomorphic encryption (PHE) to protect data in use, it is imperative to have a system that is independent of these several varying dimensions while also efficiently achieving end-to-end confidentiality guarantees on data processing. We propose Scylla, a mechanism-agnostic confidential analytics framework, built on top of the popular Spark data analytics engine. Scylla utilizes a customizable combination of TEEs and PHE schemes to achieve end-to-end confidentiality guarantees with prime performance. Our evaluation shows that Scylla's query execution times are 1.91x faster than state-of-the-art system Opaque providing similar guarantees. Scylla's novel general architecture enables integrating latest TEEs such as AWS Nitro, AMD SEV-SNP, and Intel TDX with zero rebuilding efforts.
The landscape of machine learning applications has shifted from monolithic end-to-end models to compositions of pretrained large foundation models. For instance, multi-modal chatbots are often built by composition of a large language model and modalityspecific encoder models. Such applications often feature dynamic workflows, with models conditionally evoked according to different inputs and intermediate processing results. Conditional model execution prevents conventional request batching and hinders efficient hardware utilization, due to dynamic, diverging execution paths across requests. Separately deploying models as dedicated services and invoking them on the go during dynamic workflow executions can potentially allow service-wise request batching, boosting resource efficiency. However, generic workflow orchestrators are proven inefficient for machine learning applications, due to schedulers that do not exploit batching, communication methods that are suboptimal for GPU tensors, and the considerable cold-start delays associated with model deployment. We introduce DyOrc, an efficient serving system for dynamic machine learning workflows with conditionally activated model components. DyOrc deploys each model as a dedicated service and scales each service according to demand. Requests from different workflows that share the same component are batched on the go to improve hardware utilization and reduce serving latency. DyOrc further advocates three core designs: (i) speculative scheduling, that utilizes runtime statistics to predict component invocations for more informed inference scheduling and request batching among models; (ii) multi-tier message passing, which runs multiple services in one process and exploits specialized tensor communication libraries to substantially reduce communication overhead of heterogeneous messages across components; and (iii) proactive component loading, that cooperates with the speculative scheduler to hide cold-start overhead of components with billions of parameters. Extensive prototype experiments on representative inference workloads (including multi-modal chatbots, ensemble models, image processing pipelines, and multi-agent debate systems) demonstrate that DyOrc achieves 4-198% serving latency improvement over available baselines such as Ray Serve and Triton Inference Server.
Recent advances in large language models (LLMs) have intensified the need for serving LLMs that are cost-efficient and QoS-guaranteed. Existing frameworks often co-locate computationally distinct prefill and decode instances on homogeneous GPUs, over-looking their unique resource demands and under-utilizing heterogeneous GPUs. This leads to suboptimal resource utilization and increased capital expenditure. We present Cauchy, a LLM serving framework that adaptively deploys prefill and decode computation to the most suitable heterogeneous GPUs and dynamically schedules user requests. At the core of Cauchy is choosing proper GPU Combo, a conceptual GPU combination encompassing diverse GPU configurations, for their cost efficiency in running prefill-decode pairs. Cauchy deploys a set of combos to satisfy QoS requirements (e.g., goodput) of LLM inference. Cauchy further employs hierarchical scheduling to handle user requests, using opportunistic scheduling within the allocated GPU Combos and a goodput-weighted round-robin policy across GPU Combos. Dynamic autoscaling is used to stabilize the cost-efficiency in the face of surging requests. Experiments show that Cauchy achieves up to a 38.3% improvement in Tokens/USD efficiency over the state-of-the-art baselines, while maintaining strict Service Level Objectives (SLOs). Our work highlights the importance of leveraging workload and GPU heterogeneity to achieve superior cost-efficient LLM serving.
