Serving embedding-based recommendation (EMR) models requires a mix of GPUs, CPUs, and DRAM. Current systems typically provision these resources on monolithic servers with a fixed ratio across resource types, leading to inefficient resource utilization and inflated operational costs. To solve this problem, we propose FlexEMR, a system architecture that fully disaggregates these resources, and interconnects them via an optimized RDMA network. This design enables independent scaling of resources, enhances failure isolation, improves overall resource efficiency, and reduces operational costs. We achieve this by introducing two classes of techniques to address the networking challenges introduced by disaggregation: (1) optimizing embedding lookup communication by leveraging workload locality, and (2) improving network transport through a high-performance, multithreaded RDMA engine. We detail our design considerations and share early performance insights, highlighting the potential of FlexEMR for enabling fully disaggregated EMR model serving.
Large Language Models (LLMs) are increasingly explored as general-purpose assistants for infrastructure operations, helping automate tasks like querying data, analyzing logs, and suggesting fixes. In this paper, we consider the more general and ambitious problem of fully automating root cause analysis (RCA) in microservice systems, where LLMs must collect information, reason about it, and interact with the environment to detect, localize and resolve issues. Anecdotal evidence offers useful insights and partial solutions, but the broader challenge remains unresolved. We systematically evaluate multiple LLM agent architectures across a range of incident scenarios. We study how different tool-augmented agents perform, and shed light on common failure modes, including hallucinated reasoning paths and inefficient use of context. Our findings reveal both the promise and the limitations of current approaches, and point to concrete directions for more robust and effective use of LLMs in this domain.
Memory management (MM) and scheduling are two core components of the operating system (OS) that are traditionally studied and implemented separately, resulting in doubled development and maintenance efforts for kernel developers. In this work, we identify the conceptual similarities between these two subsystems and propose their unification under a generalized resource mapping framework called Crazy. To validate this approach, we developed a prototype, CrazyOS, by merging two semi-microkernels for MM and CPU management. We then implemented several policies atop CrazyOS, including the sophisticated Completely Fair Scheduler (CFS).
Generative inference with long prompts often exceeds GPU memory limits in resource-constrained systems (e.g., workstations and edge devices), thereby causing inference failures. We present HyperGen, a lightweight generative inference framework that optimizes the prefill stage (i.e., when input prompts are processed) via two fine-grained partitioning techniques: (i) partitioning and loading model parameters with size awareness into GPU memory; and (ii) partitioning computations of different inference steps to fit into GPU memory and offloading concatenation of partial results to CPU memory. Evaluation shows that HyperGen supports a maximum prompt length of up to 3.8× longer than an existing GPU-based inference approach and reduces the time-to-first-token from hours to seconds compared to the CPU-based prefill approach.
In this paper, we look at the problem of concurrent progress from an unconventional perspective and find new benefits for practical wait-freedom by considering security and reliability implications of concurrent data structures. Although lock-free data structures are increasingly used in many system applications, they can be prone to denial-of-service (DoS) attacks and pose some dilemmas for a user. For example, when can we safely conclude that one side of a communication channel is buggy or malicious? What should we do about temporary slowdowns due to scheduling peculiarities? We observe that when using wait-free approaches, these questions can be fully resolved due to strict theoretical upper bounds of wait-free algorithms; e.g., all loops are finite and the worst-case number of iterations is known beforehand. This discussion is crucial since other existing mechanisms, such as read-copy-update (RCU), are also quite problematic from the security standpoint. Moreover, even some well-known hardware primitives are less resilient to DoS attacks. We show that with recent advancements in lock- and wait-free algorithm design, many past challenges can now be fully overcome, potentially making such data structures more appealing and easier to use than present RCU equivalents. We present real-life examples that show benefits of wait-free synchronization using Go-like channels. Our overheads with respect to lock-free synchronization remain fairly low while advantages with respect to more traditional fine-grained locking are visible even under fairly moderate contention due to a reduction of synchronization system calls.
Container technology improves storage efficiency by sharing base images through overlay file systems (overlayfs). Overlayfs enables temporary modifications while preserving original files by merging multiple directories into a single unified view. However, in container-in-container environments, the current overlayfs approach incurs inefficiency in terms of storage usage, as it does not support nesting due to file deletion operations. To delete a preserved original file, a special file is created indicating that the file is deleted in the overlayfs, but this mechanism lacks expressiveness for nesting. This paper introduces ShadowWhiteout, a new representation for file deletion in overlayfs. We identify key technical challenges and implement a ShadowWhiteout-capable overlayfs. Our system allows nesting overlayfs while maintaining reasonable runtime overhead.
As performance and energy-conservation demands keep growing, the industry moves towards dedicated hardware for demanding workloads such as networking and AI. One instance of this trend is the use of data-processing units (DPUs) in datacenters: NVIDIA BlueField, AWS Nitro, and Azure Boost exemplify a widening trend. So far, development on DPUs involves the use of low-level APIs with little abstraction from the underlying hardware, spreading application logic across many different C callback functions. This approach is cumbersome and error-prone. To address these issues, we introduce SHOC, a simpler-to-use programming model for DPUs built around C++20 coroutines. This paper explains the model, how to implement it, and demonstrates that its high level of abstraction can be achieved with negligible performance overhead.
Cloud computing paradigms such as microservices and functions-as-a-service have made autoscaling an essential component of cloud application management. However, existing autoscalers struggle at capturing application dynamics, and have difficulties with precisely allocating quotas of shared system resources to the applications. We argue that one fundamental issue is the gap between native OS resource interfaces and surrogate user metrics that existing autoscalers use. In this paper, we take CPU autoscaling as an example: the cloud interface treats CPU resources as a percentage of the host CPU (e.g., millicore), while the OS kernel interprets CPU resources as time-shared quota slices allowed to run within a set period. We advocate for OS kernel support for CPU autoscaling to close the semantic gap, as it allows the autoscaler to perform precise, highly responsive resource allocation. We demonstrate the idea by developing Kscaler, a millisecond-scale CPU autoscaler for Linux. With kernel-level observability of fine-grained scheduler behavior, Kscaler outperforms state-of-the-art CPU autoscalers in responsiveness, precision, and efficiency while employing simple statistical methods.
System policies—such as scheduling, caching, and resource allocation—play a critical role in determining performance and efficiency. Traditionally, these policies are implemented using handcrafted heuristics designed to perform reasonably well across general workloads. While tech giants invest substantial resources in customizing operating systems and other systems for their particular workloads, such optimization remains out of reach for most organizations and individuals. In this paper, we argue that Large Language Models (LLMs) enable a promising paradigm: a centralized decision engine that generalizes across diverse system tasks. A unified LLM architecture could let operating systems and other software adapt to workload changes, optimize policies globally, and improve continuously without manual tuning. Realizing this vision poses challenges in system integration, latency, inter-pretability, and safety. We offer a first step toward unipolicy systems, where a single foundation model governs policy decisions across the system stack.
Performance tuning has never been more critical in modern datacenters, where rapid advances in networking and I/O speeds expose potential bottlenecks in CPUs, memory, and operating systems. Yet, tuning remains notoriously complex, brittle, and opaque, especially at the microsecond-scale latencies demanded by modern dataplanes. We argue that there is a pressing need to unleash the power of machine learning in datacenter operating systems. We propose ML-native Dataplane Operating Systems, a radical design for ML-based performance tuning. Our design embraces real-time, microsecond-scale ML-based adaptation as a foundational principle, treating ML not as a byproduct but as the backbone, to continuously optimize dataplane performance under dynamic workloads and operating conditions. Our preliminary experiments show that a traditional static dataplane OS incurs severe queuing delays (over milliseconds) under dynamic and parallel workloads, whereas our adaptive approach sustains microsecond-scale tail latencies by dynamically tuning OS parameters in response to runtime signals.
In SAN systems with a multilevel caching hierarchy, the storage cache lacks visibility into the page cache of the compute node, resulting in redundant caching of the same blocks and inefficient use of limited cache space. Prior efforts have explored exclusive caching, but they often incur network overhead by transferring evicted pages for cache admission and require intrusive modifications to components on compute nodes. In this paper, we present BNEX, a metadata-driven exclusive caching system for NVMe-oF SANs that significantly reduces network overhead by avoiding full-page transfers. BNEX non-intrusively traces page cache evictions using eBPF and transmits only lightweight block-level metadata. Leveraging metadata of evicted pages, the storage cache prefetches nearby blocks in fixed-size block groups and selectively retains them without compromising exclusivity. Our evaluation shows that BNEX improves throughput by up to 1.15× and cache hit ratio by up to 8× compared to inclusive SAN caching systems.
Modern cloud infrastructures are increasingly complex, driven by heterogeneity, disaggregation, and dynamic service composition—exposing critical limits in traditional attestation models. These models struggle to scale when trust must span multiple domains and elastic services. We present scale-out attestation, a paradigm decoupling platform trust verification from app-level attestation. Our design introduces a recursive attestation framework leveraging abstract service identities and trusted deployment workflows: a single infrastructure agent verifies platforms via abstract policies, while services derive instance-agnostic identities enabling secure recursive dependency attestation. We implement the system on FractOS, a distributed OS for disaggregated data centers, and plan to extend Confidential Containers for practical deployment. Evaluation shows strong security with minimal overhead, enabling scalable confidential computing across heterogeneous and dynamic cloud environments.
Memory disaggregation decouples memory from compute nodes, enabling flexible and scalable memory provisioning beyond the physical limits of single servers. With the rapid evolution of hardware technologies, future systems are expected to adopt multi-tier remote memories with diverse performance characteristics. However, most existing studies focus only on two-tier designs, leaving heterogeneous hierarchies underexplored. This paper presents a lightweight, scalable memory management technique for multi-tier disaggregated memory systems. Central to our approach is the comeback interval, defined as the time between a page's eviction to remote memory and its subsequent re-access. By maintaining a histogram of comeback intervals, the system estimates ranking of page hotness without sorting or synchronization, and then places pages across tiers based on access patterns. To enhance adaptability, we introduce an aggressive placement policy and a rebalancing mechanism for runtime promotion and demotion. We implement our design on a real disaggregated system with RDMA-based remote DRAM and NVMe SSD. Our results show that even when fast remote memory is reduced to 25%, our system achieves performance comparable to configurations using only fast memory, demonstrating its efficiency and robustness.
In recent years, vector databases have become a core component in Retrieval-Augmented Generation (RAG) systems for Large Language Models (LLM), enabling fast retrieval of documents similar to a given query. However, storing a large number of high-dimensional vectors requires substantial storage capacity. A common solution is to reduce vector precision through quantization, but this often degrades retrieval recall. To address this trade-off, we propose Chimera-VDB, which uses a mixture of high- and low-precision vectors to reduce data size while maintaining retrieval recall. Our approach leverages the graph structure of Hierarchical Navigable Small World (HNSW) networks, selectively preserving only the most search-critical vectors in high-precision, while quantizing the rest to lower precision. Experimental results show that our method reduces storage usage to 24% compared to storing all vectors in FP32, while maintaining 93% recall, demonstrating its effectiveness in balancing storage capacity and retrieval performance.
Key-value (KV) cache is a crucial optimization for large language model (LLM) serving, particularly in long-context inference scenarios. While existing KV stores suffer from a fundamental mismatch between the CPU-centric KV cache storage and the GPU-accelerated computation: (1) the low-parallelism CPUs cannot satisfy the fine-grained, highly parallel I/O demand, and (2) the synchronization between CPUs and GPUs introduced by layerwise asynchronous KV operations prevents the adoption of low-level optimizations (such as CUDA Graph). This paper presents TARDIS, a GPU-centric KV cache service for efficient long-context LLM inference. Inspired by file-to-memory mapping (mmap) supported by traditional CPU file systems, the key idea of TARDIS is to map KVs directly onto GPU's high-bandwidth memory (HBM) via modern GPU file systems, so that GPU kernels can access KVs without CPU intervention thus achieving high-concurrency, fine-grained KV storage/retrieval. At the core of TARDIS is a GPU-driven KV store (called GStore), which leverages GeminiFS to allow GPUs to directly access KVs on NVMe SSDs. Based on GStore, TARDIS designs an on-GPU scheduler that can adaptively schedule KV requests onto HBM/SSDs, and enables asynchronous, layer-wise token swapping through CUDA Graph, overlapping computation and KV cache access without CPU synchronization overhead. Evaluation reveals that TARDIS significantly outperforms state-of-the-art CPU-centric KV cache designs for LLM inference, boosting serving throughput by up to 20.52% while maintaining performance within 2.04% of an ideal in-memory KV cache.
Zoned storage enables higher storage densities with Shingled Magnetic Recording (SMR) hard disks and reduces costs and improves tail latency with Zoned Namespace (ZNS) SSDs. However, the adoption of zoned storage has been limited by the absence of suitable general-purpose file systems allowing the use of unmodified applications. We introduce Zoned XFS, which adds zoned storage support to XFS by implementing a zoned space allocator, intelligent data placement, and a high-throughput garbage collector. Our performance evaluation shows that Zoned XFS achieves higher performance for large file read and write workloads compared to the BTRFS and F2FS file systems, even in the presence of garbage collection. With RocksDB, Zoned XFS achieves significantly higher throughput with mixed read/write and overwrite workloads compared to F2FS. Furthermore, Zoned XFS compares favorably with the RocksDB special-purpose ZenFS zoned storage backend. Zoned XFS seamlessly enables the benefits of zoned storage for unmodified applications such as key-value stores or distributed storage systems.
In-network computing has gained some traction recently with the advent of programmable switches in the datacenter, however, for widespread adoption as a general computing paradigm, more flexible deployment platforms are required. We argue that FPGAs, suitably virtualized, can serve such a role, covering the continuum from edge to cloud, addressing the key barrier of hardware deployability and flexibility for in-network computing. FPGAs offer orders of magnitude more efficient execution than general purpose CPUs and low overhead packet ingestion—and with the right hardware virtualization techniques, they can also be flexible. We propose to bring these capabilities together in a Stream-Oriented Hardware Offload paradigm supported on lightweight host-less FPGA platforms to enable In-Network Acceleration of present and emerging distributed applications.
Checkpointing is a special task in the modern deep learning training process as it poses a hard tradeoff between training efficiency and reliability. Frequent checkpoints of model states can enhance resilience to random system failures, yet obtaining checkpoints must halt the training process to prevent it from updating the model, which, however, inevitably hurts the overall training efficiency. To mitigate this dilemma, the GPUDirect storage (GDS) technique is gaining attention as it enables direct PCIe peer-to-peer accesses between SSDs and GPUs. Conceptually, this technique has the potential of fully exploiting the high PCIe bandwidth to accelerate the checkpointing process. Nevertheless, despite its promising features, GDS has not seen wide deployment in production. In this paper, we examine the feasibility of GDS for checkpointing in real-world environments. Through careful analysis and experiments, we identify the fundamental constraints that hinder GDS deployment: the bandwidth of in-production storage devices lags behind the bandwidth expected by GDS in checkpointing. We argue that CPU-centric checkpointing remains indispensable in the foreseeable future and propose a single-copy optimization to further reduce memory overhead during checkpointing. Our results show that optimizing the checkpointing performance necessitates advancements in both storage and software.
Bursty, data-intensive serverless workloads, such as server-less big data analytics and DNN inference, amplify the fundamental tension for cloud providers between meeting strict tail-latency service-level objectives (SLOs) and maintaining cost-effectiveness. A key reason is that today's multi-tenant platforms cannot reallocate physical cores between hardware-isolated virtual machines (VMs) at the microsecond speeds these applications require. This capability is critical for instantly responding to demand spikes by shifting resources from low-priority to latency-sensitive tasks. This position paper introduces HyperFlux, a hyper-reactive virtualization stack comprising three co-designed components: a virtual machine monitor (VMM), a lightweight guest OS, and a Linux kernel module as its core arbiter. HyperFlux achieves core reallocation and vertical core scaling (up/down) among hardware-isolated VMs within tens of microseconds. By coupling transparent, fast vertical core scaling with microsecond-level core reallocation, HyperFlux advances hardware-isolated (KVM) serverless provisioning agility and resource elasticity to the microsecond scale. We discuss the system design and aim to stimulate discussion on next-generation, cloud-native computing with extreme agility.
In confidential computing, it is not trivial to provide trusted persistent storage for trusted execution environments (TEEs). Conventional methods utilize the sealing function of TEEs, untrusted storage such as the operating system's file system, and a trusted monotonic counter. These methods have problems such as relying on local monotonic counters with short lifespans and uncertain resistance to shutdown attacks, or requiring complex distributed protocols. To address these problems, this paper proposes implementing persistent storage using a tamper-resistant device. The proposed method provides a persistent key-value store in a tamper-resistant device, thus avoiding complex distributed protocols. The proposed method uses log-structured file system technology on embedded Multi Media Card (eMMC), and achieves extended device lifespan and high resilience against shutdown attacks. As an application of the key-value store, the authors have implemented the basic functions of an electronic voting system that operates within an SGX enclave. This shows that the key-value store possesses sufficient functionality and useful performance.
