Serving embedding-based recommendation (EMR) models requires a mix of GPUs, CPUs, and DRAM. Current systems typically provision these resources on monolithic servers with a fixed ratio across resource types, leading to inefficient resource utilization and inflated operational costs. To solve this problem, we propose FlexEMR, a system architecture that fully disaggregates these resources, and interconnects them via an optimized RDMA network. This design enables independent scaling of resources, enhances failure isolation, improves overall resource efficiency, and reduces operational costs. We achieve this by introducing two classes of techniques to address the networking challenges introduced by disaggregation: (1) optimizing embedding lookup communication by leveraging workload locality, and (2) improving network transport through a high-performance, multithreaded RDMA engine. We detail our design considerations and share early performance insights, highlighting the potential of FlexEMR for enabling fully disaggregated EMR model serving.
Large Language Models (LLMs) are increasingly explored as general-purpose assistants for infrastructure operations, helping automate tasks like querying data, analyzing logs, and suggesting fixes. In this paper, we consider the more general and ambitious problem of fully automating root cause analysis (RCA) in microservice systems, where LLMs must collect information, reason about it, and interact with the environment to detect, localize and resolve issues. Anecdotal evidence offers useful insights and partial solutions, but the broader challenge remains unresolved. We systematically evaluate multiple LLM agent architectures across a range of incident scenarios. We study how different tool-augmented agents perform, and shed light on common failure modes, including hallucinated reasoning paths and inefficient use of context. Our findings reveal both the promise and the limitations of current approaches, and point to concrete directions for more robust and effective use of LLMs in this domain.
Memory management (MM) and scheduling are two core components of the operating system (OS) that are traditionally studied and implemented separately, resulting in doubled development and maintenance efforts for kernel developers. In this work, we identify the conceptual similarities between these two subsystems and propose their unification under a generalized resource mapping framework called Crazy. To validate this approach, we developed a prototype, CrazyOS, by merging two semi-microkernels for MM and CPU management. We then implemented several policies atop CrazyOS, including the sophisticated Completely Fair Scheduler (CFS).
Generative inference with long prompts often exceeds GPU memory limits in resource-constrained systems (e.g., workstations and edge devices), thereby causing inference failures. We present HyperGen, a lightweight generative inference framework that optimizes the prefill stage (i.e., when input prompts are processed) via two fine-grained partitioning techniques: (i) partitioning and loading model parameters with size awareness into GPU memory; and (ii) partitioning computations of different inference steps to fit into GPU memory and offloading concatenation of partial results to CPU memory. Evaluation shows that HyperGen supports a maximum prompt length of up to 3.8× longer than an existing GPU-based inference approach and reduces the time-to-first-token from hours to seconds compared to the CPU-based prefill approach.
In this paper, we look at the problem of concurrent progress from an unconventional perspective and find new benefits for practical wait-freedom by considering security and reliability implications of concurrent data structures. Although lock-free data structures are increasingly used in many system applications, they can be prone to denial-of-service (DoS) attacks and pose some dilemmas for a user. For example, when can we safely conclude that one side of a communication channel is buggy or malicious? What should we do about temporary slowdowns due to scheduling peculiarities? We observe that when using wait-free approaches, these questions can be fully resolved due to strict theoretical upper bounds of wait-free algorithms; e.g., all loops are finite and the worst-case number of iterations is known beforehand. This discussion is crucial since other existing mechanisms, such as read-copy-update (RCU), are also quite problematic from the security standpoint. Moreover, even some well-known hardware primitives are less resilient to DoS attacks. We show that with recent advancements in lock- and wait-free algorithm design, many past challenges can now be fully overcome, potentially making such data structures more appealing and easier to use than present RCU equivalents. We present real-life examples that show benefits of wait-free synchronization using Go-like channels. Our overheads with respect to lock-free synchronization remain fairly low while advantages with respect to more traditional fine-grained locking are visible even under fairly moderate contention due to a reduction of synchronization system calls.
