Large language model (LLM) inference serving is increasingly constrained by memory rather than compute. As long-context and long-form reasoning workloads become more prevalent, the key-value (KV) cache dominates both memory footprint and memory traffic during LLM token generation, i.e., decode. In particular, HBM capacity has become a scarce and costly resource that heavily limits inference batch size and system throughput. This paper presents OasisKV, a memory-centric LLM inference system design that alleviates HBM capacity pressure by decoupling full KV-cache storage from HBM during LLM decoding. Because decode-time attention is naturally sparse, OasisKV keeps only the KV entries of the most relevant tokens in HBMs for attention computation. We observe that future important tokens can be predicted accurately in advance using lookahead tokens drafted by speculative decoding (SD). OasisKV employs an efficient attention background pipeline to identify important KV blocks. They are then prefetched from higher-capacity memory tiers (e.g., host or remote memory) and staged in HBMs before being used in the next decode step. We implement OasisKV based on vLLM. The lookahead prediction is accurate enough to keep accuracy within 0.7 points of full attention under a 2,048-token KV budget. This lets OasisKV turn sparsity into throughput gain: 1.69× over dense vLLM on the reasoning workload at 0.1 points of accuracy loss, and up to 2.1× on multi-GPU long-context serving. Under prefill–decode disaggregation, OasisKV reaches about 2× dense throughput while admitting each request with 6.5–9.7× less KV and holding 2.2-2.6 less decode-node host memory than full KV transfer.

Caches reduce latency and network traffic, and cache performance largely depends on the eviction policy. Eviction effectiveness is typically measured by byte and object miss ratios. Although learning-based eviction policies can reduce misses, their high computational overhead limits practical deployment. This work presents 3L-Cache+, an extended and refined version of 3L-Cache. 3L-Cache+ is an object-level learning-based eviction policy that achieves low computational overhead while delivering strong overall miss-ratio performance. To reduce overhead, we introduce two key techniques. First, we design an efficient training-data collection scheme that filters redundant requests and dynamically adjusts the training frequency. Second, we propose a lightweight eviction method that combines bidirectional sampling, which prioritizes unpopular objects, with an efficient eviction-candidate selection strategy. In addition, an auto-tuning mechanism improves adaptability across traces. We evaluate 3L-Cache+ in a testbed using 4,855 traces. The results show that 3L-Cache+ reduces average CPU overhead by 74.7% and 95.5% compared with HALP and LRB, respectively, and by 22.7% compared with the prior 3L-Cache baseline. 3L-Cache+ incurs only 4.8× LRU overhead under small cache sizes and 3.1× under large cache sizes, while achieving the best byte- or object-miss-ratio performance among 11 state-of-the-art policies.

Long-context inference increasingly operates over CPU-resident KV caches, either because decoding-time KV states exceed GPU memory capacity or because disaggregated prefill-decode systems place KV data in host memory. Although block-sparse attention reduces attention cost in this setting, sparsity alone is insufficient for end-to-end efficiency. GPU-only designs remain constrained by PCIe bandwidth and metadata memory overhead, while CPU-GPU hybrid designs still suffer from substantial GPU idle time and bottlenecks in CPU-side top-k selection and sparse attention computation. Fluxion is built on three key insights: output-aware KV budgeting, head-specific and granularity-aware sparse configuration, and cross-device coordinated execution for sparse attention over CPU-resident KV caches. Guided by these insights, Fluxion combines a lightweight head-property predictor, a granularity-budget selector, and a priority-based scheduler to jointly optimize budget allocation, sparse configuration, and CPU-GPU execution overlap. This co-design enables hybrid sparse attention to achieve both accuracy and system efficiency in long-context inference. Across 2 models, 3 benchmarks, and 40 tasks, Fluxion preserves quality well – the worst average degradation is only -0.26 relative to FULL, while delivering 1.5×-3.7× speedup over the strongest fixed sparse hybrid baseline, whose KV budget is only 0.05.

Online data preprocessing has become a major bottleneck for DNN training on resource-constrained edge servers. A promising line of work targets data augmentation, which is often the most time-consuming stage in the preprocessing pipeline. Such approaches alleviate this bottleneck by reusing a subset of augmented samples in future iterations without provisioning additional compute resources. However, existing state-of-the-art reuse methods still incur substantial training inefficiencies, even though they reduce preprocessing time. This paper presents ReCache, a group-based dynamic reuse policy that maximizes CPU-GPU pipeline utilization while preserving model accuracy. ReCache makes three key contributions. First, it derives a reuse threshold that eliminates GPU idle time by balancing CPU preprocessing time with GPU training time. Second, it introduces a multi-dimensional sample assessment mechanism to accurately identify reusable samples. Third, it dynamically partitions training data into groups with fine-grained reuse limits and selectively refreshes stale samples via replacement or reaugmentation to maintain accuracy while minimizing training cost across diverse training settings. We implement and evaluate ReCache across nine models, two tasks on four datasets, and two hardware platforms. ReCache outperforms four state-of-theart methods that do not provision additional compute resources: at the same target accuracy, it reduces training cost by 43% on RTX 4090 and by 30% on V100. Our code is available at https://github.com/optiq-lab/ReCache.

Caches can effectively reduce request latency and network traffic, with the eviction policy serving as a core component. The effectiveness of an eviction policy is measured by both the byte miss ratio and the object miss ratio. To reduce these miss ratios, various learning-based policies have been proposed. However, the substantial computation overhead introduced by learning limits their deployment in production systems. This work presents 3L-Cache, an object-level learning policy with Low computation overhead, while achieving the Lowest object miss ratio and the Lowest byte miss ratio among learning-based policies. To reduce overhead, we introduce two key advancements. First, we propose an efficient training data collection scheme that filters out unnecessary historical cache requests and dynamically adjusts the training frequency without compromising accuracy. Second, we design a low-overhead eviction method that integrates a bidirectional sampling policy to prioritize unpopular objects and an efficient eviction strategy to effectively select evicted objects. Furthermore, we incorporate a parameter auto-tuning method to enhance adaptability across traces. We evaluate 3L-Cache in a testbed using 4855 traces. The results show that 3L-Cache reduces the average CPU overhead by 60.9% compared to HALP and by 94.9% compared to LRB. Additionally, 3L-Cache incurs only 6.4x the average overhead of LRU for small cache sizes and 3.4x for large cache sizes, while achieving the best byte miss ratio or object miss ratio among twelve state-of-the-art policies.

In-network aggregation (INA) offloads gradient aggregation onto switches, and thus effectively reduces the aggregation latency and the volume of traffic. However, INA resources are limited due to the high cost of on-chip memory, which imposes distinct challenges to the effective scheduling of these resources in multi-job MLaaS scenarios. In this paper, we explore the scheduling of INA resources in spatial and temporal dimensions, specifically focusing on its impact on the average job completion time (JCT) and the efficiency of INA resources. We propose Mina,an innovative co-design of algorithm and system that intelligently assigns INA resources to each job and effectively schedules these resources among multiple jobs. Our experiments show that Minaattains an INA efficiency score of 0.9998, implying that almost all jobs run nearly as efficiently as they would with exclusive INA acceleration.

In-network aggregation (INA) offloads gradient aggregation onto switches, and thus effectively reduces the aggregation latency and the volume of traffic. However, INA resources are limited due to the high cost of on-chip memory on switches, which imposes distinct challenges to the effective scheduling of these resources in multi-job Machine Learning as a Service (MLaaS) scenarios. In this paper, we explore the scheduling of INA resources in spatial and temporal dimensions, specifically focusing on its impact on the average job completion time (JCT) and the efficiency of INA resources. We propose Mina, an innovative co-design of algorithm and system that intelligently assigns INA resources to each job and effectively schedules these resources among multiple jobs. Our experiments show that Mina attains an INA efficiency score of 0.9099 on average, 2.67x higher than the baseline, implying that almost all jobs run nearly as efficiently as they would with exclusive INA acceleration. Furthermore, Mina proves to be highly adaptable to varied cluster configurations and incurs only minimal additional overhead.

ACL configuration is essential for managing network flow reachability, yet its complexity grows significantly with topologies and pre-existing rules. To carry out ACL configuration, the operator needs to (1) understand the new configuration policies or intents and translate them into concrete ACL rules, (2) check and resolve any conflicts between the new and existing rules, and (3) deploy them across the network. Existing systems rely heavily on manual efforts for these tasks, especially for the first two, which are tedious, error-prone, and impractical to scale. We propose Xumi to tackle this problem. Leveraging LLMs with domain knowledge of the target network, Xumi automatically and accurately translates the natural language intents into complete ACL rules to reduce operators' manual efforts. Xumi then detects all potential conflicts between new and existing rules and generates resolved intents for deployment with operators' guidance, and finally identifies the best deployment plan that minimizes the rule additions while satisfying all intents. Evaluation shows that Xumi accelerates the entire configuration pipeline by over 10x compared to current practices, addresses O(100) conflicting ACLs and reduces rule additions by 40

Datacenter network topology contains multiple paths between server machines, with each path assigned a weight. Software switches perform traffic splitting, an essential networking operation in datacenters. Previous studies leveraged software switches to distribute network connections across paths, under the assumption that the software switches accurately divide connections according to path weights. However, our experiments reveal that current traffic splitting techniques exhibit significant inaccuracy and resource inefficiency. Consequently, real-world datacenter services (e.g., data mining and deep learning) experience communication completion times that are ∼2.7× longer than the ideal. To address these problems, we propose VALO, a new traffic splitting technique for software switches, to accomplish two goals: high accuracy and resource-efficiency. For the goals, we introduce new concepts: score graph and VALO gravity. We implement VALO using the de-facto software switch, Open vSwitch, and evaluate it thoroughly. On average, VALO achieves 13.1× better accuracy and 25.4× better resource efficiency compared to existing techniques, with maximum improvements reaching up to 34.8× and 67.7×, respectively. As a result, VALO demonstrates 1.3×-2.5× faster average communication completion times for real-world datacenter services compared to existing techniques.

Efficient scheduling of service applications is critical for improving cluster resource utilization while minimizing makespan and application completion time. However, existing schedulers often struggle with coordinating task placement on worker machines due to the lack of runtime estimations. This limitation leads to two major performance issues: the non-synchronization problem and the contention-oblivious problem, both of which result in suboptimal application completion times. To address these challenges, Morbius is proposed, a scheduler that explicitly leverages the spatial structure of service applications to enhance scheduling decisions. Morbius adopts an all-or-nothing scheduling policy, ensuring that all tasks of an application are scheduled to run simultaneously, thereby effectively mitigating the non-synchronization problem. Within each priority queue, Morbius follows a shortest total time first policy, which facilitates contention-aware scheduling. Moreover, Morbius incorporates work conservation and starvation avoidance policies to better handle execution uncertainties and further improve application completion times. A prototype of Morbius is implemented on Yarn and evaluated in two environments: a homogeneous cluster with 36 machines and a heterogeneous cluster with 122 machines. Experimental results show that Morbius significantly outperforms existing approaches, improving average application completion time by up to 10.41x and reducing makespan by over 32.80%.

In cloud datacenter operations, telemetry and logs are indispensable, enabling essential services such as network diagnostics, auditing, and knowledge discovery. The escalating scale of data centers, coupled with increased bandwidth and finer-grained telemetry, results in an overwhelming volume of data. This proliferation poses significant storage challenges for telemetry systems. In this article, we introduce HyperDrive, an innovative system designed to efficiently store large volumes of telemetry and logs in data centers using programmable switches. This in-network approach effectively mitigates bandwidth bottlenecks commonly associated with traditional endpoint-based methods. To our knowledge, we are the first to use a programmable switch to directly control storage, bypassing the CPU to achieve the best performance. With merely 21% of a switch's resources, our HyperDrive implementation showcases remarkable scalability and efficiency. Through rigorous evaluation, it has demonstrated linear scaling capabilities, efficiently managing 12 SSDs on a single server with minimal host overhead. In an eight-server testbed, HyperDrive achieved an impressive throughput of approximately 730 Gbps, underscoring its potential to transform data center telemetry and logging practices.

Network function (NF) offloading on SmartNICs has been widely used in modern data centers, offering benefits in host resource saving and programmability. Co-running NFs on the same SmartNICs can cause performance interference due to contention of onboard resources. To meet performance SLAs while ensuring efficient resource management, operators need mechanisms to predict NF performance under such contention. However, existing solutions lack SmartNIC-specific knowledge and exhibit limited traffic awareness, leading to poor accuracy for on-NIC NFs. This paper proposes Yala, a novel performance predictive system for on-NIC NFs. Yala builds upon the key observation that co-located NFs contend for multiple resources, including onboard accelerators and the memory subsystem. It also facilitates traffic awareness according to the behaviors of individual resources to maintain accuracy as the external traffic attributes vary. Evaluation using BlueField-2 SmartNICs shows that Yala improves the prediction accuracy by 78.8% and reduces SLA violations by 92.2% compared to state-of-the-art approaches, and enables new practical usecases.

Containers are widely embraced for their deployment and performance benefits over virtual machines. Yet, for many data-intensive applications in containerized clouds, bulky data transfers may impose performance issues. In particular, communication across co-located containers on the same host incurs large overheads in memory copy and the kernel's TCP stack. Existing solutions such as shared-memory networking and RDMA have their own limitations, including insufficient memory isolation and limited scalability. This paper presents PipeDevice, a new system for low overhead intra-host container communication. PipeDevicefollows a hardware-software co-design approach - it offloads data forwarding entirely onto hardware, which accesses application data in hugepages on the host, thereby eliminating CPU overhead from memory copy and TCP processing. PipeDevicepreserves memory isolation and scales well to connections, making it deployable in public clouds. Isolation is achieved by allocating dedicated memory to each connection from hugepages. To achieve high scalability, PipeDevicestores the connection states entirely in host DRAM and manages them in software. Evaluation with a prototype implementation on commodity FPGA shows that for delivering 80 Gbps across containers PipeDevicesaves 63.2% CPU compared to kernel TCP stack, and 40.5% over FreeFlow. PipeDeviceprovides salient benefits to applications. For example, we port baidu-allreduce to PipeDeviceand obtain similar to 2.2x gains in allreduce throughput.

Runtime traffic analysis on programmable data-plane requires substantial human effort, and the high speed and complexity of dataplane often make human capacity the efficiency bottleneck. While existing work has proposed LLM-based approaches, they typically rely on offline network logs, failing to address the human capacity limitations in real-time environments. This paper explores the potential of leveraging evolving LLMs to mitigate these human-centric challenges in real physical dataplane. It outlines a novel framework called NetSophon, which features an LLM-based brain for efficient decision-making and an effective arm to manipulate and perceive the physical programmable dataplane. Through interactions among the brain, arm, and dataplane, NetSophon acts as a "super-copilot" for human operators, facilitating real-time dataplane traffic analysis at scale. A case study demonstrates NetSophon’s potential to assist human operators in interacting with dataplane.

This article explores the computing and communication overhead of network processing in Internet of Things (IoT) devices, focusing on containers, a major building block for the edge computing. Our experiments reveal that containers on IoT devices suffer similar to 2.6 x higher CPU usage for SoftIRQ processing, similar to 59% less network throughput, and 2x higher per-packet latency on average than native processes. While several existing studies enhance networking performance, they often sacrifice interoperability by requiring special hardware or modifying networking semantics or APIs. Thus, we design and implement a kernel networking accelerator, called SCON, that maintains interoperability, crucial for IoT devices. SCON addresses major bottlenecks in container networking through system-level profiling. We evaluate SCON with three types of IoT devices. On the Raspberry Pi 4, SCON reduces the latencies of major IoT application protocols (e.g., HTTP and MQTT) by similar to 10 x , achieving a similar level of latency to the native process. Further analysis shows that SCON reduces CPU usage for SoftIRQ processing by similar to 26%. We also report similar improvements on the other two IoT devices. Our conclusion is that SCON is unique in significantly reducing the computing and communication overhead of container networking in IoT devices while maintaining interoperability. Furthermore, it works consistently across different types of devices, whether wired or wireless, and regardless of heavy or sporadic traffic.