Large language models (LLMs) increasingly rely on context caching to enhance serving efficiency. However, this optimization inadvertently compromises fairness in multi-tenant LLM serving systems. Existing fair schedulers, which account only for compute resources, are unable to handle the multi-dimensional resource demands introduced by context caching. We identify that tenants with high cache hit rates are perceived as having lower compute costs, leading schedulers to unfairly prioritize them over others. Furthermore, the first tenant to compute a shared prefix fully bears the prefill cost for shared prefixes, while subsequent consumers free-ride on the cached data without contributing to such prefill cost. Addressing such unfairness via rigid allocation strategies (e.g., static partitioning) overlooks workload heterogeneity, inevitably wasting host memory on low-reuse workloads. To address the challenges above, we introduce FairCache, the first system that achieves comprehensive resource fairness in multi-tenant LLM serving with context caching. FairCache unifies compute, memory, and I/O costs into a single scheduling metric, and redistributes the prefill cost of shared prefixes across all benefiting tenants to mitigate free-riding. For cache management, FairCache combines the max-min fair allocation with a lightweight reuse-value model that reclaims cold data without harming fairness. Extensive evaluations using a prototype implementation with diverse workloads demonstrate that FairCache effectively eliminates cache-induced unfairness while maintaining high serving efficiency. Theoretical analysis further establishes FairCache's fairness properties and Pareto-efficient cache utilization under bounded request costs, work-conserving operation, and deterministic tie-breaking.
The Maximum Common Subgraph (MCS) problem is a fundamental challenge in graph theory. It generalizes the subgraph isomorphism problem and is known to be NP-complete and difficult to approximate. This inherent complexity underscores the need for efficient solutions. This work introduces Hydra-MCS, a novel hybrid CPU-GPU approach that significantly accelerates MCS computation. Our method uses a lightweight metric to accurately estimate the remaining computation (RC) for a given solution. Then it uses this metric to offload computations to the GPU, maintain load balancing, and minimize unnecessary task sharing between threads. We identify the essential information the CPU and GPU must share to optimize performance by carefully analyzing our hybrid implementation. Furthermore, we discuss the substantial advantages of offloading computation to the GPU. Our method establishes a new state of the art in performance. Under matched CPU resources, GPU integration provides approximately a $3\times$ additional speedup over Hydra-MCS CPU and approximately a $4\times$ speedup over parallel McSplit on the scaling workload; across long-running benchmark instances, speedups over McSplit typically reach 4-$5\times$, with peaks exceeding $12\times$. We also address scalability and robustness, as our algorithm's behavior significantly surpasses that of the original as the number of threads increases and the problems become harder. A comprehensive analysis comparing solved instances over time demonstrates that the hybrid implementation consistently solves 30-40% of the instances not yet solved by the original framework, peaking at 70% for the most complex graph pairs as they approach the timeout threshold.
Pretrained Foundation Models (PFMs) enable high-accuracy inference services but are typically deployed in remote datacenters, resulting in prohibitively high inference delay. Mobile Edge Computing (MEC) can mitigate such high delays by caching PFMs or their fine-tuned variants on cloudlets located close to end users. However, the substantial resource requirements of PFMs and the significant parameter redundancy among their fine-tuned variants make it impractical and uneconomical for resource-constrained cloudlets to cache every large model independently. Motivated by this critical issue, we investigate the collaborative large model caching and inference offloading problem in an MEC network driven by a service market consisting of multiple selfish service providers. The novelty of our study lies in exploring parameter sharing among large models to jointly optimize inference accuracy, delay, and cost while maximizing the total payoff of the service market. To this end, we propose a coalition formation mechanism that enables economic cooperation and fair payoff allocation among different cloudlets. Specifically, we first formulate the optimization problem of collaborative large model caching and inference offloading with parameter sharing as a Non-Linear Integer Program (NLIP), and then reformulates it into an equivalent and computationally tractable Integer Linear Program (ILP). We then develop a resource-aware randomized algorithm with a provable approximation ratio. We further incorporate an $\epsilon$-perturbed best-response process to ensure that the proposed mechanism eventually converges to a stable coalition structure. We also analyze the Strong Price of Anarchy (SPoA) of the proposed mechanism. Experimental results on a real-world dataset demonstrate that the proposed algorithm reduces the average delay and cost by at least 19% and 9%, respectively, while achieving a 10% increase in the total payoff.
In this paper, we present a latency-aware scheduler for large-language-model (LLM) inference across mobile devices, edge servers, and a remote cloud. Our fine-grained delay model captures OFDMA uplink/downlink rates, KV-cache backhaul serialization, and profiled GPU planning-chunk resource constraints, enabling per-request routing among (i) cloud-only execution, (ii) cloud-prefill + edge-decode split execution, and (iii) edge prefill–decode mix execution. The scheduler targets high goodput and SLO attainment under hard deadlines, TTFT/TPOT targets, and guarded SM-resource constraints via a four-stage loop: windowed admission, $\delta$-similar prompt bucketing, tile selection, and SM-feasible prefill/decode allocation with dynamic execution adjustment. We bound padding waste, prefill-latency standard deviation, and the number of buckets, and analyze how the feasible allocation frontier is shaped by active register, SMEM, and warp walls under high- bandwidth, low-load conditions. On an A800/A6000 heterogeneous testbed, repeated real-serving experiments exercise all three routing decisions. In a Qwen3-8B heterogeneous replay, the online policy lowers mean end-to-end latency by 6.4% relative to the best fixed mean baseline and attains 100% SLO, while its P95 confidence interval overlaps those of fixed Cloud and edge-local execution. In a heterogeneous Llama-3 replay, the policy selects Cloud/Mix/Split for 18/21/6 requests; it attains 100% SLO across the four primary workload regimes and 95.6--100% across the offered-load points. Real CUDA-launch analysis further shows that the dominant Prefill/Decode kernel pair fails the joint-residency resource bound for all 37 Llama shapes on both GPUs; guarded alternative pairs account for 41.4% and 26.9% of the analyzed pair weight on A800 and A6000, respectively. These measurements support profile-guided admission and fallback rather than a blanket claim that concurrent Mix is always beneficial.
Heterogeneous graph neural networks (HGNNs) are highly effective in processing heterogeneous graph data and have been widely adopted in critical domains. As real-world graph data continues to scale, performing direct inference on entire graphs becomes increasingly infeasible, making mini-batch methods the standard approach. However, in end-to-end HGNN inference, metapath-based mini-batch sampling constitutes a significant performance bottleneck due to the extensive random memory accesses induced by the irregular traversal of graph structures. Existing sampling paradigms suffer from excessive redundant traversals caused by inherent semantic redundancy, severely degrading sampling efficiency and, consequently, leading to suboptimal mini-batch inference performance. In this work, we propose a redundancy-aware HGNN sampling paradigm that leverages a metapath trie to reuse traversal paths, effectively eliminating redundant memory accesses. We then map it onto a multi-channel hardware sampling unit denominated ESR-HGNN. Furthermore, we introduce a reusability-driven metapath grouping technique that optimally clusters metapaths to maximize reusable traversal paths within hardware channels, enhancing efficiency in scenarios with semantic parallelism. Extensive experimental results demonstrate that ESR-HGNN achieves an average sampling performance improvement of one order of magnitude over CPU and GPU, accompanied by significant energy savings. Additionally, it delivers substantial speedup in end-to-end mini-batch inference when integrated with GPU and state-of-the-art HGNN inference accelerator.
Recent advancement of distributed deep learning have heightened the demands for efficient cluster resource management. Existing frameworks are primarily designed for deterministic jobs, assuming exclusive resource assignments over predictable execution durations. However, they overlook one unique characteristic of DNN model exploration that it is a trial-and-error process. Exploratory training jobs constitute a significant portion of the deep learning workload. Unlike deterministic jobs, they are usually much shorter, in terms of execution time. This is caused by early-terminations during the exploration process, e.g., hyperparameter tuning and code debugging. This makes exploratory jobs more sensitive to queuing latency than deterministic jobs, causing both poor user-experience and low system utilization. This paper revisits the resource management for exploratory training jobs from a novel perspective by enabling exploratory jobs to opportunistically share resources assigned to deterministic jobs. A novel framework, CouchSurfer, is proposed to host exploratory jobs as resource-agnostic, low-priority surfers attach to long-running deterministic jobs. CouchSurfer provides low queuing latency by bypassing traditional scheduling process for exploratory jobs. This paper focuses on solving two key challenges: (1) identifying appropriate placements for exploratory jobs, and (2) mitigating performance interference caused by co-location. We introduce Exploratory Job Placement Problem (EJPP), targeting on low-latency model exploration, and present a threshold-guided algorithm to compute fast and efficient placement plans. Furthermore, CouchSurfer incorporates a Job Fusion Mechanism (JFM) that performs fine-grained kernel-level scheduling to achieve low-interference job co-location. Both testbed experiments and large-scale simulations are conducted to demonstrate CouchSurfer's effectiveness that it reduces 85.2% job queuing latency than other schedulers.
Coupled simulation, also known as co-simulation, has been proposed to support task schedulers by simulating, at runtime, the Quality of Service (QoS) resulting from scheduling actions. However, existing co-simulation methods typically assume a static arrival time series. This assumption limits the diversity of traffic scenarios. To address this, we propose an online adaptive arrival forecasting framework that integrates a change-point detection module and a probabilistic transformer model to couple co-simulators with arrival series forecasting. This framework also updates the prediction model in response to detected changes. Additionally, we introduce the Co-simulated Adaptive Recurrent Surrogate Scheduler (CARSS), which uses simulated QoS metrics from the co-simulator to make scheduling decisions that enhance system performance. Experiments show that our online adaptive forecasting framework has lower forecasting errors than traditional models and reduces co-simulator prediction error by 11% in average response time and 22% in average service-level agreement (SLA) violation on real-world traces. Furthermore, CARSS improves QoS metrics, achieving 2.5% reduction in average energy consumption and reductions of 8.2% and 82.7% in average response time and average SLA violation, respectively, compared to the best baseline scheduler.
Dataflow-based coarse-grained reconfigurable architectures (CGRAs) and dynamic high-level synthesis (DHLS) are both promising for accelerating applications with nontrivial control and memory behavior, but existing compilation flows are typically fragmented and often struggle with control handling, memory ordering, and effective resource reuse. In particular, prior CGRA compilers are still limited in compiling control-rich programs, while existing DHLS flows are not well aligned with dataflow CGRA compilation. This paper presents RedPanda, a unified compilation frame work for both dataflow-based CGRAs and DHLS. The main contribution of RedPanda is a shared front- and middle-end built on a common RedPanda IR, which enables the two targets to reuse the same core compiler infrastructure. On top of this foundation, RedPanda introduces static analyses and transformations for branch handling, memory ordering, aggressive loop pipelining, and time-multiplexing-aware compilation, thereby improving support for control-/memory-irregular workloads while reducing unnecessary serialization and instruction overhead. After this shared compilation stage, the IR is lowered to target-specific back-end flows for CGRA mapping and DHLS circuit generation. Experimental results show that RedPanda reduces CGRA spatial execution cycles by 33.4%, improves temporal-CGRA resource efficiency by 37%, and reduces instruction count by about 13%. For HLS, RedPanda achieves at least 34% faster execution while using fewer LUT, DSP, and FF resources. These results demonstrate that RedPanda provides a practical unified compilation foundation for both CGRA and DHLS targets.
Scientific workflows that require HPC resources are critical in many areas of scientific exploration. Because these workflows tend to be data intensive, severe bottlenecks emerge in storage systems and I/O networks. Although there has been much prior work on coordination of workflows, scheduling algorithms, and HPC storage systems, there are no comprehensive workflow performance diagnosis suites that can automatically identify data flow bottlenecks. We present DataFlowDrs, a new comprehensive suite of tools for performance optimization of HPC workflows that especially focuses on data flow and storage. Our suite introduces (a) lightweight high-resolution measurement and visualization tools for workflow profiling and tracing; (b) rapid modeling and analysis that reduces analysis data by compressing common repeated coordination patterns; (c) novel methods for predicting data flow scaling using automatically generated interpretable models of data flow; (d) effective performance analysis and bottleneck detection that can automatically quantify and rank bottlenecks for different combinations of task parallelism and storage resources; (e) actionable performance optimization in the form of new schedules and resource assignments. We evaluate DataFlowDrs on a representative set of workflows and demonstrate performance improvements over baseline and state-of-the-art methods. DataFlowDrs automates several previously difficult manual analyses and substantially reduces the impact of data flow bottlenecks by recommending the right tradeoffs between task parallelism and storage performance. DataFlowDrs has been successfully reproduced in the Student Cluster Competition (SCC) at the Supercomputing Conference (SC) 2024. We also summarize and discuss the student teams' results and findings. DataFlowDrs is openly available [1].
Modern High Performance Computing (HPC) workloads exchange datasets at high velocity to enable data-driven science. HPC systems employ hardware and software I/O accelerators to enable efficient data exchange. However, domain scientists and software engineers must manually request specific hardware accelerators and manually integrate software accelerators into their workloads, significantly hindering their adoption. Recently, scientists have proposed I/O intents, defined as “why”, “what”, and “how” users or applications perform certain I/O operations, that can be used to automatically configure storage systems for the user. We designed an intent-driven scheduler plugin called IFlux that uses the workload's I/O intents to assist the HPC system scheduler to automatically and dynamically allocate hardware and software accelerators. In this work, IFlux advances the current state-of-the-art in three key aspects. First, IFlux automatically and efficiently maps different I/O intents to various hardware and software accelerators with a throughput of 3.5M intents per second with a scheduler overhead of 0.02%. Second, IFlux provides a scheduler workflow that uses a scheduler's job specification enhanced with intents to allocate the required storage accelerators, deploy necessary middleware software, and enable workloads to use the allocated hardware and deployed software transparently with a runtime overhead of 0.01%. Finally, IFlux speeds up popular benchmarks such as the IOR and DLIO Benchmarks, which represent six classes of workloads from simulation, data analytics, and artificial intelligence, by up to 147× for specific use cases. In conclusion, IFlux enhances existing HPC schedulers with an intent-driven approach to speed up large-scale HPC workloads such as the 1000 Genomes workflow and Megatron Deepspeed by up to 3.12× on an institutional cluster.
Deep neural networks (DNNs) with billions of parameters power many important applications, but their training is fundamentally constrained by the limited on-chip memory of GPUs. This memory wall forces training to rely on distributed execution or memory offloading, both of which introduce substantial inefficiencies. Existing offloading techniques can scale model size but often incur severe throughput degradation, while conventional distributed training suffers from poor hardware utilization due to limited cross-device bandwidth, leaving accelerator resources underused. We present SYNERGYSCALE, a software framework that addresses the GPU memory wall by jointly optimizing memory usage and computation efficiency. SYNERGYSCALE combines dynamic hierarchical memory offloading across GPUs, CPU, and secondary storage tiers to reduce GPU memory pressure with fine-grained multi-stream task partitioning that takes advantage of the concurrent execution capabilities of modern GPUs. The framework automatically selects offloading parameters that balance data movement overhead with parallel execution efficiency. Experimental results show that SYNERGYSCALE can train a 114.41B-parameter model on a single 80 GB NVIDIA A100 GPU and a 547.77B-parameter model on eight A100 GPUs, supporting models up to 29.5× larger than those enabled by existing approaches. By co-optimizing offloading and computation, SYNERGYSCALE achieves up to 4.5× higher training throughput than offloading-only methods, without altering training semantics, making billion-scale model training both efficient and cost-effective.
Serverless computing has emerged as a compelling cloud paradigm due to its simplified development model, automatic scalability, and fine-grained billing. While its stateless execution model enables high elasticity and resource efficiency, it poses noteworthy challenges for building complex stateful applications. To bridge this gap, modern serverless platforms rely on external cloud storage systems to manage persistent state, giving rise to stateful serverless computing. However, this decoupled architecture introduces two fundamental limitations: performance bottlenecks caused by frequent remote state accesses, and state reliability issues stemming from function crash retries. We propose Arcus, a high-performance function state I/O system co-designed for both idempotent execution and low-latency access. Arcus integrates two key techniques: CFLog, a coordination-free logging mechanism that eliminates internal ordering overhead while ensuring idempotent function execution, and FCCache, a cache system optimized for the dispersed invocation pattern of serverless workloads, which reduces read/write latency by improving local hit rates for reads and shortening the write path. Compared to the state-of-the-art system Halfmoon, Arcus reduces read/write latency by 80.4% and 87.9%, respectively. In evaluations using three real-world applications, Arcus achieves an average 60.3% reduction in end-to-end request latency, demonstrating its effectiveness for fast and reliable function state I/O.
Irregular workloads challenge GPUs with small tasks, unpredictable branching, and uneven load distribution. Quantum perturbation theory and Density Functional Theory (DFT) represent a prominent example, where such irregular patterns arise in large-scale simulations of materials and molecules. Despite recent efforts to accelerate these simulations on GPUs, State-of-the-Art (SOTA) methods still suffer from low GPU utilization and severe load imbalance, which hinder efficiency and scalability. In this paper, we redesign the expression of the data structure and propose an efficient implementation for quantum perturbation named HIP-DFPT. A series of general GPU optimizations are proposed, including micro-task composition strategy, GPU memory management, and multi-stream pipelines. We present a branch-dominant hybrid online performance model to predict quickly and accurately with only 5% sampled data. Furthermore, we introduce a multi-level load balancing strategy guided by performance models for both inter-GPU and intra-GPU task distribution. Our experimental results on a large-scale GPU cluster show significant improvements in computational efficiency and scalability. Using up to 8192 GPUs for simulations with up to 200,006 atoms with all-electron precision, we achieve an average load balancing efficiency of 1.06, a 55% increase in GPU occupation, and a 70% enhancement in throughput.
With the rapid growth of artificial intelligence (AI) and high-performance computing (HPC), GPUs and other accelerators have become a shared computing substrate for a wide range of workloads. However, many shared accelerator clusters still rely on coarse device-level allocation, which often leads to low effective utilization, resource fragmentation, and long queueing delays. Although pooling technologies offer a promising direction, existing approaches remain limited in supporting fine-grained, low overhead sharing across heterogeneous accelerators and diverse co-located workloads. This paper presents gPooling, a hardware-agnostic accelerator pooling framework based on driver-level interception. gPooling creates elastic virtual devices on demand and extends fine-grained sharing across heterogeneous accelerators through a unified control path. We evaluate gPooling using benchmarks derived from real cluster traces and through deployment in a production GPU cluster. Results show that gPooling improves accelerator utilization, reduces user waiting time, and increases the overall efficiency of shared accelerator environments.
Dynamic workflow systems enable local applications to scale out by decomposing their computational needs into task graphs on-the-fly and executing them on large clusters. However, this dynamic task generation raises a challenging resource allocation problem: upon scheduling, a task needs to be allocated some amount of resources, yet its resource consumption is only known after its execution. This uncertainty, combined with workflows' large scale and internal and external stochastic behaviors, can result in huge resource waste when resource allocation decisions are manually or statically made in advance. To tackle this problem, we introduce X-Bucket, a family of four general-purpose, prior-free, online, and robust algorithms for adaptive resource al location in workflow systems. Specifically, algorithms in X-Bucket share the approach of optimally separating collected resource reports of completed tasks into different buckets to minimize resource waste, while differ in the way buckets are discovered and used for resource allocation prediction of subsequent tasks. We implement X-Bucket algorithms in WorkQueue and TaskVine, two mature dynamic workflow systems, and show through our evaluation on eight diverse workflows that the X-Bucket family consistently outperform five alternative algorithms in resource allocation efficiency, with the best algorithm incurring at most 3.9 ms per prediction in the steady state.
As deep learning models grow in size and complexity, data movement, rather than raw compute, has become the dominant performance bottleneck. Analog Compute-In-Memory (ACIM) mitigates this by performing multiply-accumulate operations directly in memory, eliminating repeated weight transfers and achieving order-of-magnitude energy efficiency gains. However, their high weight write cost and lack of native support for non-linear operations require tight integration with digital units that support diverse execution modes such as vector-style and data-parallel processing. Despite growing interest, there is still lack of optimized tools to evaluate such heterogeneous ACIMbased systems at full-network scale. To address this, we present HARP, a unified framework for mapping and scheduling transformer workloads on heterogeneous systems combining ACIM units with digital accelerators supporting multiple execution flows. HARP explores latency-, area-, and balanced mapping strategies for ACIM, exploiting crossbars weight reuse to preserve efficiency, and constructs hybrid schedules that exploit both inter- and intra-layer parallelism through fine-grained pipelining, overlapping execution across analog and digital units while respecting each unit's dataflow constraints. By accepting arbitrary architectural parameters, HARP serves as both a fast and accurate performance estimator and a tool for rapid designspace exploration or hardware-aware neural architecture search. Evaluated on transformers, HARP reduces end-to-end latency by up to 14.54× over a non-pipelined baseline and 7.59× over a single-vector pipeline adopted in state-of-the-art ACIM simulators. HARP will be open-sourced upon the paper's acceptance.
