Heterogeneous clusters with diverse devices mitigate computational and memory burdens in large language model (LLM) training, yet their inherent resource heterogeneity, characterized by divergent computation, memory, and bandwidth capabilities, renders manual parallelization strategy optimization both challenging and time-intensive. Automatic parallelization is critical for scaling complex workloads across heterogeneous architectures. However, previous methodologies face significant inefficiencies. First, insufficient pruning of the parameter initialization space results in impractically large search spaces. Second, the prevailing automatic parallel search strategies exhibit suboptimal performance in load balancing and resource constraint adaptation. Third, dynamic parallel strategy tuning incurs substantial overhead due to redundant latency calculations for operators with unchanged configurations, leading to unnecessary computational costs. Therefore, insufficient search space pruning, suboptimal load/resource adaptation, and redundant latency computation are identified as the major bottlenecks in our research. To address these challenges, we propose AutoHAAP ((A) under bar utomated (H) under bar eterogeneity-(A) under bar ware (A) under bar symmetric (P) under bar artitioning), a novel framework incorporating three core innovations: (1) memory-aware initialization to drastically reduce viable search spaces; (2) a heterogeneity-aware load-balancing estimator that guides resource-efficient configuration search; and (3) state caching mechanisms eliminating redundant latency calculations. Evaluations across GPT3 and Llama3 models of varying scales on both homogeneous and heterogeneous clusters demonstrate that AutoHAAP achieves 0.68-98x search efficiency gains, 6.57%-106.9% x throughput improvements in homogeneous environments, and 10.1%-22.28% x throughput enhancements in heterogeneous setups. These results validate AutoHAAP's effectiveness in distributed LLM training on diverse hardware.
Cache compressions are proven effective in improving the performance of caches in conventional processors. They compress data into a smaller size, allowing caches to accommodate more blocks. This helps reduce cache misses and expensive memory accesses, ultimately improving performance. However, conventional cache compression is less effective for energy harvesting systems (EHSs) which experience frequent power failure, as many compressed blocks end up not being used before their loss upon power outage. This wastes hard-won energy, which would otherwise be used for making more program progress. To address this issue, this paper introduces Kagura, an adaptive cache compression extension with frequent power failure in mind. Specifically, Kagura disables cache compression when it finds out that many cached blocks are unlikely to be reused before the next power outage. That way, Kagura avoids the energy waste on useless compressions/decompressions, and the resulting speedup is on par with the ideal intermittence-aware cache compressor. Experimental results show that when combined with an existing cache compressor, Kagura reduces the total energy consumption by an average of 4.53% (up to 16.21%) and improves the performance by an average of 4.74% (up to 17.87%) compared to the baseline EHS without cache compression.
The rise of LLMs has driven demand for private serverless deployments, characterized by moderate-sized models and infrequent requests. While existing serverless solutions follow exclusive GPU allocation, we take a step back to explore modern platforms and find that: Emerging CPU architectures with built-in accelerators are capable of serving LLMs but remain underutilized, and both CPUs and GPUs can accommodate multiple LLMs simultaneously. We propose SLINFER, a resource-efficient serverless inference scheme tailored for small- to mid-sized LLMs that enables elastic and on-demand sharing across heterogeneous hardware. SLINFER tackles three fundamental challenges: (1) precise, fine-grained compute resource allocation at token-level to handle fluctuating computational demands; (2) a coordinated and forward-looking memory scaling mechanism to detect out-ofmemory hazards and reduce operational overhead; and (3) a dual approach that consolidates fragmented instances through proactive preemption and reactive bin-packing. Experimental results on 4 32-core CPUs and 4 A100 GPUs show that SLINFER improves serving capacity by 47% - 62% through sharing, while further leveraging CPUs boosts this to 86% - 154%.
Image synthesis task has recently drawn enormous attention from both the academia and the industry due to the recent advancements of generative models, which now can generate photorealistic images with conditional words from human. Among the generative models for image synthesis task, (V) under bar isual (A) under bar uto (R) under bar egressive (VAR) model is a promising avenue due to its strong scalability. Nevertheless, its exorbitant computational cost poses a formidable obstacle to widespread adoption. To this end, we propose a dedicated software/hardware co-design framework dubbed VAR-Turbo for unlocking the potential of VAR models. Specifically, in the software level, we propose a Draft-Free Parallel Decoding scheme by exploiting the Image Redundancy, which can decrease the sample steps by > 80%, and a combination of Token Aggregation and Dynamic Bypass that capitalizes on the Model Redundancy introduced by the generative Transformer to reduce the computational load by > 60%. In the hardware level, we propose a dedicated accelerator featuring 1) A Unified Attention Core and 2) Radix Sort Core, which can support the aforementioned algorithm pipeline seamlessly and efficiently. Under the collaborative design and synergy of software and hardware, VAR-Turbo achieves averagely 5047.4x, 210.3x, 6.1x, 3.8x speedups and 24818.2x, 423.5x, 6.0x, 7.8x energy-efficiency improvements over Xeon 8168 CPU, Nvidia V100, ViTCoD and AdapTiV, while maintaining the generation quality.
Scaling fault tolerant quantum computers, especially cryogenic systems based on the surface code, to millions of qubits is very challenging due to poorly-scaling data processing and power consumption overheads. One key challenge is the design of decoders for real-time quantum error correction (QEC), which demands high data rates for error processing; this is particularly apparent in systems with cryogenic qubits and room temperature (RT) decoders. In response, cryogenic predecoding using lightweight logic has been proposed to handle common, sparse errors within the cryogenic domain. However, prior work only accounts for a subset of the error sources present in real-world quantum systems with limited accuracy, often degrading performance below a useful level in practical scenarios. Furthermore, prior reliance on SFQ logic precludes detailed architecture-technology co-optimization. To address these shortcomings, this paper introduces Pinball(1), a comprehensive design in cryogenic CMOS of a QEC predecoder for the surface code, tailored to realistic, circuit-level noise. By accounting for error generation and propagation through QEC circuits, our design achieves higher predecoding accuracy, outperforming logical error rates of the current state-of-theart cryogenic predecoder by nearly six orders of magnitude. Remarkably, despite operating under much stricter power and area constraints, Pinball also reduces logical error rates by 32.58x and 5x, respectively, compared to the state-of-the-art RT predecoder and an RT ensemble configuration. By increasing cryogenic coverage, we also reduce syndrome bandwidth up to 3780.72x. Through co-design with 4 K-characterized 22nm FDSOI technology, we achieve a peak power consumption under 0.56mW. Voltage/frequency scaling and body biasing enable 22.2x lower typical power consumption, yielding up to 67.4x total energy savings. Assuming a 4K power budget of 1.5W, our predecoder can support up to 2,668 logical qubits at d = 21.
Processing-in-Memory (PIM) architectures offer a promising solution to the memory bottlenecks in data-intensive machine learning, yet often overlook the growing challenge of activation memory footprint. Conventional PIM approaches struggle with massive KV cache sizes generated in long-context scenarios by Transformer-based models, frequently exceeding PIM ' s limited memory capacity, while techniques like sparse attention can conflict with PIM ' s need for data locality. Existing PIM approaches and quantization methods are often insufficient or poorly suited for leveraging the unique characteristics of activations. This work identifies an opportunity for PIM-specialized activation quantization to enhance bandwidth and compute efficiency. We explore clustering-based vector quantization approaches, which align well with activation characteristics and PIM ' s internal bandwidth capabilities. Building on this, we introduce AQPIM, a novel PIM-aware activation quantization framework based on Product Quantization (PQ), optimizing it for modern Large Language Models (LLMs). By performing quantization directly within memory, AQPIM leverages PIM ' s high internal bandwidth and enables direct computation on compressed data, significantly reducing both memory footprint and computational overhead for attention computation. AQPIM addresses PQ ' s accuracy challenges by introducing several algorithmic optimizations. Evaluations demonstrate that AQPIM achieves significant performance improvements, drastically reducing of GPU-CPU communication that can account for 90 similar to 98.5% of decoding latency, together with 3.4x speedup over a SOTA PIM approach.
With the slowdown of process scaling and the advancement of packaging technologies, multi-chiplet GPUs have emerged as a highly promising architecture to improve the scalability of GPU performance further. Moreover, requiring adherence to atomicity and memory consistency models for shared data efficient synchronization is crucial to leverage the performance advantages of the multi-chiplet GPU architecture. However, the memory systems of multi-chiplet GPUs introduce deeper cache hierarchies and increased non-uniformity, both of which significantly exacerbate the overhead of synchronization. Specifically, acquire/release synchronization operations should invalidate/flush caches, an overhead that is significantly increased by the presence of additional cache level, and atomic operations for synchronization performed across chiplets are further impacted by the limited bandwidth of inter-chiplet links. To address these challenges, this paper proposes LRM-GPU to provide efficient synchronization support for multi-chiplet GPUs. In order to reduce the overhead caused by the additional cache level, LRM-GPU leverages lazy release consistency in multi-chiplet GPUs, whereby the additional level of cache only performs coherence actions when the ownership of synchronization variables changes between different chiplets. LRM-GPU also implements a directory in the last-level cache to track the synchronization variables. To mitigate the overhead of atomic operations for inter-chiplet synchronization under limited interchiplet bandwidth, LRM-GPU proposes an in-network synchronization atomic merging unit to merge atomic requests across chiplets, thereby reducing the inter-chiplet synchronization traffic of atomic operations. Experimental evaluation demonstrates that, compared with the MCM-GPU, LRM-GPU achieves an average speedup of 1.33x. Moreover, compared with the state-of-the-art work HMG, it also achieves the speedup of 1.22x, reduces 52% of inter-chiplet traffic, and reduces 32% of energy consumption on average.
The growing adoption of prediction serving systems (PSSes) has made cost-saving deployment on preemptible spot instances crucial, yet frequent preemptions severely harm availability. While coded computation (CC) can keep availability cost-effectively by encoding original jobs into parity ones, its direct application to spot instances incurs prohibitive decoding overhead and tail latency under frequent preemptions. We identify two findings for optimization: (i) decoding asymmetry (only original job failures require decoding); (ii) preemption unevenness (variation in preemption rates across cloud regions). Leveraging these findings, we propose SpotCC, a new CC framework that strategically dispatches parity jobs to high-preemption (volatile) regions and original jobs to low-preemption (stable) regions. SpotCC designs locality-based and fine-grained volatility identification to reduce decoding operations and mitigate job congestion, respectively, and adaptively tunes configurations for decoding minimization. Experiments show that SpotCC improves P99 latency by 83.9% over state-of-the-arts, while maintaining ultra-low monetary costs.
As NAND flash memory continues to scale, the increasing word-line (WL) stack height and higher program voltage requirements have led to severe peak-current overlaps across dies within a package. Such overlaps cause power-management-IC (PMIC) limit violations, voltage droops, and reliability degradation. Traditional static solutions - such as controller throttling or slope control - cannot fully address these issues because dynamic process, voltage, and temperature (PVT) variations and wear out-induced drift cause both nondeterministic interdie timing mismatches and peak-current magnitude fluctuations, forcing conservative guardbands that limit system parallelism and performance. To overcome these challenges, we propose N DIPPER (NAND Distributed Inter-die Peak PowER Management Network), a cooperative runtime inter-die scheduling framework that dynamically schedules high-current (HC) phases through lightweight in-package (inter-die) fabric. N-DIPPER is based on token-based scheduling across the fabric to serialize scheduling of HC phases while device-aware information is leveraged to avoid utilizing worst-case guardbands. Our evaluation results show that N-DIPPER eliminates PMIC-limit violations (100%) while sustaining 97% of the baseline throughput (without any peak power management) across diverse workloads and die configurations.
Multiple Low-Rank Adapters (Multi-LoRA) are gaining popularity for task-specific Large Language Model (LLM) applications. For Multi-LoRA serving, caching hot LoRAs and KV caches in the GPU memory can improve inference performance. However, existing Multi-LoRA inference systems fail to optimize serving performance like Time-To-First-Token (TTFT), neglecting usage dependencies when caching LoRAs and KV caches. We therefore propose ELORA, a Multi-LoRA caching system to optimize the serving performance. ELORA comprises a dependency-aware cache manager and a performance-driven cache swapper. The cache manager maintains the usage dependencies between LoRAs and KV caches during inference with a unified caching pool. The cache swapper determines the swap-in or swap-out of LoRAs and KV caches based on a unified cost model, when the GPU memory is idle or busy, respectively. Experimental results show that ELORA reduces the TTFT by 45.7% on average, compared to state-of-the-art works.
Rigid Body Dynamics (RBD) computation is a critical component of robotic control, often dominating system runtime due to its algorithmic complexity and high parallelism demands. CPUs suffer from limited parallelism and cache-unfriendly access patterns, while GPUs incur prohibitive memory-access latency and per-task response time, making them unsuitable for real-time control. Both platforms also consume excessive power for edge deployment. FPGAs offer superior latency, energy efficiency, and customizable hardware-level parallelism, emerging as promising targets for RBD acceleration. However, existing FPGA designs still face critical limitations. First, the intensive use of multiply-accumulate operations leads to high Digital Signal Processing (DSP) slices consumption-especially for high degrees-of-freedom (DOF) robots-resulting in limited scalability. Second, RBD functions include mass matrix inversion function, which is inefficient on FPGA due to reciprocal operations falling on the longest latency path, severely limiting performance. Third, mismatched processing rates across modules introduce idle cycles, resulting in poor DSP utilization. To address these issues, we propose DRACO, a hardware-efficient and high-performance RBD accelerator based on FPGA, introducing three key innovations. First, we propose a precision-aware quantization framework that reduces DSP demand by up to 4x while preserving motion accuracy. This is also the first study to systematically evaluate quantization impact on robot control and motion for hardware acceleration. Second, we leverage a hardware-efficient division deferring optimization in mass matrix inversion algorithm, which decouples reciprocal operations from the longest latency path to improve the performance. Finally, we present an inter-module DSP reuse methodology to improve DSP utilization and save DSP usage. Experiment results show that DRACO achieves up to 8x throughput improvement and 7.4x latency reduction over state-of-the-art (SOTA) RBD accelerators across various robot types, demonstrating its effectiveness and scalability for high-DOF robotic systems.
The IBM Z((R)) platform and the underlying processor chip designs supporting it are optimized for processing vast amounts of data and transactions, while delivering consistent system performance, throughput, and response latencies with a sustained processor utilization of over 90% under all workload conditions in a highly virtualized and secured computing environment. The IBM Telum((R)) series of processor chip designs that support the platform introduced the industry to a novel modular scalable heterogeneous processor compute framework with an integrated multi-tier unified cache hierarchy all within one chip. The processor chip design leverages a unique approach to ensure all elements work in unison to continuously deliver performance to the evolving needs of mission critical workloads running on the platform while responding to those changing demands at processor clock speeds. This paper will detail the varying compute, accelerator, and cache units within the IBM Telum II processor chip design, how they adaptively work in unison without generating a cacophony of agents competing for scarce hardware resources, and how this forms the backbone of the scalable multi-processor system that our modern economy is built upon.
Artificial intelligence (AI) has been increasingly applied to gaming, with large language models (LLMs) playing a key role in character control. However, efficiently co-locating game rendering and LLM inference on one GPU presents challenges due to resource constraints, diverse latency requirements, and fine-grained task scheduling. We propose LEGO, an algorithm-system co-design that enables the efficient co-location of LLM inference and game rendering tasks. Algorithmwise, LEGO features a resource-oriented layer-skipping adaptor, which distills knowledge from skipped layers to reduce computational demand while maintaining inference accuracy. System-wise, LEGO proposes a headroom-maximizing LLM scheduler, which dynamically partitions inference tasks to utilize available rendering headroom. Evaluations on an Nvidia RTX 4090 show that LEGO meets latency targets in all scenarios, improves rendering headroom utilization by up to 28.6 %, and reduces LLM inference accuracy loss by up to 86.3 % compared to current layer-skipping approaches.
The growing complexity and parallelism demands of modern GPU workloads have driven architectural innovations toward asynchronous tile transfers (ATTs) to overlap computation and data movement. While ATT units such as the NVIDIA's Tensor Memory Accelerator (TMA) introduce high-throughput memory transfers, programmers must deal with wavefront specialization, select tile sizes, queue slots, and synchronization primitives, all of which are hardware-specific and workloaddependent. Existing GPU libraries fall short-offering limited ATT support and configurability-so developers still resort to manual exploration of this vast parameter space, which is laborious, error-prone, and fundamentally limits performance portability across GPUs. In this work, we present QuCo (Queue Configurator), a single lightweight hardware unit embedded in the GPU that fully automates the ATT configuration process. Inspired by Blackwell GPU design, QuCo includes a compact RISC-V processor, small memory structures for instructions and data, and a GPU Specification Table (GST) storing key architectural parameters. Using the GST and workload characteristics, along with built-in heuristics, QuCo computes optimal queue configurations at kernel launch. This relieves the programmer of the tedious, time-consuming task of tuning and offline profiling, while simultaneously increasing post-compilation performance portability.
Quantum Optimal Control (QOC) frameworks are powerful tools for compiling quantum circuits into low-latency hardware control pulses, but recent studies suffer from two critical limitations: lengthy compilation times and potential logical inconsistencies from flawed gate grouping strategies. In this work, we introduce d'ArQ, a novel QOC framework that solves these challenges. (i) We identify and resolve the causality problem, a flaw in greedy partitioning that can produce invalid schedules, by introducing a DAG-based grouping algorithm with assigning mergeability to each group so that it guarantees logical correctness. (ii) To mitigate compilation times, we use a pre-computed library of pulses derived from random unitary matrices to provide a high-quality random initialization for pulse optimization. (iii) Diverging from prior work based on GRAPE, d'ArQ is built on the GOAT algorithm. We demonstrate that the choice of analytic basis is a critical hyperparameter and introduce a heuristic cost model to dynamically select the optimal basis for each synthesis task, improving pulse performance. When evaluated against the state-of-the-art baseline PAQOC on a realistic, inhomogeneous hardware model, d'ArQ demonstrates superior performance. Notably, d'ArQ reduces circuit latency up to 22.8% and compilation time up to 56.8%, establishing a more robust and physically realistic path for circuit compilation.
Important applications like quantum search and quantum simulation rely on control flow implemented by oracle circuits, which may account for up to 99% of the total gate count [1] as oracle circuits consist of expensive multi-controlled (MC) quantum operations that formulate the control flow. Existing efforts optimize them by simplifying Boolean expressions associated with MC gates. They merge MC gates, if associated Boolean expressions are adjacent, to reduce the gate cost. Being more aggressive, our work introduces control line encoding (CLE), a technique that even allows merging non-adjacent MC gates by modifying their control (line) polarity. We further develop a plug-in compiler framework that efficiently applies this technique. The framework is grounded in the following observations: 1) Control flows often commute with one another and can be globally reorganized to maximize their merging by CLE; 2) numerous CLE schemes exist to allow gate merge and we can co-design CLE applications to diminish the overhead of CLE itself. Experiments demonstrate that, compared to the baseline, our framework further reduces the CX and T cost, on average by 54.7% and 56.8%, respectively.
While distributed training is crucial for scaling deep learning models, it incurs significant overhead due to the collective communication of gradients. To alleviate the burden, compression techniques are commonly used to improve network bandwidth utilization. However, compression poses challenges for synchronized AllReduce collective communications, even more so in scalable systems. Non-uniform data sizes resulting from compression can cause bandwidth under-utilization, as faster nodes remain idle while waiting for slower nodes to complete data exchanges, increasing overall communication and consequently, training time. However, the inherent similarity in gradients across consecutive batches presents an opportunity to mitigate these inefficiencies. By leveraging the quantization of gradients and consistent distribution of zeros, the gradients can be partitioned logically to speedup communication. Splitting them into groups with and without zeros can allow different compression approaches for both. The bandwidth under-utilization due to non-uniform data size can also be solved by partitioning the gradients into variable-sized chunks, leading to more balanced compressed data sizes and reduced idle waiting time. We propose two novel strategies in OSCAR, where gradient splitting is designed to improve communication and training. OSCAR- SW is a novel software-based technique supporting direct AllReduce that splits gradients into probable zeros and non zeros to apply count sketch compression. OSCAR-HW, a novel hardware/software co-designed gradient splitting technique is proposed with ASC (Adaptive Stepwise Coding), an encoding technique for gradient compression in distributed training. OSCAR- HW dynamically splits fixed-point quantized gradients for AllReduce communications and maximizes bandwidth utilization for state-of-the-art hardware compression techniques. ASC is a variant of Adaptive Arithmetic Coding (AAC) that generates a distinct probability table for each timestep of AllReduce to adapt to its unique value ranges and avoids sending the probability table during the communication of gradients. Our experimental results show that OSCAR-SW achieves 1.22x speedup and 7% better accuracy over the SOTA CountSketch algorithm. OSCAR- HW achieves an average AllReduce speedup of 3.77x, and an average end-to-end training speedup of 1.38x. ASC achieves an average AllReduce speedup of 1.05x over Atalanta and 4.66x over no compression.
3D Gaussian Splatting (3DGS) is a breakthrough in 3D reconstruction using 3D Gaussians. However, even on high-end GPUs like the NVIDIA A100, reconstructing complex scenes remains time-consuming, taking over 15 minutes. The main bottleneck is a-computation, which accounts for 71.25% of training workload, yet 93.03% of it is invalid due to the localized influence of Gaussians. To address this issue, we propose Cambricon-GS, an accelerator for 3DGS training with Gaussian-Pixel hybrid parallelism. At the software level, we introduce a hybrid parallel workflow that breaks the limitation of conventional pixel-only parallelism through two key techniques: Center-Pixel Gaussian Culling (CPGC), which eliminates invalid Gaussians early, and Seed-Driven Gaussian Region Exploration (SDGRE), which reduces invalid computation for partially valid Gaussians by selectively exploring valid regions. Overall, the workflow significantly reduces a computations, lowering the workload to 17.99%. At the hardware level, Cambricon-GS decouples alpha-computation and alpha blending into GUnits and PUnits, organized in a 2D mesh-based NoC that supports asynchronous execution and efficient data routing. We further boost performance via Gaussian/Pixel load balancing and tiled SSIM based pipelining. The evaluation results show that Cambricon-GS achieves 19.63x, 14.86x, 15.42x, 2.98x and 2.63x speedup, and 78.62x, 63.00x, 61.72x, 3.89x and 3.22x energy saving, compared to A100, GSCore, GBU, GSArch, and GauSPU, respectively, with negligible image quality loss.
Hardware prefetching is a well-established technique for bridging the processor-memory performance gap. To improve cache miss coverage, modern processors often integrate multiple prefetchers. However, multi-prefetcher systems without proper management often suffer from suboptimal performance due to a surge of useless prefetches. Several techniques have been proposed to select appropriate prefetchers for issuing requests, but they all face limitations. Specifically, existing (1) static schemes lack feedback regulation mechanisms and suffer from inflexible prefetcher selections; (2) reinforcement learning (RL)based schemes incur high overhead and suffer from adjustment lag; and (3) performance-counter-based schemes rely on inefficient runtime metrics that fail to accurately and clearly reflect a prefetcher ' s true impact on performance. In this paper, we propose I-POP, a high-performance and low-over-head prefetcher management scheme for multi-prefetcher systems. I-POP introduces a novel runtime metric, Prefetch Effectiveness (PE), which aggregates each prefetch request ' s beneficial and harmful effects to precisely quantify the impact of a prefetcher on performance, effectively overcoming the limitations of prior metrics. To compute and leverage this metric, I-POP incorporates two key components: the Metric Collector, which periodically calculates each prefetcher ' s PE, and the Control Engine, which dynamically manages all prefetchers based on their PE values. Specifically, I-POP ignites (enables) prefetchers with positive PE values, adaptively tuning their aggressiveness, and disables those with non-positive PE. We evaluated I-POP on numerous workloads, and the results show I-POP outperforms two state-of-the-art approaches, Bandit and Alecto, by 4.2% and 3.5% across three benchmark suites in a single-core system, and 6.6% and 8.6% in a 16-core system, while incurring only 1.46 KB of storage overhead.
A recent advancement in quantum computing shows a quantum advantage of certified randomness on the racetrack processor. This work investigates the execution efficiency of this architecture for general-purpose programs. We first explore the impact of increasing zones on runtime efficiency. Counterintuitively, our evaluations using variational programs reveal that expanding zones may degrade runtime performance under the existing scheduling policy. This degradation may be attributed to the increase in track length, which increases ion circulation overhead, offsetting the benefits of enhanced parallelism. To mitigate this, the proposed Plutarch exploits 3 strategies: (i) unitary decomposition and translation to maximize zone utilization, (ii) prioritizing the execution of nearby gates over ion circulation, and (iii) implementing shortcuts to provide the alternative path.
