Mixture-of-Experts (MoE) LLMs route each token to a few expert FFNs, making decode-time expert execution bandwidth-sensitive and vulnerable to routing imbalance. Logicdie near-memory processing (NMP) provides high memory bandwidth and capacity for expert-weight storage and access, but scaled-out NMP serving with expert-level mappings suffers from routing-induced load imbalance across chips, leaving some chips overloaded while others idle. Increasing Expert Tensor Parallelism (ETP) can reduce imbalance, but exposes intra-/interchip communication. We propose Split-GEMM, a fine-grained mapping strategy that decomposes expert FFNs into GEMMlevel pipeline stages and schedules expert-local tiles in fixed-size tile groups. By balancing chip-level work and communicating intermediate-activation shards, Split-GEMM achieves up to a 2.58× speedup over the best evaluated EP/ETP baseline.
This paper argues for the Bias-Corrected and Accelerated (BCa) bootstrapping method to add statistical rigor by computing confidence intervals when reporting average performance evaluation metrics in computer architecture and systems research and development. BCa can be computed for any average metric and for small data sets that follow a skewed, non Gaussian distribution, illustrating its broad applicability. The paper further demonstrates the importance and relevance of confidence intervals in concrete case studies.
Large language model (LLM) inference is increasingly constrained by accelerator memory capacity rather than compute throughput. High-bandwidth flash (HBF) is an emerging 3D-stacked NAND flash technology that provides multi-terabyte near-accelerator capacity, making it a promising capacity tier for storing LLM weights. However, existing HBF-based proposals face three adoption challenges: they (i) rely on coarse-grained static prefetching, (ii) expose NAND flash refresh to the acceleratorvisible critical path, and (iii) assume solid-state drive (SSD)-class flash-management mechanisms that are poorly matched to a read-only HBF weight tier. Our goal is to design an efficient HBF substrate that integrates HBF as a memory capacity tier alongside high bandwidth memory (HBM) while addressing these three challenges. To this end, we propose FLINT (flash inference tier), a workload-driven HBF substrate for capacity-scalable LLM inference. FLINT introduces three mechanisms: (i) a hardware burst-buffer controller controller, which pipelines HBF reads with acceleratorside memory requests using existing NAND flash page/cache buffers, (ii) a phantom-plane refresh mechanism, which removes refresh from the LLM decode critical path, and (iii) a read-only flash translation layer (FTL), which replaces SSD-class support for arbitrary writes with a compact table that translates logical weight bursts to physical HBF locations.We evaluate FLINT across dense and mixture-of-experts (MoE) LLM inference models. Our evaluation shows that FLINT improves LLM decode throughput by 355×, 3.0×, and 1.17× and reduces energy consumption by 294×, 2.5×, and 1.14× compared to an SSD-equipped graphics processing unit (GPU) system, an HBM-only GPU system, and a prior hybrid HBM+HBF GPU system, respectively. FLINT adds small area cost to an HBF die (3.1%).
Data prefetching is a key component in modern computer architecture to sustain performance requirements. Prior data prefetchers face a tradeoff between area budget and prediction accuracy, and capturing complex memory access patterns with hidden regularities remains challenging. We propose a lightweight yet strong data prefetcher leveraging fully online perceptron learning to predict non-periodic but correlated stride patterns. We identify and address a weight drifting problem, where perceptron weights saturate and prevent predictions, through a weight and bias centering technique. On memory-intensive traces, experimental results show a $\sim$21% geometric-mean IPC improvement over no prefetching, and $\sim$8% over the best-performing baseline evaluated, under the same area budget. To the best of our knowledge, this is the first lightweight prefetcher to employ perceptron-based learning for direct memory stride prediction.
Large language model (LLM) serving is increasingly agentic, with multi-turn sessions that idle between actions yet must retain their full context. Limited GPU memory capacity forces inactive KV states to be evicted, so resuming a session incurs either costly recomputation or slow interconnect transfers. To address this, high bandwidth flash (HBF)-an on-package 3D-NAND memory offering orders-of-magnitude greater capacity than high bandwidth memory (HBM) at comparable read bandwidth-has emerged as a strong candidate. However, its high read energy and limited write endurance make it impractical to serve all KV traffic. Fortunately, our analysis shows that agentic KV states exhibit distinct access patterns: a small hot set is read for every decoding step, while a large cold pool is read only when a paused session resumes. Exploiting this, we place the hot set in HBM and the cold pool in HBF, forming a hot-cold KV hierarchy within the GPU memory tier. On agentic workloads with Qwen3-Coder-30B-A3B, our design delivers 14 ms time-between-tokens (TBT) and adds only $\approx \!0.1$ ms of resume latency on top of prefill, while hosting $24\times$ more concurrent sessions per GPU. By confining HBM to the hot set, our design also cuts read power by 7.6 kW per 8-GPU node relative to serving all KV from flash-establishing HBF as a cold-tier complement to HBM rather than its replacement.
In long-context LLMs, attention is costly due to repeated accesses to a Key–Value (KV) cache. HBM-PIM can accelerate this memory-bound attention by leveraging its massive internal bandwidth. To effectively exploit this parallelism, prior PIM accelerators have proposed request-wise, head-wise, and block-wise KV-cache mappings to distribute attention workload across channels. However, these mapping schemes do not fully exploit the available internal bandwidth because they do not account for load changes caused by asynchronous request completion. To address this limitation, we propose DYCA-HBM, a dynamic channel allocation scheme that maintains balanced channel utilization across the entire decoding phase. DYCA-HBM tracks how each request's KV is distributed across all channels. It maps new KV blocks to the least-loaded channel for load balancing, and prevents a single request's blocks from excessively accumulating in specific channels. Experimental results show that DYCA-HBM improves throughput by up to 1.08× and 1.07× over AttAcc and BlockPIM, respectively, while reducing energy consumption by up to 7% with less than 0.07% logic-die area overhead.
Large Language Models (LLMs) have proliferated as the driver of many modern applications, including coding agents, personal assistants, and chat bots. Because LLM execution consists primarily of General Matrix Multiplies (GEMMs) and General Matrix Vector Multiplies (GEMVs) over hundreds to thousands of gigabytes of parameters, GPUs equipped with High Bandwidth Memory (HBM) have become the dominant platform for serving them. However, the plateau in DRAM scaling has placed a ceiling on the capacity attainable through HBM alone. High Bandwidth Flash (HBF) is an emerging memory technology that addresses this ceiling by vertically stacking 3D NAND dies to deliver approximately 512 GiB of capacity per stack, offering 16x greater capacity and bandwidths similar to that of HBM. NAND reads, however, incur latencies over three orders of magnitude higher than DRAM, and when exposed to GPUs directly, cause over 3x lower throughput (IPC). In this work, we propose a migration-based Heterogenous Memory Architecture (HMA) that improves performance over an HBF baseline by 1.56-1.70x while offering over 16x more capacity compared to an HBM only baseline, enabling the execution of 100B+ parameter models on a single GPU.
Recent GPUs increase aggregate memory bandwidth by integrating multiple HBM stacks, but memory bandwidth scaling is limited by GPU die shoreline. In this work, we propose Memory Fabric – an in-package memory interconnect that leverages un-utilized “memory” shoreline in custom HBM base dies to directly interconnect neighboring HBM stacks using die-to-die (D2D) links. As a result, Memory Fabric increases the effective bisection bandwidth of multi-chiplet GPU packages while enabling direct memory-to-memory transfers without routing data through compute chiplets. To enable these capabilities, each HBM stack in Memory Fabric introduces a lightweight Data Movement Engine (DME) that offloads common memory-to-memory operations, including data-layout transformations for mixture-of-experts (MoE) dispatch and combine, as well as collective communication pack and unpack operations. Evaluations show that Memory Fabric improves remote-HBM bandwidth by up to 75% and accelerates single-layer MoE execution by up to 36% through DME-assisted data movement offloading.
Query-Key Normalization (QK-Norm) improves the training stability and quality of modern Large Language Models (LLMs). However, under Tensor Parallelism (TP), layerwise QK-Norm introduces additional cross-GPU communication because the normalization factor depends on the full hidden vector. We present SwiftQK, a multi-GPU RMSNorm kernel that exchanges only scalar normalization statistics and overlaps the remaining Peer-to-Peer reduction with independent element-wise computation in a deadlock-safe persistent kernel. Evaluations on recent LLMs show that SwiftQK reduces QK-Norm latency by 81.4–93.9
As the memory need for KV cache grows rapidly with the advent of multi-turn inferences and extremely long context requirements, limited GPU memory has become a critical bottleneck. While offloading the KV cache to CPU DRAM or SSDs has been explored, these solutions face limitations in capacity, latency, and scalability. Compute Express Link (CXL) memory offers a promising alternative that provides low-latency, disaggregation, and scalability benefits. In this paper, we evaluate KV cache offloading using various novel CXL memory architectures to compare with existing solutions. We tested memory expanders, CMM-D and CMM-H, and memory pooling solutions, including CMM-B and CMM-MP. For performance evaluation, vLLM, coupled with the LMCache framework, is employed to characterize five offloading targets. This comprehensive study provides insights into KV cache offloading tiers that can be used by end users to optimize their AI data center system architectures. CXL memory devices show significant performance benefits for KV cache offloading, with CMM-MP's novel memory pooling architecture offering a particularly scalable and cost-efficient solution.
Prefetchers are built and evaluated as predictors: is this line likely to be used? We argue that on a bandwidth-constrained multicore the operative question is instead one of admission — is this line worth the shared bandwidth it will consume, given who else is on the bus — and we show the two questions have different answers. Prediction quality alone does not settle the second one: across eight 4-core mixes spanning 14 SPEC CPU2006/2017 traces, the state-of-the-art learned filter PPF [7] leaves the bandwidth-bound co-runner 28–44% slower than with no prefetching at all, and an accuracy-feedback throttle (FDP [9]) narrows this only to 12–14%. This is not an artifact of how those mixes were assembled: on eight further mixes drawn at random from the same pool, PPF leaves the worst core below the line in all eight. We then locate the structural reason in the locality of the control signal: a mechanism driven by a global rate, bandwidth, or accuracy signal still emits at least one prefetch per trigger, and even a global-signal admission variant — the same gate, keyed on a bandwidth window — cuts traffic to 1.25× yet leaves the victim below the no-prefetch line. Among the mechanisms we evaluate, only per-candidate demand confirmation (detect $\to$ admit $\to$ issue) localizes a zero-prefetch state to exactly the candidates that harm a co-tenant. A 768-byte gate instantiates the idea and carries across three out-of-order detectors, with intervention scaling with over-speculation rather than detector identity. The resulting trade is explicit and tunable rather than free: admission protects the latency-sensitive victim (1.04 vs. PPF's 0.69 in mix A) at 1.10× vs. 1.72× DRAM traffic, but it reduces worst-core harm more often than it removes it, and PPF keeps the aggregate-throughput lead. We map when that trade is worth making, and when it is not.
LLM inference is dominated by GeMV and softmax computations during attention. These computations impose substantial compute and memory bandwidth demands. Prior works have demonstrated that HBM-based PIM architectures effectively accelerate GeMV. However, softmax remains a critical bottleneck due to frequent data movement. In this work, we make the case for computing softmax operations in memory and integrate them with in-memory GeMV operations to provide a comprehensive in-memory LLM accelerator engine. We propose SoftmaxPIM, an in-HBM compute architecture that tightly integrates GeMV with a lightweight near-bank softmax unit. The proposed softmax unit employs an approximation-based softmax formulation that removes reduction operations, enabling the entire GeMV–softmax pipeline to execute within the HBM memory stack. By eliminating host-memory transfers, the proposed design significantly reduces softmax latency. Our simunlation results demonstrate up to 42.7× speedup over a CPU and 2.64× over an RTX3090 GPU, as well as up to 1.45× improvement over the prior in-memory attention accelerator AttAcc.
Multi-Tile Accelerators (MTAs) have emerged as a promising paradigm to scale the computational throughput of NPUs for modern deep learning workloads. However, existing NPU simulation frameworks are insufficient for evaluating modern MTAs, as they fail to capture the inter-core data transactions that serve as the primary spatial data reuse mechanism in MTAs. To bridge this gap, we introduce NeuroMTA, a cycle-accurate simulation framework that couples a reconfigurable hardware template with a compiler stack. The hardware template models both NUMA-based and FIFO-driven data sharing, and the compiler stack exposes fine-grained control over data reuse patterns across multiple compute tiles. Our experimental validation against a real MTA device shows that NeuroMTA achieves a Pearson correlation of 0.9.
Large Language Models (LLMs) have intensified the memory wall during inference, where performance is increasingly limited by memory bandwidth rather than computation. While Processing-In-Memory (PIM) and model compression techniques mitigate this issue, existing sparse PIM designs rely on CSR/COO formats that assume ultra-high sparsity (e.g., $\gt 90\%$), which is difficult to achieve in practical LLMs without significant accuracy loss. Moreover, fine-grained indexing introduces substantial metadata overhead, diminishing the benefits of pruning, especially under low-bit quantization. To address this mismatch, we propose BlockPIM, a PIM architecture that employs hardware-aligned coarse-grained pruning to efficiently execute sparse LLMs on standard dense computation units. BlockPIM replaces per-element indices with compact bitmaps and contiguous non-zero blocks, enabling sparse workloads to be processed directly on dense execution pipelines without complex index decoding or irregular control flow. Experimental results show that BlockPIM achieves up to 8.0× and 3.6× speedups over the NPU and state-of-the-art sparse PIM baselines, respectively, while reducing energy consumption by up to 95.2% and providing superior memory efficiency under practical accuracy constraints.
Securing CXL-based tiered memory introduces substantial overhead. To ensure the confidentiality, integrity, and freshness of data stored in CXL memory, transmitting security metadata over the CXL link consumes link bandwidth and increases memory access latency. In addition, re-encrypting migrated pages makes page migration more expensive. To address these challenges, we propose two mechanisms. First, Self Protective CXL Memory (SPCM) secures memory by offloading the cryptographic hardware components and decoupling BMTs across memory tiers, eliminating the need to fetch security metadata over the CXL link. Second, Temporary Seed Encryption (TSE) removes migration overhead by assigning a dynamic page-level temporary seed to each migrated page. Our evaluation demonstrates that the proposed mechanisms reduce the total execution time by 65.4% compared to the baseline system.
Existing processing-in-memory (PIM) compilers commit to bit-serial execution and so suffer from low utilization at small data-parallel widths, expensive multi-cycle arithmetic on wide types, and rigid lockstep control that excludes mixed precision. Configurable PIM hardware that exposes both bit-parallel (BP) and bit-serial (BS) modes can address these limitations, but enlarges the compiler search space across data layouts, tile shapes, execution modes, instruction mappings, and CPU/PIM placement. We present Meta-PIM, an end-to-end compiler that handles this joint problem. Its core is two equality-saturation stages sharing a unified cost model over arithmetic operations, per-tile memory access trace, and inter-slice data movement: TileSat for polyhedral loop scheduling, and a per-statement layout/mode optimizer for CPU/BP/BS execution with explicit conversion costs. Evaluated on a cycle-accurate gem5 model, Meta-PIM extends automatic PIM compilation to irregular applications that no prior tool supports, achieves a 5.85× geomean speedup over CPU across both regular and irregular workloads, and delivers a 1.14× improvement over the state-of-the-art bit-serial PIM compiler on regular workloads. At a representative medium-parallelism operating point ($P{=}128$), configurable BP further outperforms a BS-only design by 2.67× geomean, an advantage that is parallelism-dependent.
