Large language models (LLMs) require inference systems that can handle both compute- and memory-intensive workloads. GPUs and NPUs (referred to as xPUs) efficiently process compute-intensive layers, while Processing-In-Memory (PIM) architectures are well suited for memory-bound stages. To exploit this complementary relationship, recent LLM inference systems have adopted heterogeneous architectures integrating xPUs with PIM units. However, this integration poses several challenges. Tight execution dependencies between the two devices limit concurrency, as PIM often must wait for data produced by the xPUs. Moreover, batch size and sequence length affect the computational load on PIM and the accelerator differently, leading to execution imbalance across devices and degrading their utilization. To address these challenges, we propose LibraPIM, a novel PIM framework that orchestrates workload rebalancing and concurrent execution between compute accelerators and in-memory compute units, enabling efficient and scalable LLM inference. LibraPIM addresses the above challenges through two key techniques: Dynamic Batch Offloading (DBO), which adaptively redirects portions of the workload to the underutilized device based on runtime profiling, and Dual-Path Execution (DEX), which enables concurrent PIM operations and memory accesses through sub-bank partitioning with minimal hardware overhead. Together, these techniques improve resource utilization across heterogeneous devices, thereby increasing system throughput in LLM inference. Our experimental results demonstrate that LibraPIM achieves $6.2 \times$ average speedup over the baseline PIM-enabled system and delivers a $2.1 \times$ average speedup compared to a state-of-the-art approach.
Sparse-sparse matrix multiplication (SpGEMM) is a well-studied problem on CPUs, GPUs, accelerators (e.g. FPGAs), and distributed systems. The main computational bottleneck in SpGEMM is the reduction process, which involves matching indices to accumulate partial products that map to the same output locations and requires irregular memory accesses. Efficient implementations must use the memory hierarchy effectively so that this reduction is done in fast local (cache) memory as much as possible. This is challenging, especially on GPUs, where the local memory is managed explicitly, as different rows of the result may have vastly different numbers of nonzero elements and vastly different numbers of partial products required to produce the output row which may not fit in the local memory. We demonstrate an SpGEMM implementation on GPUs which perfectly partitions the partial products into equal-size blocks such that each block maps to disjoint output locations. In this way, the block size can be chosen to maximize local memory, all reductions happen in local memory, and each block can be processed independently without communication or data from other blocks. We show how this partitioning can be achieved by solving many instances of a multiway merge partitioning problem. There are several algorithms in the literature for solving this problem. We present the mathematical formulation, missing from all papers providing algorithms for this problem, and show that this is a useful framework for parallelizing it efficiently for GPUs. To our knowledge, this partitioning scheme has never been applied to SpGEMM. We evaluate our SpGEMM implementation, MMSpGEMM, with respect to state-of-the-art implementations including cuSPARSE, TileSpGEMM, spECK, and AC-SpGEMM, achieving speedups of up to $5.3 x, 10.0 x, 1.3 x$, and $1.1 x$, respectively, on a select set of 20 SuiteSparse matrices, and much higher on synthetic matrix multiplications with different left and right matrices. Finally, we discuss improvements and limitations and suggest ways to incorporate the idea into more general SpGEMM routines.
Optimizing convolution operations is critical for enhancing the performance of convolutional neural networks (CNNs). The Winograd convolution algorithm, renowned for its significant reduction in computational complexity, has been widely adopted in convolution acceleration. While the Winograd convolution algorithm has been highly optimized for traditional computing platforms such as SIMD-based CPUs and SIMT-based GPUs, research on alternative architectures better suited for convolution operations remains ongoing. MIMD architectures, characterized by their high parallelism and thread divergence mitigation capabilities, present promising potential; nevertheless, their applicability and performance in convolution operations remain underexplored. Additionally, GPU-based convolution acceleration, despite delivering exceptional performance, faces escalating energy consumption challenges, necessitating a balanced optimization of hardware performance and energy efficiency. To address these, we implement and optimize the Winograd convolution algorithm on a low-power MIMD many-core processor PEZY-SC3s, aiming to investigate its viability for convolution workloads. Experimental evaluations using convolutional layer parameters from widely adopted CNNs demonstrate: 1) 78.37% average bandwidth utilization in data-intensive stages; 2) 98% single-core and 92.52% system-wide computational efficiency in compute-intensive phases. The optimized Winograd algorithm achieves significantly superior computational efficiency compared to cuDNN-based implementation on Nvidia A100 GPU and oneDNN-based implementation on Intel Xeon Silver 4314 CPU, with energy efficiency ratios (GFLOPS/W) of $2.58 \times$ and $27.52 \times$, respectively.
As Deep Learning (DL) training has emerged as a dominant workload on clusters in recent years, there has been significant interest in parallelizing individual models and scheduling them efficiently. However, to date, no prior work has taken an integrated approach to addressing both problems while simultaneously accounting for heterogeneity in the execution environment. This paper introduces, demonstrating that the integrated optimization of parallelization and scheduling is both feasible and highly effective. The main insight of our work is to generate a set of possible target resources for each job and to develop corresponding parallelization strategies for each. Each jobstrategy pair is then treated as a scheduling candidate. Scheduling decisions are made by extending prior Linear Programming (LP)based formulations and incorporating new heuristics to reduce decision overhead.
Graphics Processing Units (GPUs) offer high computational throughput and memory bandwidth, making them promising accelerators for relational OnLine Analytical Processing (OLAP). GPU-accelerated relational OLAP executes the relational operations of a Structured Query Language (SQL) query on a GPU instead of the host Central Processing Unit (CPU). Depending on where input columns and their values reside in, a GPU-accelerated SQL query execution can be classified into two scenarios: 1) in-GPU, in which all input columns fit in the GPU memory, or 2) in-host, in which the input columns reside in the host memory and get transferred to the GPU memory when needed. However, both scenarios incur significant intraGPU and host-to-GPU data movement overheads, respectively. In-GPU executions incur excessive GPU cache misses and thus frequent off-chip GPU memory accesses. In-host executions suffer from the limited host-to-GPU data transfer bandwidth. This paper presents DMO-DB, a Data Movement-Optimized GPU-accelerated relational OLAP engine. Since modern GPUaccelerated relational OLAP decomposes SQL queries into multiple pipelines-sequences of relational operations that can be executed on input columns from the same table, DMO-DB leverages inter-pipeline dependencies to overcome the two data movement bottlenecks. DMO-DB introduces two key ideas: cache-fit bloom filtering and Ahead-of-Time value Discarding (AoTD), which preemptively eliminate unnecessary input values before their movement across the memory hierarchies. For in-GPU execution, GPU L1 data cache-fit filters discard non-contributing values before triggering costly off-chip DRAM accesses. For in-host execution, host CPU last level cache-fit filters strategically prune unnecessary input values, minimizing PCIe transfer overhead. After that, AoTD exploits multiple inter-pipeline dependencies by collecting these cache-fit bloom filters to earlier pipeline execution stages. Our evaluation using NVIDIA RTX A4000 and TITAN RTX GPUs shows that DMO-DB achieves speedups of $\mathbf{1. 5 3 x}$ over in-GPU Crystal-Opt and 6.10x over in-host HeavyDB.
