Loops constitute a small fraction of static code but dominate dynamic execution time and control overhead, posing significant challenges for energy-efficient mobile and embedded processors. This paper introduces Loop Semantic Recognition (LSR) , a microarchitectural technique that dynamically infers loop semantics to eliminate loop control overhead without requiring compiler support or ISA extensions. We further propose three complementary optimizations: Branch Strength Reduction (BSR), Hardware Loop Unrolling (HLU), and Stride Value Prediction (SVP), which progressively remove redundant control instructions, reduce pipeline pressure, and optimize loop execution. Evaluation on TSVC, PolyBench, NPB, and Swan benchmarks demonstrates that LSR and its extensions deliver substantial efficiency and energy benefits. HLU reduces committed instructions by up to 33.23% , while SVP lowers executed instructions by as much as 46.59% . LSR also reduces loop branch mispredictions and consistently decreases dynamic energy consumption, achieving an average processor-wide Energy-Delay Product (EDP) improvement of 8.36% across these four benchmark suites on a 2-wide baseline core. Scaling to a 4-wide core further raises the average EDP reduction to 15.39% . Additionally, LSR maintains consistent effectiveness across compiler optimization levels from -O1 to -O3 , confirming its robustness to code generation strategies. A complementary evaluation of SPEC CPU2017 shows an average EDP reduction of 2.93% , indicating that LSR optimization opportunities also exist in general-purpose workloads. These results are achieved with only 0.45 KB of total storage overhead, confirming LSR as a lightweight and practical solution for improving energy efficiency across mobile and embedded processor designs.
The continuous growth in processor speed and the persistent memory latency bottleneck make large-capacity cache critical for mitigating the “memory wall”. However, topology-oblivious OS scheduling frequently induces severe cache contention and cross-die migrations, diminishing these hardware benefits. To address this, we propose a thread scheduling framework driven by runtime microarchitectural feedback. Instead of treating performance profiling and thread placement as isolated steps, our approach directly uses hardware performance counters to guide thread configurations to maximize cache utilization. First, the Thread and CPU Monitoring (TCM) framework tracks CPU and cache contention across both Simultaneous Multi-Threading (SMT) and non-SMT modes. Based on this feedback, the Thread Extension and Placement (TEP) algorithm dynamically adapts thread concurrency. To balance computational efficiency and runtime overhead, TEP determines the optimal thread count using cubic spline interpolation combined with a golden ratio search. Concurrently, a topology-aware affinity strategy binds threads to specific cores to maintain cache locality. Evaluations on AMD EPYC processors show that this approach improves overall execution efficiency. On average, the framework reduces the L3 miss rate from 62.71% to 51.89%, lowers CPI from 2.35 to 2.02, decreases cross-core thread migrations from 36902 to 33339, and reduces wall-clock time by 10.52%. The framework operates in user space without kernel modifications, ensuring ease of practical deployment for HPC and AI inference.
Zero-Knowledge Proofs (ZKPs), particularly zk-SNARKs, have become a cornerstone for privacy-preserving protocols and blockchain scalability. However, the generation of proofs remains computationally prohibitive due to the immense overhead of underlying cryptographic primitives, namely Multi-Scalar Multiplication (MSM) and Fast Number Theoretic Transform (NTT). As applications scale up to billion-scale constraints, conventional GPU-accelerated frameworks tailored for smaller instances suffer from overwhelming memory usage, invalid architectural assumptions, and suboptimal end-to-end orchestration. Driven by the motivation to complete massive proof generation on a single memory-constrained node, this paper presents LightZK, an efficient GPU acceleration framework for large-scale zero-knowledge proofs based on a tile-based execution model. LightZK not only aims to support ultra-large-scale ZKP generation, but also strives to improve the practical performance of proof generation. LightZK contributes a micro-architectural diagnosis of MSM on GPUs: hardware performance counters expose the register-pressure and low instruction-level-parallelism mismatches that under-utilize the GPU during bucket-merge computation, and targeted kernel optimizations derived from this diagnosis yield a 1.5 × –2.5 × MSM speedup over the SOTA baseline. LightZK further makes large-scale NTT executable where the SOTA GPU baseline fails, via storage and structural optimizations that deliver nontrivial acceleration (2.2 × –2.5 × over a functional baseline). At the systemic level, LightZK builds on a data-dependency graph (DDG) analysis of the Groth16 proving flow to drive a three-tier pipeline orchestration: a hand-crafted DRAM-VRAM pipeline for Poly-phase that overlaps sparse matrix operations with data transfer, and a producer-consumer disk-DRAM-VRAM pipeline for Group-phase that treats each MSM tile as an independently schedulable unit. Experimental results demonstrate that LightZK achieves a 2.8 × end-to-end performance improvement over the leading executable GPU baseline at standard scales. Crucially, while existing GPU frameworks exceed available memory at billion-scale, LightZK successfully sustains 2 30 -scale proof generation with near-linear scalability on a single node.
Retrieval-Augmented Generation (RAG) significantly improves Large Language Models (LLMs) but introduces massive input sequences that severely bottleneck the prefill stage. While KV-cache reuse reduces redundant computation for shared document prefixes, the reusable KV working set in RAG serving can exceed GPU memory capacity, requiring KV chunks to be retained across host DRAM and SSDs. However, naive multi-tier storage extensions suffer from severe I/O bottlenecks, suboptimal eviction, and high CPU-GPU data transfer overheads, which often negate the latency benefits of cache reuse. In this paper, we propose PCR, a P refetch-enhanced C ache R euse system for low-latency RAG serving. PCR transforms passive SSD-backed storage into an active, latency-hiding memory hierarchy through three core modules: (1) a Multi-Tier Prefix-Tree Cache Manager that unifies the organization and tracking of reusable KV chunks across the entire memory hierarchy of GPU, host DRAM, and SSD; (2) a Queue-Guided Runtime Scheduler that leverages the post-retrieval waiting queue as a look-ahead signal to proactively protect hot chunks and prefetch SSD-resident data into host DRAM before execution; and (3) a Latency-Hiding KV Transfer Pipeline that overlaps fine-grained PCIe data movement and asynchronous SSD operations with model computation. Extensive evaluations across diverse models and RAG workloads show that PCR reduces TTFT over the evaluated KV-cache reuse systems in most settings, with a 2.45 × speedup over vLLM for Llama3.1-8B on A6000 under Workload 1 at 1.0 request/s, while maintaining lower tail latency under high load.
Modern deep learning workloads are increasingly co-located on a single GPU to improve hardware utilization, yet existing MPS-based sharing mechanisms expose concurrency without providing QoS-aware control. Deadline-constrained training jobs can suffer severe slowdown when co-executed with best-effort workloads, because MPS does not reason about job urgency or deadline slack. This paper presents DPSM, a transparent runtime system for QoS-first, throughput-aware co-scheduling of concurrent deep learning jobs on a single NVIDIA GPU. DPSM does not replace the native GPU scheduler, generate deadlines, or assign deadlines to individual kernels. Instead, it intercepts CUDA kernel launches, tracks job-level progress and slowdown online, and identifies the most urgent job using a slack-based scoring model. DPSM then enforces the scheduling decision through two complementary mechanisms: launch-timing control for large-grid best-effort kernels, and TPC-mask control that limits the TPC budget of selected small-grid best-effort kernels. Under MPS, DPSM treats TPC masks as budget controls that limit the number of available TPCs. This design preserves the MPS execution model while mitigating temporal and resource-budget interference among co-running jobs. We evaluate DPSM on NVIDIA RTX A6000 GPUs using representative two-job and three-job deep learning co-location workloads. Across 1260 job-deadline instances, DPSM achieves the highest all-job deadline completion rate (DCR) of 39.2%, compared with 37.6% for TGS, 28.7% for MPS, and 18.6% for Orion. DPSM also maintains competitive normalized iteration throughput of 0.471, close to the highest-throughput baseline Orion at 0.499, while providing substantially better deadline satisfaction. We envision DPSM as a valuable tool for the community and have open-sourced it to facilitate future research at https://github.com/HIT-CeeCG/DPSM.
