As long-context language modeling becomes increasingly important, the cost of maintaining and attending to large Key/Value (KV) caches grows rapidly, becoming a major bottleneck in both training and inference. While prior works such as Multi-Query Attention (MQA) and Multi-Latent Attention (MLA) reduce memory by sharing or compressing KV features, they often trade off representation quality or incur runtime overhead. We propose Memory-Keyed Attention (MKA), a hierarchical attention mechanism that integrates multi-level KV caches (local, session, and long-term) and learns to route attention across them dynamically. We further introduce Route-Fused MKA (FastMKA), a broadcast-routed variant that fuses memory sources before attention computation for improved efficiency. Experiments on different sequence lengths show that FastMKA achieves a favorable accuracy-efficiency trade-off: comparable perplexity to MLA while achieving up to 5x faster training throughput and 1.8x lower evaluation latency. These results highlight MKA as a practical and extensible framework for efficient long-context attention.
Multi-GPU architectures are increasingly being deployed in cloud data centers, but using GPUs efficiently from high-level programming languages remains a challenge. Moreover, exploiting the full capabilities of multi-GPU systems is an arduous task due to the complex interconnection topology between available accelerators and the variety of inter-GPU communication patterns exhibited by different workloads. This work introduces a novel scheduler for multi-task GPU computations that provides transparent asynchronous execution on multi-GPU systems without requiring prior information about the program dependencies or the underlying system architecture. It integrates with the polyglot GraalVM ecosystem and is therefore available for multiple high-level languages, providing a general framework that can significantly lower the barriers to entry to multi-GPU acceleration. We validate our work on representative workloads to investigate scalability and inter-GPU communication. Experimental results show how our scheduler automatically achieves 80-90% peak performance against hand-optimized CUDA host code on Volta and Ampere multi-GPU systems.
Sparse-dense matrix multiplication (SpMM) is a key operation in scientific computing and deep learning. While previous endeavors have been made to harness the tensor cores for accelerating SpMM, challenges such as the overhead of format conversion, highly imbalanced sparsity, and irregular memory access patterns in SpMM make it inherently difficult to fully exploit the computational power of tensor cores. In this paper, we propose FastSpMM, a new algorithm using dense tensor cores to accelerate SpMM. We introduce a novel Compressed Sparse Tile format CST, which significantly reduces the overhead of format conversion. We then propose a virtual-row-based merging algorithm to generate dense tensor core blocks (TCBs) and apply a load-aware TCB mapping strategy to alleviate the load-balance problem. We also incorporate a series of runtime kernel optimizations, such as software pipelining, increased MMA instruction issues, and data prefetching techniques to maximize the computational efficiency of GPU. Experimental results on both graph neural network and SuiteSparse matrices show that FastSpMM has a geometric mean speedup of 1.88x, 1.80x, and 1.75x compared to state-of-the-art implementations such as DTC, Sputnik, and ASpT, respectively.
Mobile Edge Computing (MEC) is a key solution to overcome vehicles' limited on-board computation capabilities when dealing with data-intensive tasks. Existing studies usually assume guaranteed resource availability at base stations (BSes), or restrict a user to a single BS, overlooking the resource-competition failures due to multiple BS accesses. This paper proposes a decentralized offloading algorithm (NeurDORA) for multi-user-multi-BS (MUMB) scenarios, ensuring efficient and fair allocation of BS resources. With NeurDORA, each vehicle predicts its chance of successfully securing communication and computation resources at each candidate BS, and then competes for offloading opportunities in an iterative auction process that converges to Nash Equilibrium. Simulation results show that NeurDORA reduces offloading failures by 35-53% and achieves a 12-26% reduction in average offloading latency.
Logic Gate Networks (LGNs) exploit the similarity between neural networks and logic circuit networks and replace the neurons with logic gates. Consequently, the computation inside the neurons can be replaced by Boolean operations (16 operations for two-input logic). LGNs can be implemented by logic-based instructions in processors and significantly reduce the computation overhead during inference. However, the encoding and decoding processes at the input and output stages of LGNs face efficiency challenges when using traditional RISC-V instruction sets. This limitation arises because these processes rely on one-bit operations, which cannot fully utilize the 32-bit bandwidth of standard instructions. In this work, we proposed four custom RISC-V-based instructions to accelerate the encoding and decoding processes of LGNs. An applicationspecific RISC-V processor, called RV-LGN, has been implemented on FPGA and synthesized using Synopsys (R) Design Compiler with the CMOS 55nm process. The custom instructions can be called via in-line assembly in C code, making RV-LGN highly promising for implementation in edge devices. Benchmark tests on MIT-BIH, MNIST, and CIFAR-10 classification tasks demonstrate that RV-LGN achieves a runtime reduction of over 87% compared to a generic RISC-V RV32IM ISA processor. Additionally, power consumption during LGN inference is significantly reduced. For the MIT-BIH dataset, the energy consumption is 0.098 mu J/Beat, while MNIST and CIFAR-10 tasks require 0.18 mu J/Image and 0.51 mu J/Image, respectively. These results highlight the superior efficiency of RV-LGN compared to other processors.
A customized RISC-V ISA with integrated digital accelerators offers a promising solution to improve energy efficiency in neural network inference. However, it often requires multiple instructions per accelerator operation, which limits computational efficiency during deep neural network inference. To overcome the instruction overhead, this design introduces a dedicated instruction set that enables scalable and fine-grained accelerator control. By incorporating the pattern-driven instruction mode, this design exploits the neural layer regularity to support efficient instruction iteration. Furthermore, this digital accelerator leverages hardware reuse for logic operations, forming a fusion-style architecture that integrates reconfigurable components. Experimental results demonstrate that the custom RISC-V ISA achieves an average runtime speedup of 8.26x and reduces the instruction count by 14.71x. This design also yields an average 8.73x reduction in cycles per instruction across MobileNetV2, ResNet50, VGG19, EfficientNet, and DenseNetBC, validating its effectiveness across representative benchmarks. Additionally, it improves average energy efficiency by 1.74x, outperforming state-of-the-art designs.
In this work, a scalable algorithm for the approximate quantum state preparation problem is proposed, facing a challenge of fundamental importance in many topic areas of quantum computing. The algorithm uses a variational quantum circuit based on the Standard Recursive Block Basis (SRBB), a hierarchical construction for the matrix algebra of the $SU(2^n)$ group, which is capable of linking the variational parameters with the topology of the Lie group. Compared to the full algebra, using only diagonal components reduces the number of CNOTs by an exponential factor, as well as the circuit depth, in full agreement with the relaxation principle, inherent to the approximation methodology, of minimizing resources while achieving high accuracy. The desired quantum state is then approximated by a scalable quantum neural network, which is designed upon the diagonal SRBB sub-algebra. This approach provides a new scheme for approximate quantum state preparation in a variational framework and a specific use case for the SRBB hierarchy. The performance of the algorithm is assessed with different loss functions, like fidelity, trace distance, and Frobenius norm, in relation to two optimizers: Adam and Nelder-Mead. The results highlight the potential of SRBB in close connection with the geometry of unitary groups, achieving high accuracy up to 4 qubits in simulation, but also its current limitations with an increasing number of qubits. Additionally, the approximate SRBB-based QSP algorithm has been tested on real quantum devices to assess its performance with a small number of qubits.
With the emerging and rapid growing of novel Web protocols such as the InterPlanetary File System(IPFS), blockchain-based storage paradigm and networks have been extensively studied and deployed to decentralize keyWeb functionalities to offer users greater control and security on storing their own data. Typically, peer-to-peer and brokerless transactions are used to allocate storage resources in most practical blockchain storage networks, and have been proved to be able to improve the effectiveness and efficiency. However, most blockchain storage networks focus more on consensus and auditing mechanisms, while less on transaction mechanisms such as auctions and bidding designs. Specifically, the existing bidding mechanisms do not take such unique market characteristic as diminishing marginal storage cost into consideration, and thus fail to guarantee stable and competitive prices in data storage markets, leading to possible storage price volatility and reduced revenue for storage users. These flaws motivate our work. In this paper, we propose a novel multi-winner auction-based transaction mechanism for blockchain storage networks, aiming at establishing secure, stable and flexible blockchain storage networks with reduced storage prices. We present detailed game-theoretic analyses for the allocation model and the bidding model proposed in our multi-winner auctions, and analyze storage miners' optimal and security bidding strategies, as well as the associated Nash equilibrium. We also compare our work with the existing multi-winner auction mechanisms, and the experimental results indicate that our model can offer enhanced security and flexibility, and can also provide lowered storage price at the same security level.
The growing adoption of RISC-V architectures in safety-critical applications has increased the demand for robust and reliable computing platforms. In operating system-based architectures, radiationinduced faults can alter system memory or register contents, leading to unintended behavior or, in the worst case, system halts. To mitigate such risks, the dependability of operating systems has to be investigated and enhanced, reducing the probability of system failures in the presence of errors. This work proposes a reliabilityoriented open-source platform based on the porting of the MIT PDOS xv6-riscv education Operating System for the VisionFive 2 board.
Ensuring reliable execution of Deep Neural Networks (DNNs) is crucial for safety-critical applications. Traditional software-based approaches fail to capture real-world fault scenarios, overlooking accelerator datapath effects. We propose a hardware-aware, software-based fault injection platform that emulates systolic array processing, enabling effective fault propagation analysis without the overhead of time-consuming methods.
Chiplet architectures are on the rise as they promise to overcome the scaling challenges of monolithic chips. A key component of such architectures is an efficient inter-chiplet interconnect (ICI). While ICI simulators are important to get reliable performance estimates, they are not fast enough to explore hundreds of thousands of design points or to be used as a cost function for optimization algorithms or machine learning models. To address this issue, we present RapidChiplet, a fast and easy to use ICI latency and throughput prediction toolchain. Compared to cycle-level simulations, we trade 0.25%-30.15% accuracy for 427x-137,682x speedup.
Temporal prefetching is a promising technology to predict the memory addresses of irregular memory accesses. It retains correlations of cache miss addresses in metadata, which can be stored either on-chip or off-chip. Recent advancements have favoured on-chip metadata storage within portions of the last level cache, making the optimization of metadata storage effectiveness crucial, because the benefits brought by temporal prefetching can be easily offset by the reduced capacity for data in the last level cache. However, current state-of-the-art temporal prefetchers employ static strategies to filter the metadata, which often results in suboptimal performance gains. In this study, we introduce AdaTP, a novel method that dynamically adjusts metadata filtering strategy based on the runtime measurement of data and metadata demands. This adaptive filtering strategy leverages criticality and reuse conditions of load instructions. Specifically, it permits only the most critical loads with repetitive access patterns to store correlations when metadata storage is limited, and allows all loads to store correlations when sufficient storage is available. Our evaluations show that AdaTP achieves a 22.1% speedup compared to baseline stride prefetch in irregular memory intensive benchmarks in SPEC CPU2006 and SPEC CPU2017, and outperforms state-of-the-art temporal prefetcher Triage and Triangel by 4.0% and 6.5% respectively.
Memory-bound Graphics Processing Unit (GPU) applications are limited by memory bandwidth, as the rapid growth in computational power has outpaced the slower increase in memory bandwidth. Consequently, approaches such as memory compression, are gaining prominence to synthetically enhance memory bandwidth and accelerate bandwidth-limited applications. Traditionally, compression techniques have been tailored towards achieving high compression ratios without considering the high bandwidth of modern GPU memory systems which makes hardware integration costly and impractical. We analyze several state-of-the-art memory compression techniques and finds that the throughput of Bit-Plane Compression (BPC) and Frequent Pattern Compression (FPC) is limited by Zero Run-Length Encoding (ZRLE), which efficiently compresses zero blocks, however, GPUs often do not benefit as even heavily compressed blocks require the transfer of a full MemoryAccess Granularity (MAG). We propose simplifying the BPC and FPC techniques by removing ZRLE and introducing a fixed-size tag section. Together with higher word-level parallelism, our simplifications increase compressor throughput by 14.0x and decompressor throughput by 13.5x without any loss in the effective compression ratio. Additionally, the area required for hardware integration is significantly reduced; for instance, the area cost of BPC is decreased by 3.6x, and power consumption by 1.8x, making hardware integration of memory compression more practical and cost-effective.
With the rising demand for machine learning-based applications, efficient and cost-effective inference serving systems have become imperative. These systems are tasked with meeting customer requirements outlined by Service Level Objectives (SLOs), encompassing model accuracy, response time, and cost considerations. Despite the adoption of proactive scheduling techniques by modern inference serving systems, dynamic factors such as fluctuating query patterns still pose challenges such as delayed response time. To address these, we propose an adaptive solution leveraging SLOaware scheduling techniques to optimize resource allocation. Our approach aims to minimize the need for additional resources per inference service. By introducing malleable inference pipelines, we enhance flexibility in resource allocation during peak loads by readjusting the resource assignment to processing pipelines to accommodate maximum possible queries dynamically. Our findings indicate that the proposed scheduler effectively utilizes system resources throughout execution while meeting most SLOs (4.2x less SLO violations). We observe an average reduction of 1.6x in the endto-end latency of query processing, compared to baseline methods. We also demonstrate the impact of dynamically reducing the resources per inference query to accommodate more inference queries in the system. Our solution accommodates 1.4x more queries on average compared to the baselines and achieves 1.6x higher system throughput in terms of queries per second on average.
The integration methodology can significantly affect the performance of dedicated accelerators. This work explores this aspect, considering Keccak, a pivotal hashing standard in Post-Quantum Cryptography (PQC), as a case of study. The paper presents three versions of KRONOS (Keccak RISC-V Optimized eNgine fOr haShing): a loosely coupled memory-mapped accelerator, a tightly coupled approach, and an Instruction Set Extension (ISE). The latter two versions leverage the Core-V eXtension InterFace (CV-X-IF) interface, with and without, respectively, an additional register file to store the Keccak state. Experimental results demonstrate that the tightly coupled integration achieves the highest throughput-to-area ratio (1.070 Mb/(s center dot LUTs)), outperforming both the loosely coupled (0.937 Mb/(s center dot LUTs)) and coprocessor-based (0.386 Mb/(s center dot LUTs)) implementations. This confirms that a tightly integrated accelerator balances resource consumption and performance most effectively.
As heterogeneous supercomputing architectures leveraging GPUs become increasingly central to high-performance computing (HPC), it is crucial for computational fluid dynamics (CFD) simulations to maintain performance portability. In this paper, we examine the performance and scalability of CFD framework SOD2D in a crosslayer manner, i.e. across application, software and hardware infrastructure related parameters. Single-GPU performance characterization across server-grade NVIDIA and AMD GPU architectures and vendor-specific compiler stacks, show the potential as well as the diverse effect of memory access optimizations, i.e. 0.69x - 3.96x deviations in acceleration speedup. Performance variability of SOD2D at scale is then further examined on the LUMI multi-GPU cluster, showcasing analogous diverse effects on throughput, demonstrating the ineffectiveness of adopting performance projections, thus underscoring the importance and necessity of cross-layer informed performance analysis and tuning for multi-GPU configurations.
Modern autonomous vehicles require efficient and predictable hardware and software to guarantee a high level of safety. Meeting response time deadlines while processing large amounts of sensor data and maintaining a reasonable power consumption is a complex task on embedded devices. Hardware acceleration on FPGA fabric proposes itself as a predictable and certifiable solution to this problem. Research on embedded heterogeneous platforms, however, has a more complex development lifecycle. This paper presents a complete hardware/software platform for autonomous driving research on a commercial off-the-shelf Industrial-Grade FPGA-based MPSoC (an AMD KR260), with a particular focus on autonomous racing use-cases. On the software side, we release the first iteration of our open-source autonomous racing stack based on ROS2. Finally, we present a case-study on a hardware-accelerated 2D LiDAR localization pipeline, which is developed and integrated on the platform. The FPGA implementation provides a 2.64x speed-up over its host counterpart and is released as well as open hardware.
This paper addresses the problem of deployment of LLMs on RISCV-based CPU systems by optimizing LLM inference on the Sophon SG2042. We evaluate the inference performance of two state-of-theart LLMs optimised for reasoning: DeepSeek R1 Distill Llama 8B and DeepSeek R1 Distill QWEN 14B. Thanks to our optimizations on top of the llama.cpp inference library, we achieve token generation speeds of 4.32/2.29 tokens per second and prompt processing speeds of 6.54/3.68 tokens per second, with a significant speedup of up to 2.9x/3.0x compared to a direct porting of the same library.
GPUs play an important role in High-Performance Computing. The choice of GPU programming models plays a crucial role in achieving portability and performance. High-level programming models, such as SYCL and OpenMP offloading, have emerged, offering unified abstractions that enable developers to target multiple architectures with a single, maintainable codebase. However, achieving consistent performance across different models remains a significant challenge due to variations in abstraction levels, compiler optimizations, and runtime behavior. We present a profiling-based methodology for systematically comparing GPU programming models on NVIDIA and AMD GPUs. We apply our methodology to over 150 benchmarks of HeCBench, demonstrating its effectiveness in identifying performance issues in OpenMP, SYCL, HIP and CUDA implementations for AMD and NVIDIA GPUs.
