As long-context language modeling becomes increasingly important, the cost of maintaining and attending to large Key/Value (KV) caches grows rapidly, becoming a major bottleneck in both training and inference. While prior works such as Multi-Query Attention (MQA) and Multi-Latent Attention (MLA) reduce memory by sharing or compressing KV features, they often trade off representation quality or incur runtime overhead. We propose Memory-Keyed Attention (MKA), a hierarchical attention mechanism that integrates multi-level KV caches (local, session, and long-term) and learns to route attention across them dynamically. We further introduce Route-Fused MKA (FastMKA), a broadcast-routed variant that fuses memory sources before attention computation for improved efficiency. Experiments on different sequence lengths show that FastMKA achieves a favorable accuracy-efficiency trade-off: comparable perplexity to MLA while achieving up to 5x faster training throughput and 1.8x lower evaluation latency. These results highlight MKA as a practical and extensible framework for efficient long-context attention.
Multi-GPU architectures are increasingly being deployed in cloud data centers, but using GPUs efficiently from high-level programming languages remains a challenge. Moreover, exploiting the full capabilities of multi-GPU systems is an arduous task due to the complex interconnection topology between available accelerators and the variety of inter-GPU communication patterns exhibited by different workloads. This work introduces a novel scheduler for multi-task GPU computations that provides transparent asynchronous execution on multi-GPU systems without requiring prior information about the program dependencies or the underlying system architecture. It integrates with the polyglot GraalVM ecosystem and is therefore available for multiple high-level languages, providing a general framework that can significantly lower the barriers to entry to multi-GPU acceleration. We validate our work on representative workloads to investigate scalability and inter-GPU communication. Experimental results show how our scheduler automatically achieves 80-90% peak performance against hand-optimized CUDA host code on Volta and Ampere multi-GPU systems.
Sparse-dense matrix multiplication (SpMM) is a key operation in scientific computing and deep learning. While previous endeavors have been made to harness the tensor cores for accelerating SpMM, challenges such as the overhead of format conversion, highly imbalanced sparsity, and irregular memory access patterns in SpMM make it inherently difficult to fully exploit the computational power of tensor cores. In this paper, we propose FastSpMM, a new algorithm using dense tensor cores to accelerate SpMM. We introduce a novel Compressed Sparse Tile format CST, which significantly reduces the overhead of format conversion. We then propose a virtual-row-based merging algorithm to generate dense tensor core blocks (TCBs) and apply a load-aware TCB mapping strategy to alleviate the load-balance problem. We also incorporate a series of runtime kernel optimizations, such as software pipelining, increased MMA instruction issues, and data prefetching techniques to maximize the computational efficiency of GPU. Experimental results on both graph neural network and SuiteSparse matrices show that FastSpMM has a geometric mean speedup of 1.88x, 1.80x, and 1.75x compared to state-of-the-art implementations such as DTC, Sputnik, and ASpT, respectively.
Mobile Edge Computing (MEC) is a key solution to overcome vehicles' limited on-board computation capabilities when dealing with data-intensive tasks. Existing studies usually assume guaranteed resource availability at base stations (BSes), or restrict a user to a single BS, overlooking the resource-competition failures due to multiple BS accesses. This paper proposes a decentralized offloading algorithm (NeurDORA) for multi-user-multi-BS (MUMB) scenarios, ensuring efficient and fair allocation of BS resources. With NeurDORA, each vehicle predicts its chance of successfully securing communication and computation resources at each candidate BS, and then competes for offloading opportunities in an iterative auction process that converges to Nash Equilibrium. Simulation results show that NeurDORA reduces offloading failures by 35-53% and achieves a 12-26% reduction in average offloading latency.
Logic Gate Networks (LGNs) exploit the similarity between neural networks and logic circuit networks and replace the neurons with logic gates. Consequently, the computation inside the neurons can be replaced by Boolean operations (16 operations for two-input logic). LGNs can be implemented by logic-based instructions in processors and significantly reduce the computation overhead during inference. However, the encoding and decoding processes at the input and output stages of LGNs face efficiency challenges when using traditional RISC-V instruction sets. This limitation arises because these processes rely on one-bit operations, which cannot fully utilize the 32-bit bandwidth of standard instructions. In this work, we proposed four custom RISC-V-based instructions to accelerate the encoding and decoding processes of LGNs. An applicationspecific RISC-V processor, called RV-LGN, has been implemented on FPGA and synthesized using Synopsys (R) Design Compiler with the CMOS 55nm process. The custom instructions can be called via in-line assembly in C code, making RV-LGN highly promising for implementation in edge devices. Benchmark tests on MIT-BIH, MNIST, and CIFAR-10 classification tasks demonstrate that RV-LGN achieves a runtime reduction of over 87% compared to a generic RISC-V RV32IM ISA processor. Additionally, power consumption during LGN inference is significantly reduced. For the MIT-BIH dataset, the energy consumption is 0.098 mu J/Beat, while MNIST and CIFAR-10 tasks require 0.18 mu J/Image and 0.51 mu J/Image, respectively. These results highlight the superior efficiency of RV-LGN compared to other processors.
