
The rapid development of domain-specific deep neural network (DNN) accelerators, particularly spatial architectures such as systolic arrays, has become essential for improving power–performance efficiency in edge computing. However, despite advances in hardware design tools, deploying deep learning models on custom spatial accelerators remains a major bottleneck. Existing machine learning (ML) compilers are primarily designed for GPUs and CPUs, making the integration of new accelerators complex and labor-intensive due to diverse programming interfaces and architecture-specific execution models. This lack of a unified deployment interface limits the scalability and generality of current compiler toolchains. In this work, we address this challenge by shifting the abstraction from accelerator-specific interfaces to kernel functions, the fundamental building blocks for executing DNN workloads. We present AutoUMA, an automated compilation framework that externalizes kernel generation and introduces an AutoScheduler to automatically derive optimized kernels from accelerator mappings. The scheduler performs structured transformations, including multi-level loop tiling, memory transfer staging, and instruction emission, and incorporates a guard mechanism to eliminate redundant memory operations. In addition, we introduce AutoUMA ’s compiler retargeting pipeline that leverages kernel functions as a uniform, coarse-grained deployment interface and abstracts low-level modifications through high-level accelerator descriptions. We evaluate our framework on the Gemmini accelerator and demonstrate that it significantly reduces integration effort while generating efficient kernel libraries. Our approach achieves speedups of up to 4.4× over manually developed static libraries in low-level C and delivers performance competitive with expert-optimized kernel implementations using high-level scheduling languages.
Cloud LLM services increasingly adopt Mixture-of-Experts (MoE) models for cost-effective scaling, yet under heavy multi-task workloads their serving performance is often bottlenecked by severe tail latency. The execution semantics of MoE inference are poorly aligned with cloud-native serving abstractions, causing spatial and temporal mismatch in expert placement and demand, and consequently frequent expert loading. Expert demand, however, exhibits structured locality where correlated requests and decoding steps often induce overlapping expert working sets, creating opportunities for activated-expert reuse. Exploiting such reuse is non-trivial because request routing, expert reuse, and resource allocation are tightly coupled over time, forming a sequential decision problem whose coupled dimensions existing systems address only in isolation. This paper presents Maestro, a learning-based framework for activated-expert reuse in cloud-native MoE inference. Maestro models tasks, experts, instances, and execution stages as a heterogeneous graph with relational message passing to capture structured system state, and combines a lightweight Neural Optimistic Value Policy (NOVP) with Pareto-guided action ranking (AdaFilter) for efficient online decision making. It further introduces a prefill-driven expert prefetch mechanism that overlaps loading with computation, where the per-window transfer budget is jointly determined by the RL controller to balance reuse coverage against memory pressure. Evaluated on Mixtral-8 × 7B, Phi-3.5-MoE, and Qwen1.5-MoE, Maestro reduces P50 and P99 latency by up to 34.02% and 28.94%, respectively, and achieves up to 1.51 × the throughput of vLLM. Against the SOTA learning-based methods, it reduces P99 and TTFT by 8.58%–17.47% and 6.22%–17.43%, respectively, while improving throughput by 12.09%–22.70%.
The rapid advancement of deep learning models has led to their widespread application across various domains; however, this has resulted in a significant increase in both hardware and software costs, necessitating optimizations for resource efficiency. To address this issue, performance prediction techniques evaluate the performance impact of corresponding optimizations on deep learning systems to maximize performance gains. This paper presents a comprehensive survey of performance prediction techniques for deep learning systems, categorizing them into allocation, cluster, and accelerator levels based on the system level at which their corresponding optimizations are situated, ranging from high to low. Allocation-level techniques primarily construct the relationships between resource configurations and their performance, cluster-level techniques model the optimizations within deep learning frameworks at given configurations, while accelerator-level techniques assess the efficiency of specific hardware. We point out that prediction techniques at different levels exhibit a hierarchical relationship rather than operating in isolation. High-level prediction techniques can incorporate the results from low-level techniques into their performance models to accommodate more low-level optimizations and improve their prediction accuracy. However, this will slow down the construction and execution speed of performance models, requiring trade-offs for specific use cases. Finally, we discuss future challenges and research directions in performance prediction techniques for deep learning systems.
Ray tracing (RT) is increasingly used in interactive graphics and edge rendering, but efficient deployment remains difficult because its two major phases, BVH construction and BVH traversal, have fundamentally different computation patterns, precision requirements, and architectural affinities. Existing designs largely optimize these phases in isolation and therefore do not provide an efficient full-pipeline solution. We present an ISA-centric RISC-V heterogeneous architecture for energy-efficient RT acceleration through phase-aware hardware-software co-design. The design has three components: a full-BVH compression framework that combines exact lossless compression for leaf-triangle blocks with conservative multi-level quantization for AABBs and rays; custom RISC-V ISA extensions for both vectorized BVH-construction kernels and correctness-preserving low-precision traversal; and a phase-aware heterogeneous organization that couples a RVV-enhanced core for construction with a precision-aware traversal coprocessor. For BVH construction, our design achieves speedups of 2.38 × and 1.97 × over XT910 with RVV for SAH-2 and LBVH-2. Compared with Jetson AGX Orin across different BVH widths (BVH2/4/6/8), our design achieves up to 1.59× and 1.77× speedups for SAH and LBVH, while consuming only 20.53%–32.46% of its energy. For BVH traversal, it achieves 1.42 × speedup over AQB8, while consuming only 31.79% of AQB8’s energy. Across different BVH widths, our design achieves up to 2.50× and 2.64× speedups for SAH and LBVH over Orin, while consuming only 27.19%–51.02% of its energy. These results demonstrate that phase-aware ISA-centric design enables efficient, programmable full-pipeline ray tracing.
Graph Convolutional Networks (GCNs) have become a key model for representation learning on graph structured data. The introduction of residual connections further breaks the depth limitation of traditional GCNs, giving rise to Deep Graph Convolutional Networks (DeepGCNs). We observe that in DeepGCNs, as network depth increases, residual feature values progressively accumulate and grow in magnitude, which significantly reduces the model’s sensitivity to variations in intermediate layer features. Consequently, even when strong feature interventions such as aggregation skipping or mean reuse are applied during inference, the residual features of the optimized model still preserve high similarity to those of the original baseline. Based on this observation, this paper proposes a redundancy aware inference algorithm DGCN for DeepGCNs and further designs a dedicated hardware accelerator DeGCN that works in coordination with it. Unlike existing GCNs accelerators that mainly focus on intra layer sparsity and storage optimization, DeGCN explicitly identifies aggregation skipping and mean reuse behaviors and dynamically organizes differentiated data paths and scheduling strategies, thereby avoiding redundant computations in intermediate layers. For the execution characteristics of the initial layers, where features are sparse and memory access is the performance bottleneck, DeGCN incorporates the CBSR adaptive sparse feature storage format and a multi encoding support mechanism to accommodate input features with different sparsity levels, thereby reducing feature access and decoding overhead. In addition, through cross layer scheduling and execution restructuring mechanisms, DeGCN enables coordinated advancement of multi level computations at the architectural level, thereby maximizing execution overlap and improving overall throughput. Compared with HS-GCN, the strongest evaluated prior accelerator, DeGCN achieves a 4.16 × speedup and 5.12 × higher energy efficiency.