Exploiting Intra-Core Heterogeneity for High-Performance Dense Linear Algebra on Huawei Ascend 910 NPUs

Dense linear algebra kernels—General Matrix Multiplication (GEMM), Triangular Solve with Multiple Right-Hand Sides (TRSM), and LU decomposition with partial pivoting (GETRF)—serve as foundational computational building blocks for both High-Performance Computing (HPC) and modern Artificial Intelligence (AI). While highly optimized implementations exist for CPUs and GPUs, emerging Neural Processing Units (NPUs) present unique architectural challenges that demand novel algorithmic approaches to achieve peak performance. The Huawei Ascend 910 NPU exemplifies these challenges: each of its 20 AI cores integrates heterogeneous functional units (Cube, Vector, Scalar) capable of concurrent execution. This architecture enables theoretical performance beyond linear core scaling, but only if algorithms can coordinate fine-grained parallelism across these units while addressing complex memory hierarchies. This paper makes following contributions: (1) New Parallelism Discovery Scheduling We identify and formalize previously unexplored parallelism in TRSM and GETRF kernels, then design a provably efficient scheduling strategy to map these opportunities onto the Ascend 910’s heterogeneous units (Cube/Vector/Scalar). This breaks performance limits imposed by core-count scaling. (2) Unified Data Partitioning Strategy We extend traditional communication optimality analyses to the Ascend 910’s intricate cache hierarchy, which features a mix of shared, distributed, inclusive, and exclusive caches. Our unified partitioning strategy achieves asymptotic communication optimality under these complex constraints. Autonomous On-Device Scheduling We develop an autonomous scheduling mechanism that offloads tiling parameter computation entirely to the device, minimizing host-device communication overhead and enabling efficient on-device kernel fusion. Notably, our work establishes the first high-performance algorithmic baseline for TRSM and GETRF on the Ascend NPU platform, addressing a critical gap in the existing ecosystem. To validate our approach, we implement the kernels integrated with these techniques on the Ascend 910 NPU, achieving consistent speedups of 2-3 × over the highly optimized NVIDIA A100 cuBLAS/cuSolver library and MAGMA. Remarkably, this performance is achieved despite the Ascend 910’s comparable theoretical FLOPS and lower memory bandwidth relative to the A100. These results demonstrate that algorithmic innovation can unlock the full potential of NPUs, enabling them to outperform state-of-the-art GPUs in dense linear algebra tasks, even without superior raw hardware specifications. This work significantly broadens the applicability of NPUs for HPC and AI workloads, challenging the notion that performance is inherently tied to core count or peak hardware capabilities.