Optimizing Full-Spectrum Matrix Multiplications on ARMv8 Multi-Core CPUs

General Matrix Multiplication (GEMM) is a key subroutine in high-performance computing. While the mainstream Basic Linear Algebra Subprograms (BLAS) libraries can deliver good performance on large and regular-shaped GEMMs, they are inadequate for optimizing small and irregular-shaped GEMMs, which are commonly seen in emerging HPC applications. Recent research has focused on improving GEMM performance on GPUs, but there is still significant room for improvement on emerging HPC hardware based on multi-core CPUs. We present LibShalom2, an open-source library to optimize full-spectrum GEMMs, taking small, irregular-shaped, and large-scale regular-shaped matrices. LibShalom2 explicitly targets the ARMv8 architecture, which is becoming common in HPC systems. LibShalom2 is designed to minimize the expensive memory accessing overhead for data packing and processing small matrices. It uses analytic methods to determine GEMM kernel optimization parameters, enhancing the computation and parallelization efficiency of the GEMM kernels. We evaluate LibShalom2 by applying it to three ARMv8 multi-core architectures and comparing it against five mainstream linear algebra libraries. Experimental results show that LibShalom2 consistently outperforms existing solutions across full-spectrum GEMM workloads and hardware architectures. We also show that LibShalom2 delivers an average speedup of 2.2x for real-life neural network workloads.