The growing demand for intensive artificial intelligence applications, such as deep neural networks and Large Language Models (LLMs), has driven the creation of architecture extensions and accelerators for matrix multiplication, a fundamental operation in machine learning. Several modern Instruction Set Architectures (ISAs) have introduced matrix extensions, each employing distinct approaches to data storage and execution. This work examines the scalability of computational intensity (CI) across different matrix data layout models and ISA-level processing, focusing on proposals centered on the RISC-V architecture.We create an analytical model that associates CI with the submatrix format and the available storage budget. Our results reveal that the outer product model achieves CI similar to that of architectures that perform full-block multiplication, but with much lower storage costs.We also demonstrate an upper bound for the computational intensity of outer product approaches under storage constraints. Finally, we suggest a technique that reuses existing RISC-V vector registers for matrix computation, organizing the data into two-dimensional grids to optimize CI. The proposed approach achieves up to 99.6% of the upper bound with 256-bit registers and 99.2% with 2048-bit registers. We conclude that it is possible to achieve high efficiency without increasing the architectural area, allowing scalability and compatibility with already standardized vector extensions.

The IBM Power11 processor extends the system architecture introduced in Power10 with a focus on memory-centric design and system-level scalability. While maintaining a largely compatible core microarchitecture and programming model, Power11 delivers substantial improvements in memory bandwidth, capacity, and system integration. Enhancements to the Open Memory Interface (OMI) enable up to 3× higher memory bandwidth and 2× increased capacity per socket, while improved coherence and interconnect bandwidth support more efficient scaling across larger configurations. These advances reflect a broader shift in computer architecture, in which performance is increasingly limited by data movement rather than raw compute capability. By combining high-performance cores with an enhanced memory subsystem and system architecture, Power11 supports a new class of enterprise and AI-oriented systems optimized for the emerging memory-centric era of computing.

The standardization of an interface for dense linear algebra operations in the BLAS standard has enabled interoperability between different linear algebra libraries, thereby boosting the success of scientific computing, in particular in scientific HPC. Despite numerous efforts in the past, the community has not yet agreed on a standardization for sparse linear algebra operations due to numerous reasons. One is the fact that sparse linear algebra objects allow for many different storage formats, and different hardware may favor different storage formats. This makes the definition of a FORTRAN-style all-circumventing interface extremely challenging. Another reason is that opposed to dense linear algebra functionality, in sparse linear algebra, the size of the sparse data structure for the operation result is not always known prior to the information. Furthermore, as opposed to the standardization effort for dense linear algebra, we are late in the technology readiness cycle, and many production-ready software libraries using sparse linear algebra routines have implemented and committed to their own sparse BLAS interface. At the same time, there exists a demand for standardization that would improve interoperability, and sustainability, and allow for easier integration of building blocks. In an inclusive, cross-institutional effort involving numerous academic institutions, US National Labs, and industry, we spent two years designing a hardware-portable interface for basic sparse linear algebra functionality that serves the user needs and is compatible with the different interfaces currently used by different vendors. In this paper, we present a C++ API for sparse linear algebra functionality, discuss the design choices, and detail how software developers preserve a lot of freedom in terms of how to implement functionality behind this API.

The GraphBLAS C API is mature with an updated specification (version 2.1) and a compliant implementation (SuiteSparse GraphBLAS). We are now focused on GraphBLAS 3.0; the next major GraphBLAS revision. Potential changes include: (1) a separate math spec to make new language bindings easier to write, (2) better support for user-defined types, rank promotion, and enhanced non-blocking execution, (3) expanded scope of GraphBLAS to address a wider range of applications, and (4) support for complex heterogeneous and distributed systems.

Convolution is one of the most computationally intensive operations that must be performed for machine learning model inference. A traditional approach to computing convolutions is known as the Im2Col + BLAS method. This article proposes SConv: a direct-convolution algorithm based on an MLIR/LLVM code-generation toolchain that can be integrated into machine-learning compilers. This algorithm introduces: (a) Convolution Slicing Analysis (CSA)—a convolution-specific 3D cache-blocking analysis pass that focuses on tile reuse over the cache hierarchy; (b) Convolution Slicing Optimization—a code-generation pass that uses CSA to generate a tiled direct-convolution macro-kernel; and (c) Vector-based Packing—an architecture-specific optimized input-tensor packing solution based on vector-register shift instructions for convolutions with unitary stride. Experiments conducted on 393 convolutions from full ONNX-MLIR machine learning models indicate that the elimination of the Im2Col transformation and the use of fast packing routines result in a total packing time reduction, on full model inference, of 2.3×–4.0× on Intel x86 and 3.3×–5.9× on IBM POWER10. The speed-up over an Im2Col + BLAS method based on current BLAS implementations for end-to-end machine-learning model inference is in the range of 11%–27% for Intel x86 and 11%–34% for IBM POWER10 architectures. The total convolution speedup for model inference is 13%–28% on Intel x86 and 23%–39% on IBM POWER10. SConv also outperforms BLAS GEMM, when computing pointwise convolutions in more than 82% of the 219 tested instances.

In this paper, we present a comprehensive security architecture, Flexible Secure Execution Environment (FlexSEE), for confidential computing in modern cloud environments. FlexSEE does not require the trust of system software on the compute server and guarantees that the user data is visible only in non-privileged mode to a designated program trusted by the data owner on a designated hardware, thus protecting the data from an untrusted hardware, hypervisor, OS, or other users' applications, on the compute server. We describe the Hardware Trust Zone (HTZ), the enclave confining the clear-text data, the cryptography hardware used in the HTZ, the protocols used to move data between HTZ and the memory hierarchy beyond the HTZ, and the memory extensions for the L1-cache in the HTZ. Our simulation result show that the overhead of encrypting and decrypting data in an FlexSEE-enabled processor is modest, only 6% on average across a collection of commercial workloads, when the data encryption engine is placed between the L1 and L2 cache.

The resurgence of machine learning has increased the demand for high-performance basic linear algebra subroutines (BLAS), which have long depended on libraries to achieve peak performance on commodity hardware. High-performance BLAS implementations rely on a layered approach that consists of tiling and packing layers, for data (re)organization, and micro kernels that perform the actual computations. The creation of high-performance micro kernels requires significant development effort to write tailored assembly code for each architecture. This hand optimization task is complicated by the recent introduction of matrix engines by IBM's POWER10 MMA, Intel AMX, and Arm ME to deliver high-performance matrix operations. This paper presents a compiler-only alternative to the use of high-performance libraries by incorporating, to the best of our knowledge and for the first time, the automatic generation of the layered approach into LLVM, a production compiler. Modular design of the algorithm, such as the use of LLVM's matrix-multiply intrinsic for a clear interface between the tiling and packing layers and the micro kernel, makes it easy to retarget the code generation to multiple accelerators. The use of intrinsics enables a comprehensive performance study. In processors without hardware matrix engines, the tiling and packing delivers performance up to 22x (Intel), for small matrices, and more than 6x (POWER9), for large matrices, faster than PLuTo, a widely used polyhedral optimizer. The performance also approaches high-performance libraries and is only 34% slower than OpenBLAS and on-par with Eigen for large matrices. With MMA in POWER10 this solution is, for large matrices, over 2.6x faster than the vector-extension solution, matches Eigen performance, and achieves up to 96% of BLAS peak performance.

Interoperability between libraries is often hindered by incompatible data formats, which can necessitate creating new copies of data when transferring data back and forth between different libraries. This additional data movement incurs additional runtime costs, particularly for sparse applications, where the costs of data movement often dwarf compute costs. In this paper, we investigate interoperability in the context of the C++ GraphBLAS Specification, where C++ concepts allow GraphBLAS algorithms to accept any matrix type as long as it follows the matrix interface defined in the GraphBLAS matrix concept. We first develop non-owning, lazily evaluated adapted views for a number of external data structures, including two categories of graphs defined in the Northwest Graph Library (NWGraph) and traditional pointer-based CSR data structures. These adapted views fulfill the C++ GraphBLAS matrix concept, allowing them to be used inside GraphBLAS algorithms. We then evaluate the performance of these adapted views across two kernels, matrix reduction and sparse times dense matrix multiplication (SpMM), where the performance achieved using a single generic implementation with these views largely matches the performance achieved operating directly on the original data structures, with a slight performance loss in one case. We then propose a mechanism for automatically discovering the availability of these views, allowing algorithms to directly accept external data structures. We also discuss potential extensions to the C++ GraphBLAS specification that might eliminate the small performance dip observed for one of the views.

Power ISA™ Version 3.1 has introduced a new family of matrix math assist instructions, collectively known as the Matrix-Multiply Assist (MMA) facility. The instructions in this facility implement numerical linear algebra operations on small matrices and are meant to accelerate computation-intensive kernels. We advocate the use of compiler built-ins as the preferred way of leveraging these instructions. MMA built-ins are currently available in the GNU Compiler Collection and the LLVM-based IBM Open XL compilers. The built-ins are compatible across both compiler suites. We show that programming with these built-ins leads to efficient code that fully exploits the new facility.

Recent processors have been augmented with matrix-multiply units that operate on small matrices, creating a functional unit-rich environment. These units have been successfully employed on dense matrix operations such as those found in the Basic Linear Algebra Subprograms (BLAS). In this work, we exploit these new matrix-multiply facilities to speed up Sparse Matrix Dense Matrix Multiplications (SpMM) for highly sparse matrices.

Return-oriented programming (ROP) is a technique for hijacking the control-flow of a program and forcing it to perform computations that were never originally intended. ROP is achieved by modifying the values of return addresses saved to memory, causing a failure of control-flow integrity. In the POWER10 processor, we have adopted a cryptographic mechanism for ROP protection. Return addresses are cryptographically hashed when control-flow enters a function and the hash is saved in memory. The hash is recomputed and compared to the saved value just before a return from the function. Any mismatch is flagged as a violation and generates an exception to the supervisor. POWER10 was augmented with instructions to generate and verify those hashes. We minimize performance impact on running programs by implementing the cryptographic hash with dedicated functional units.

This chapter describes the mathematical properties that have been developed since to support the GraphBLAS. It describes the key mathematical concepts of the GraphBLAS and presents preliminary results that show the overhead of the GraphBLAS is minimal. Matrix multiplication is the most important matrix operation and can be used to implement a wide range of graph algorithms. One of the most common uses of matrix multiplication is to construct an adjacency matrix from an incidence matrix representation of a graph. The GraphBLAS performance of sparse matrix sparse vector multiplication is similar toGunrock BFS performance. The similarity in performance indicates that the GraphBLAS is not introducing a high overhead. The GraphBLAS allows these matrix properties to be readily applied to graphs in a low-overhead manner.

Iteration over opaque, generic data structures is an important feature of many C++ libraries. Aggressive compiler optimization and inlining enables generic C++ iterators to iterate over complex data structures with performance comparable to that of hand-tuned code with C-language (raw) pointers. In this paper, we describe the sparse matrix iterators in the current draft of the C++ GraphBLAS API, their support for a variety of backend data formats, and implementation strategies we have considered. We compare performance of these iterators to that of hand-tuned iteration with raw pointers, showing that our iterators introduce minimal overhead. We consider extensions to our iterator design for interoperability with the draft C++ Graph Library proposal and to support different semantics for iterating over sparse matrices (by row, by column, by specific diagonals, etc.).

The matrix-multiply assist (MMA) facility is the latest addition to IBM’s power instruction set architecture and first shipped in the recently introduced POWER10 processor. MMA is designed to accelerate matrix–matrix operations, such as matrix multiplication and convolution, using instructions that compute the outer product of vector-register operands. Outer product computations have been used for decades in linear algebra libraries to deliver high-performance implementations of matrix operations. Such libraries use conventional single-instruction–multiple-data (SIMD) instructions to emulate outer product operations. MMA in POWER10 is the first hardware with direct support for outer product operations released in the market. MMA operates with the widest diversity of data types compared to any accelerator design currently announced. Unleashing the high-performance enabled by MMA requires careful code generation. Two key considerations for optimal MMA code performance are 1) the choice of accumulation layout when maximizing the using the accumulators and 2) the selection of matrix access order. This article shows that over 92% of peak performance in POWER10 with MMA can be achieved when the code generation makes the right choices.

Matrix engines, also known as matrix-multiplication accel-erators, capable of computing on 2D matrices of various data types are traditionally found only on GPUs. However, they are increasingly being introduced into CPU architectures to support AI/ML computations. Unlike traditional SIMD functional units, these accelerators require both the input and output data to be packed into a specific 2D-data layout that is often dependent on the input and output data types. Due to the large variety of supported data types and architectures, a common abstraction is required to unify these seemingly disparate accelerators and more efficiently produce high-performance code. In this paper, we show that the hardware characteristics of a vast array of different matrix engines can be unified using a single analytical model that casts matrix engines as an accumulation of multiple outer-products (also known as rank-k updates). This allows us to easily and quickly develop high-performance kernels using matrix engines for different architectures. We demonstrate our matrix engine model and its portability by applying it to two distinct architectures. Using our model, we show that high-performance computational kernels and packing routines required for high-performance dense linear algebra libraries can be easily designed. Furthermore, we show that the performance attained by our implementations is around 90–99 % (80–95 % on large problems) of the theoretical peak throughput of the matrix engines.

This article introduces YaConv , a new algorithm to compute convolution using GEMM microkernels from a Basic Linear Algebra Subprograms library that is efficient for multiple CPU architectures. Previous approaches either create a copy of each image element for each filter element or reload these elements into cache for each GEMM call, leading to redundant instances of the image elements in cache. Instead, YaConv loads each image element once into the cache and maximizes the reuse of these elements. The output image is computed by scattering results of the GEMM microkernel calls to the correct locations in the output image. The main advantage of this new algorithm—which leads to better performance in comparison to the existing im2col approach on several architectures—is a more efficient use of the memory hierarchy. The experimental evaluation on convolutional layers from PyTorch, along with a parameterized study, indicates an average 24% speedup over im2col convolution. Increased performance comes as a result of 3× reduction in L3 cache accesses and 2× fewer branch instructions.

Recent processors have been augmented with matrix-multiply units that operate on small matrices, creating a functional unit-rich environment. These units have been successfully employed on dense matrix operations such as those found in the Basic Linear Algebra Subprograms (BLAS). In this work, we exploit these new matrix-multiply facilities to speed up Sparse Matrix Dense Matrix Multiplications (SpMM) for highly sparse matrices. SpMM is hard to optimize. The sparsity patterns lead to a highly irregular memory access behavior. Additionally, each sparse matrix has unique characteristics, making it hard to find a single SpMM strategy that works well for all sparse matrices. The addition of matrix-multiply units makes this even more challenging. In this paper, we address these challenges. First, we design Dense Dynamic Blocks (DDB), a method to utilize the new matrix units. DDB has two specialized versions: DDB-MM and DDB-HYB. DDB-MM is a strategy that only utilizes the matrix-multiply facilities. DDB-HYB is a hybrid approach that maximizes the floating-point throughput by utilizing both vector and matrix units. Furthermore, we design a prediction mechanism for identifying the best SpMM strategy for a given sparse matrix and dense matrix pair: SpMM-OPT. SpMM-OPT selects among vector unit oriented, matrix unit oriented, and hybrid strategies for the highest floating-point throughput while taking cache optimizations into account. We experiment with 440 matrices from the well-known SuiteSparse matrix collection on a POWER10 system with vector and matrix units. We show that DDB-MM and DDB-HYB can achieve a floating-point throughput of up to 1.1 and 2.5 TFLOPs/s on a POWER10 single-chip module for double- and single-precision SpMM, respectively. Our analysis also shows that SpMM-OPT effectively chooses the best SpMM strategy and can achieve an average speedup of up to 2X compared to an optimized CSR baseline.

We present the novel micro-architectural features, supported by an innovative and novel pre-silicon methodology in the design of POWER10. The resulting projected energy efficiency boost over POWER9 is 2.6x at core level (for SPECint) and up to 3x at socket level. In addition, a new feature supporting inline AI acceleration was added to the POWER ISA and incorporated into the POWER10 processor core design. The resulting boost in SIMD/AI socket performance is projected to be up to 10x for FP32 and 21x for INT8 models of ResNet-50 and BERT-Large. In this paper, we describe the novel methodology deployed and used not only to obtain these efficiency boosts for traditional workloads, but also to infuse AI/ML/HPC capability directly into the POWER10 core.

The GraphBLAS is a set of basic building blocks for constructing graph algorithms in terms of linear algebra. They are first and foremost defined mathematically with the goal that language bindings will be produced for a wide range of programming languages. We started with the C programming language and over the last four years have produced multiple versions of the GraphBLAS C API specification. In this paper, we describe our next version of the C GraphBLAS specification. It introduces a number of major changes including support for multithreading, import/export functionality, and functions that use the indices of matrix/vector elements. Since some of these changes introduce small backwards compatibility issues, this is a major release we call GraphBLAS 2.0.