The Nvidia Grace Hopper Superchip has seen a wide adoption across both HPC sites and AI data centers. This hybrid architecture places the Grace CPU and Hopper GPU on the same board, with a shared and adjustable power budget. We present an end-to-end analysis of its energy efficiency, from a verification of power sensor accuracy using microbenchmarks, an investigation of power knob effects to an energy-efficiency and TCO investigation for selected Large Language Model (LLM) inference use cases. Our study shows that the internal power sensors provide precise readings when compared to an external power meter, unlike contemporary x86 processors. Some power budget configurations are not enforced by the system, the power draw can exceed the configured limit. This behavior is not documented by Nvidia. We analyzed the energy efficiency of LLM inference for prefill and decoding-dominated workloads, with the latter having emerged as the driving factor. Increasing the module power limit while maintaining a constant GPU power did not affect the system's energy efficiency. According to our TCO analysis, no cost savings can be expected from reduced power limits for output lengths of up to 2048 tokens for our LLM setup.

Application performance of modern day processors is often limited by the memory subsystem rather than actual compute capabilities. Therefore, data throughput specifications play a key role in modeling application performance and determining possible bottlenecks. However, while peak instruction throughputs and bandwidths for local caches are often documented, the achievable throughput can also depend on the relation between memory access and compute instructions. In this paper, we present an Arm version of the well established x86-membench throughput benchmark, which we have adapted to support all current SIMD extensions of the Armv8 instruction set architecture. We describe aspects of the Armv8 ISA that need to be considered in the portable design of this benchmark. We use the benchmark to analyze the memory subsystem at a fine spatial granularity and to unveil microarchitectural details of three processors: Fujitsu A64FX, Ampere Altra and Cavium ThunderX2. Based on the resulting performance information, we show that instruction fetch and decoder widths become a potential bottleneck for cache-bandwidth-sensitive workloads due to the load-store concept of the Arm ISA.

The continuous evolution of processors requires vendors to translate ever-growing transistor budgets into performance improvements, e.g., by including more functional units, memory controllers, input/output (I/O) interfaces, graphics processing units (GPUs), and caches. This trend also increases complexity, which cannot be fully hidden from the operating system (OS) or application domains. Issues likewhere to place threads if cores have different frequency ranges or architectures, orwhere to perform a task that might be hardware-accelerated cannot be decided on a hardware level. Moreover, performance improvements need to be achieved within a limited power envelope with energy efficiency as a first order design goal. Introduced power saving techniques, however, can contradict OS and applications performance assumptions. Several processor vendors offer heterogeneous processor architectures, such as ARM's big.LITTLE or Apple M1, combining high-performance and power-efficient cores. Intel's first such architecture, Alder Lake, integrates different core architectures and various accelerating components. This work presents an architecture overview of Alder Lake and an in-depth analysis of its power efficiency properties and techniques. For example, this includes frequency scaling of different components, idle states and their latencies, integrated energy measurement capabilities, and recently introduced processor feedback interfaces and OS integration.

Modern processors, in particular within the server segment, integrate more cores with each generation. This increases their complexity in general, and that of the memory hierarchy in particular. Software executed on such processors can suffer from performance degradation when data is distributed disadvantageously over the available resources. To optimize data placement and access patterns, an in-depth analysis of the processor design and its implications for performance is necessary. This paper describes and experimentally evaluates the memory hierarchy of AMD EPYC Rome and Intel Xeon Cascade Lake SP server processors in detail. Their distinct microarchitectures cause different performance patterns for memory latencies, in particular for remote cache accesses. Our findings illustrate the complex NUMA properties and how data placement and cache coherence states impact access latencies to local and remote locations. This paper also compares theoretical and effective bandwidths for accessing data at the different memory levels and main memory bandwidth saturation at reduced core counts. The presented insight is a foundation for modeling performance of the given microarchitectures, which enables practical performance engineering of complex applications. Moreover, security research on side-channel attacks can also leverage the presented findings.

In High Performance Computing, systems are evaluated based on their computational throughput. However, performance in contemporary server processors is primarily limited by power and thermal constraints. Ensuring operation within a given power envelope requires a wide range of sophisticated control mechanisms. While some of these are handled transparently by hardware control loops, others are controlled by the operating system. A lack of publicly disclosed implementation details further complicates this topic. However, understanding these mechanisms is a prerequisite for any effort to exploit the full computing capability and to minimize the energy consumption of today's server systems. This paper highlights the various energy efficiency aspects of the AMD Zen 2 microarchitecture to facilitate system understanding and optimization. Key findings include qualitative and quantitative descriptions regarding core frequency transition delays, workload-based frequency limitations, effects of I/O die P-states on memory performance as well as discussion on the built-in power monitoring capabilities and its limitations. Moreover, we present specifics and caveats of idle states, wakeup times as well as the impact of idling and inactive hardware threads and cores on the performance of active resources such as other cores.

OpenMP is the de-facto standard for parallel programming on shared-memory systems. The choice of scheduling methods in OpenMP work sharing parallel loops is a critical aspect for performance, especially for computationally-intensive and irregular parallel loops. In this work, we explore loop scheduling enhancements in OpenMP. Three loop scheduling choices are covered today in the OpenMP standard: static, guided, and dynamic. These are no longer sufficient to address the load imbalance that adversely affects the execution of computationally-intensive and irregular parallel loops. In this work, we present a generic methodology for exploring loop scheduling enhancements in OpenMP that allows the implementation, testing, and usage of additional (more advanced) loop scheduling choices in OpenMP runtime systems. We showcase the methodology by enhancing the LLVM OpenMP runtime with an additional dynamic loop self-scheduling (DLS) technique, known to offer superior load balancing over the existing OpenMP scheduling choices for computationally-intensive and irregular parallel loops. We analyze the overhead of the (existing and newly added) OpenMP loop scheduling methods and show that the proposed methodology incurs no additional overhead. We also study the performance of four benchmarks using the enhanced LLVM OpenMP runtime. The results show that, for the four benchmarks considered, no single loop scheduling strategy outperforms the others. The newly implemented DLS technique provides an additional opportunity for improved execution time with the LLVM OpenMP runtime, which was not possible before this study. Our newly implemented scheduling strategy is competitive with the best previous scheduling choices. This methodology for exploring loop scheduling enhancements in OpenMP lays the foundation for further loop scheduling additions and explorations in OpenMP.