The Nvidia Grace Hopper Superchip has seen a wide adoption across both HPC sites and AI data centers. This hybrid architecture places the Grace CPU and Hopper GPU on the same board, with a shared and adjustable power budget. We present an end-to-end analysis of its energy efficiency, from a verification of power sensor accuracy using microbenchmarks, an investigation of power knob effects to an energy-efficiency and TCO investigation for selected Large Language Model (LLM) inference use cases. Our study shows that the internal power sensors provide precise readings when compared to an external power meter, unlike contemporary x86 processors. Some power budget configurations are not enforced by the system, the power draw can exceed the configured limit. This behavior is not documented by Nvidia. We analyzed the energy efficiency of LLM inference for prefill and decoding-dominated workloads, with the latter having emerged as the driving factor. Increasing the module power limit while maintaining a constant GPU power did not affect the system's energy efficiency. According to our TCO analysis, no cost savings can be expected from reduced power limits for output lengths of up to 2048 tokens for our LLM setup.

The continuous evolution of processors requires vendors to translate ever-growing transistor budgets into performance improvements, e.g., by including more functional units, memory controllers, input/output (I/O) interfaces, graphics processing units (GPUs), and caches. This trend also increases complexity, which cannot be fully hidden from the operating system (OS) or application domains. Issues likewhere to place threads if cores have different frequency ranges or architectures, orwhere to perform a task that might be hardware-accelerated cannot be decided on a hardware level. Moreover, performance improvements need to be achieved within a limited power envelope with energy efficiency as a first order design goal. Introduced power saving techniques, however, can contradict OS and applications performance assumptions. Several processor vendors offer heterogeneous processor architectures, such as ARM's big.LITTLE or Apple M1, combining high-performance and power-efficient cores. Intel's first such architecture, Alder Lake, integrates different core architectures and various accelerating components. This work presents an architecture overview of Alder Lake and an in-depth analysis of its power efficiency properties and techniques. For example, this includes frequency scaling of different components, idle states and their latencies, integrated energy measurement capabilities, and recently introduced processor feedback interfaces and OS integration.

Sustainable computing necessitates metrics for calculating carbon emissions of HPC systems and applications. While some established metrics focus on the data center level targeting HPC operators, any publicly reported data is too unspecific and untimely for HPC users. HPC users are often left only with the ability to estimate based on generic and sometimes arbitrarily chosen parameters. This paper fills a research gap by providing a methodology for HPC operators to calculate user-centric operational carbon footprints. Three carbon accounting levels are defined, focussing on ease of applicability for initial adoption and incrementally increasing reporting accuracy and time resolution. We apply this methodology to two German HPC centers, providing a variability analysis for the calculation parameters. The results reveal significant hourly, daily, and monthly fluctuations in the carbon intensity of the German energy mix and the clusters' power consumption, as well as hourly and monthly trends for the power usage effectiveness if free air cooling is employed.

The SPEC Power benchmark offers valuable insights into the energy efficiency of server systems, allowing comparisons across various hardware and software configurations. Benchmark results are publicly available for hundreds of systems from different vendors, published since 2007. We leverage this data to perform an analysis of trends in x86 server systems, focusing on power consumption, energy efficiency, energy proportionality and idle power consumption. Through this analysis, we aim to provide a clearer understanding of how server energy efficiency has evolved and the factors influencing these changes.

High Performance Computing (HPC) systems are among the most energy-intensive scientific facilities, with electric power consumption reaching and often exceeding 20 Megawatts per installation. Unlike other major scientific infrastructures such as particle accelerators or high-intensity light sources, which are few around the world, the number and size of supercomputers are continuously increasing. Even if every new system generation is more energy efficient than the previous one, the overall growth in size of the HPC infrastructure, driven by a rising demand for computational capacity across all scientific disciplines, and especially by Artificial Intelligence (AI) workloads, rapidly drives up the energy demand. This challenge is particularly significant for HPC centers in Germany, where high electricity costs, stringent national energy policies, and a strong commitment to environmental sustainability are key factors. This paper describes various state-of-the-art strategies and innovations employed to enhance the energy efficiency of HPC systems within the national context. Case studies from leading German HPC facilities illustrate the implementation of novel heterogeneous hardware architectures, advanced monitoring infrastructures, high-temperature cooling solutions, energy-aware scheduling, and dynamic power management, among other optimisations. By reviewing best practices and ongoing research, this paper aims to share valuable insight with the global HPC community, motivating the pursuit of more sustainable and energy-efficient HPC architectures and operations.

Modern processors, in particular within the server segment, integrate more cores with each generation. This increases their complexity in general, and that of the memory hierarchy in particular. Software executed on such processors can suffer from performance degradation when data is distributed disadvantageously over the available resources. To optimize data placement and access patterns, an in-depth analysis of the processor design and its implications for performance is necessary. This paper describes and experimentally evaluates the memory hierarchy of AMD EPYC Rome and Intel Xeon Cascade Lake SP server processors in detail. Their distinct microarchitectures cause different performance patterns for memory latencies, in particular for remote cache accesses. Our findings illustrate the complex NUMA properties and how data placement and cache coherence states impact access latencies to local and remote locations. This paper also compares theoretical and effective bandwidths for accessing data at the different memory levels and main memory bandwidth saturation at reduced core counts. The presented insight is a foundation for modeling performance of the given microarchitectures, which enables practical performance engineering of complex applications. Moreover, security research on side-channel attacks can also leverage the presented findings.

Processor stress tests target to maximize processor power consumption by executing highly demanding workloads. They are typically used to test the cooling and electrical infrastructure of compute nodes or larger systems in labs or data centers. While multiple of these tools already exists, they have to be re-evaluated and updated regularly to match the developments in computer architecture. This paper presents the first major update of FIRESTARTER, an Open Source tool specifically designed to create near-peak power consumption. The main new features concern the online generation of workloads and automatic self-tuning for specific hardware configurations. We further apply these new features on an AMD Rome system and demonstrate the optimization process. Our analysis shows how accesses to the different levels of the memory hierarchy contribute to the overall power consumption. Finally, we demonstrate how the auto-tuning algorithm can cope with different processor configurations and how these influence the effectiveness of the created workload.

In High Performance Computing, systems are evaluated based on their computational throughput. However, performance in contemporary server processors is primarily limited by power and thermal constraints. Ensuring operation within a given power envelope requires a wide range of sophisticated control mechanisms. While some of these are handled transparently by hardware control loops, others are controlled by the operating system. A lack of publicly disclosed implementation details further complicates this topic. However, understanding these mechanisms is a prerequisite for any effort to exploit the full computing capability and to minimize the energy consumption of today's server systems. This paper highlights the various energy efficiency aspects of the AMD Zen 2 microarchitecture to facilitate system understanding and optimization. Key findings include qualitative and quantitative descriptions regarding core frequency transition delays, workload-based frequency limitations, effects of I/O die P-states on memory performance as well as discussion on the built-in power monitoring capabilities and its limitations. Moreover, we present specifics and caveats of idle states, wakeup times as well as the impact of idling and inactive hardware threads and cores on the performance of active resources such as other cores.

The overwhelming majority of High Performance Computing (HPC) systems and server infrastructure uses Intel x86 processors. This makes an architectural analysis of these processors relevant for a wide audience of administrators and performance engineers. In this paper, we describe the effects of hardware controlled energy efficiency features for the Intel Skylake-SP processor. Due to the prolonged micro-architecture cycles, which extend the previous Tick-Tock scheme by Intel, our findings will also be relevant for succeeding architectures. The findings of this paper include the following: C-state latencies increased significantly over the Haswell-EP processor generation. The mechanism that controls the uncore frequency has a latency of approximately 10 ms and it is not possible to truly fix the uncore frequency to a specific level. The out-of-order throttling for workloads using 512 bit wide vectors also occurs at low processor frequencies. Data has a significant impact on processor power consumption which causes a large error in energy models relying only on instructions.

In this paper we present MetricQ, a novel infrastructure for collecting, archiving, and analyzing sensor data. Core components of MetricQ are a scalable message broker based on the Advanced Message Queuing Protocol, and a newly developed Hierarchical Timeline Aggregation (HTA) storage concept that is specifically designed for timeseries data. HTA requires moderate data processing during data collection and a storage space overhead of about 10 %, and in turn reduces the complexity of typical timeline request from O(N) to O(1). This enables access to very large metric timelines spanning years and billions of data points at a performance level that is sufficient for interactive use cases. In contrast to existing solutions in this domain, no relevant information such as very short peaks in the data is discarded. We demonstrate how we use MetricQ with few metrics at very high update rates, e.g., for energy efficiency research, and for a very large number of metrics at moderate update rates, e.g., monitoring data from the electrical and cooling infrastructure of our data center.

The rising concern for power consumption of large-scale computer systems puts a research focus on the respective measurement methods. Varying workload patterns and energy efficiency optimizations cause highly dynamic power consumption on today’s compute nodes—a challenge for every measurement infrastructure. We identify five partly contradictory requirements that characterize such infrastructures: temporal granularity, spatial granularity, well-defined accuracy, scalability, and cost. In two projects we push the boundaries for these criteria: a scalable measurement solution for hundreds of nodes at millisecond granularity that is tightly integrated into the HPC system, and a sophisticated single-node instrumentation to measure the power consumption of application events in the microsecond range. Both measurement solutions are calibrated and their accuracy is carefully studied. We discuss scalable processing of the measurements for global monitoring in large-scale systems and use this data for energy efficiency analyses in combination with contextual information such as application performance trace data.

Sleep states are an important and well-understood feature of modern server and desktop CPUs that enable significant power savings during idle and partial load scenarios. Making proper decisions about how to use this feature remains a major challenge for operating systems since it requires a trade-off between potential energy-savings and performance penalties for long and short phases of inactivity, respectively. In this paper we analyze the default behavior of the Linux kernel in this regard and identify weaknesses of certain default assumptions. We derive pathological patterns that trigger these weaknesses and lead to 'Powernightmares' during which power-saving sleep states are used insufficiently. Our analysis of a workstation and a large supercomputer reveals that these scenarios are relevant on real-life systems in default configuration. We present a methodology to analyze these effects in detail despite their inherent nature of being hardly observable. Finally, we present a concept to mitigate these problems and reclaim lost power saving opportunities.

Performance measurement and runtime tuning tools are both vital in the HPC software ecosystem and use similar techniques: the analyzed application is interrupted at specific events and information on the current system state is gathered to be either recorded or used for tuning. One of the established performance measurement tools is Score-P. It supports numerous HPC platforms and parallel programming paradigms. To extend Score-P with support for different back-ends, create a common framework for measurement and tuning of HPC applications, and to enable the re-use of common software components such as implemented instrumentation techniques, this paper makes the following contributions: (1) We describe the Score-P metric plugin interface, which enables programmers to augment the event stream with metric data from supplementary data sources that are otherwise not accessible for Score-P. (2) We introduce the flexible Score-P substrate plugin interface that can be used for custom processing of the event stream according to the specific requirements of either measurement, analysis, or runtime tuning tasks. (3) We provide examples for both interfaces that extend Score-P’s functionality for monitoring and tuning purposes.

Modern processors incorporate several performance monitoring units, which can be used to count events that occur within different components of the processor. They provide access to information on hardware resource usage and can therefore be used to detect performance bottlenecks. Thus, many performance measurement tools are able to record them complementary to information about the application behavior. However, the exact meaning of the supported hardware events is often incomprehensible due to the system complexity and partially lacking or even inaccurate documentation. For most events it is also not documented whether a certain rate indicates a saturated resource usage. Therefore, it is usually difficult to draw conclusions on the performance impact from the observed event rates. In this paper, we evaluate whether hardware performance counters can be used to measure the capacity utilization within the memory hierarchy and estimate the impact of memory accesses on the achieved performance. The presented approach is based on a small selection of micro-benchmarks that constantly stress individual components in the memory subsystem, ranging from caches to main memory. These workloads are used to identify hardware performance counters that provide good estimates for the utilization of individual components in the memory hierarchy. However, since access latencies can be interleaved with computing instructions, a high utilization of the memory hierarchy does not necessarily result in low performance. We therefore also investigate which stall counters provide good estimates for the number of cycles that are actually spent waiting for the memory hierarchy.

There is a consensus that exascale systems should operate within a power envelope of 20MW. Consequently, energy conservation is still considered as the most crucial constraint if such systems are to be realized. So far, most research on this topic focused on strategies such as power capping and dynamic power management. Although these approaches can reduce power consumption, we believe that they might not be sufficient to reach the exascale energy-efficiency goals. Hence, we aim to adopt techniques from embedded systems, where energy-efficiency has always been the fundamental objective. A successful energy-saving technique used in embedded systems is to integrate fine-grained autotuning with dynamic voltage and frequency scaling. In this paper, we apply a similar technique to a real-world HPC application. Our experimental results on a HPC cluster indicate that such an approach saves up to 20% of energy compared to the baseline configuration, with negligible performance loss.

Energy efficiency is an important aspect of future exascale systems, mainly due to rising energy cost. Although High performance computing (HPC) applications are compute centric, they still exhibit varying computational characteristics in different regions of the program, such as compute-, memory-, and I/O-bound code regions. Some of today's clusters already offer mechanisms to adjust the system to the resource requirements of an application, e.g., by controlling the CPU frequency. However, manually tuning for improved energy efficiency is a tedious and painstaking task that is often neglected by application developers. The European Union's Horizon 2020 project READEX (Runtime Exploitation of Application Dynamism for Energy-efficient eXascale computing) aims at developing a tools-aided approach for improved energy efficiency of current and future HPC applications. To reach this goal, the READEX project combines technologies from two ends of the compute spectrum, embedded systems and HPC, constituting a split design-time/runtime methodology. From the HPC domain, the Periscope Tuning Framework (PTF) is extended to perform dynamic auto-tuning of fine-grained application regions using the systems scenario methodology, which was originally developed for improving the energy efficiency in embedded systems. This paper introduces the concepts of the READEX project, its envisioned implementation, and preliminary results that demonstrate the feasibility of this approach.

In this paper we present lo2s - a lightweight performance monitoring tool to sample applications as well as the executing system. It enables the user to analyze the performance of a parallel application without requiring the time-consuming and error-prone process of application instrumentation. The collected performance data is complemented with various metric data, i.e., perf counters, kernel tracepoints, model specific registers, and custom metric data provided by plugins. Comprehensive visualization is enabled by compatibility with established tools.

The rate of innovation in IT system design and especially in High Performance Computing continues to be very high. To keep pace TU Dresden has constructed its new data center using the Plenum concept. The traditional raised floor was substituted by a full building story, creating a highly flexible space to transport power, water, and air. A strict hot-aisle air separation is used and the computer room air-handling (CRAH) units in downflow configuration are positioned directly beneath the hot aisles. This unique arrangement necessitates an unconventional downward flow of hot air from the enclosed hot aisle. Extensive testing has been performed in a cluster of 24 racks (12 per side) equipped with (3+1)×100 kW CRAH unit cooling capacity and 60 test fixtures (air heaters) with 5-15 kW heating power each. Our analysis demonstrates the extremely high efficiency of this air cooling concept even in high-density configurations, up to at least 30 kW per rack. This efficiency is mostly due to the very short airflow paths and wide open cross-sections. We also showcase that no malicious thermal stratification occurs in our hot air downflow configuration. A detailed analysis of the CRAH controls for temperature (through cooling water flow modulation) and airflow (fan speed) highlights the challenges of such control systems in enclosed hot aisle configurations at high power density and short feedback loops. The analysis also considers dynamically changing load patterns including very low partial load scenarios and aspects of operational reliability.