With the first exascale computing systems in production, tuning and scaling HPC applications to fully utilize the available hardware resources has become more important than ever. Thus, there is a strong need for software tools that assist application developers with this task. The Score-P instrumentation and measurement infrastructure plays a major role in filling this gap. Score-P is a community-driven, highly scalable tool suite for profiling and event tracing of massively parallel HPC application codes, and aimed to be easy to use. It provides measurement data via common data formats and runtime interfaces for a variety of complementary analysis tools developed by multiple institutions and companies, allowing users to gain insights into the communication, synchronization, input/output, and scaling behavior of their applications, pinpointing performance bottlenecks and their causes. In this article, we provide an overview of the current state of the Score-P infrastructure and its related tools ecosystem Cube, Extra-P, TAU, Scalasca, and Vampir. In particular, we detail Score-P's current design and architecture, both of which are highly flexible and extensible. Moreover, we describe how Score-P interacts with the analysis tools mentioned above and highlight the major extensions implemented over the past 10+ years to keep pace with the rapidly changing landscape of HPC hardware and parallel application programming interfaces. Furthermore, we discuss emerging challenges, particularly with respect to the ever-growing heterogeneity in both hardware and software, for collecting and analyzing performance data from applications running on future top-tier computing systems.

Application performance of modern day processors is often limited by the memory subsystem rather than actual compute capabilities. Therefore, data throughput specifications play a key role in modeling application performance and determining possible bottlenecks. However, while peak instruction throughputs and bandwidths for local caches are often documented, the achievable throughput can also depend on the relation between memory access and compute instructions. In this paper, we present an Arm version of the well established x86-membench throughput benchmark, which we have adapted to support all current SIMD extensions of the Armv8 instruction set architecture. We describe aspects of the Armv8 ISA that need to be considered in the portable design of this benchmark. We use the benchmark to analyze the memory subsystem at a fine spatial granularity and to unveil microarchitectural details of three processors: Fujitsu A64FX, Ampere Altra and Cavium ThunderX2. Based on the resulting performance information, we show that instruction fetch and decoder widths become a potential bottleneck for cache-bandwidth-sensitive workloads due to the load-store concept of the Arm ISA.

Power consumption has become a limiting factor in all areas of computing. Hence, making the most of the available power budget is paramount. To use the available budget most efficiently, techniques like dynamic voltage and frequency scaling and idle states can be used. This work analyzes the instructions UMWAIT, TPAUSE, and MWAITX on three different systems. We analyze their instruction latencies, power consumptions, and dependencies on core frequencies. To do so, we introduce benchmarks to gather performance and power parameters, which can be used for future software optimizations. Key findings include: The expected sleep duration passed to UMWAIT and TPAUSE can influence the depth of the user idle state. The actual sleep duration of TPAUSE increases stepwise with an increasing expected sleep duration. Requesting a deeper idle state leads to an additional sleep duration, which increases with a lower core frequency. The core frequency influences the instruction latency of TPAUSE, where a low frequency can lead to an irregular performance pattern. The latency of TPAUSE, UMWAIT, and MWAITX is most often higher than requested on the evaluated systems. Core power consumption can be reduced by similar to 20 % to similar to 70 % compared to the usage of PAUSE. The latency for waking a core in user idle reflects the underlying hardware architecture with tens (desktop architecture with shallow idle states) to hundreds (server architecture with deep idle states) of nanoseconds at nominal frequencies.

The Top-Down Microarchitecture Analysis Method introduced by Yasin [31] established a simple yet effective generic approach to performance analysis. It can be used to analyze which micro architectural resource poses a bottleneck during the execution of arbitrary codes. The recently introduced Golden Cove microarchitecture used in Intel's Sapphire Rapids processors provides an extended, automated hardware support for this analysis. In this paper, we describe the implementation of this micro architectural feature, integrate support into the performance measurement infrastructure Score-P, and explore the capabilities of the resulting tool with two example applications: The Computational Fluid Dynamics (CFD) solver for turbo-machinery applications Hydra [23], and the open-source weather and climate model ICON [32].

As simulations become more realistic, the pursuit of higher accuracy results in extended computation times and substantial energy consumption. This study explores mixed-precision computing as a promising strategy to address these challenges, leveraging computer arithmetic tools to optimize performance. To do so, we used Reactor Simulator and LULESH benchmarks as case studies to evaluate the potential of mixed-precision strategies to reduce both time-to-solution and energy-to-solution. For Reactor Simulator, we achieved more then 30 % reduction in both metrics without compromising accuracy. Similarly, results for LULESH demonstrated improvements of up to 31.5 % in time-to-solution and 25.6 % savings in energy-to-solution.

83 % of the TOP500s performance share is contributed by systems that utilize accelerators. While the overwhelming majority of accelerators in TOP500 systems are Graphics Processing Units (GPUs), other accelerators, that excel in specialized applications, are also present. The NEC SX-Aurora Tsubasa is the most common non-GPU accelerator for November 2024 TOP500 systems and has proven to be efficient for certain codes like the weather and climate model ICON. However, applications have to be carefully optimized to achieve this efficiency. Performance analysis tools, which support developers in understanding the runtime behavior of their programs can help in that regard. With profiling tools programmers can see which functions contribute to the overall runtime, but also which of them are causing high amounts of cache misses. However, the basic principle of profiling - the summarization of metrics over all invocations of a function - can hide differences between subsequent invocations of the same function. Due to its implementation a function could perhaps only exhibit high-amounts of cache misses, increasing its runtime, every fourth call. Hence, tools that allow developers to trace their applications by recording non-aggregated events in the time-domain can further provide insight into the applications performance properties. However, as of today only profiling tools exist for the NEC SX-Aurora Tsubasa. In this paper, we use its performance measurement capabilities to extend the Score-P measurement infrastructure and the lo2s system monitoring tool to enable developers to record the exact execution of programs using tracing. We further demonstrate how developers can find performance issues that cannot be detected with profiling-based tools in the ICON weather and climate model. We also describe the overhead and perturbation that our tools introduce.

Energy-efficient idling of computers has a substantial influence on their lifetime energy consumption, total cost of ownership, and environmental impact. Mechanisms to reduce power consumption during idle rely on complex operating-system support, an area that is prone to regressions. Idle-power regressions are challenging to identify, even when actively looking for them: In-band measurements can easily perturb the monitored idle states. Additionally, as the idle power strongly depends on frequency and duration of interruptions, statistically sound comparisons require long observation periods. In this paper, we present a measurement-based approach to pinpoint regressions in the Linux kernel that degrade the energy efficiency of idle systems. For that, we design an out-of-band measurement infrastructure that avoids the probe effect. Our approach based on bisection can isolate the culprit of regressions across a large number of code changes. We discuss the critical role of classification and present approaches to strengthen its reliability. Finally, we demonstrate our approach on a newly discovered power regression, as well as a known reference case by reliably finding the responsible code change.

The continuous evolution of processors requires vendors to translate ever-growing transistor budgets into performance improvements, e.g., by including more functional units, memory controllers, input/output (I/O) interfaces, graphics processing units (GPUs), and caches. This trend also increases complexity, which cannot be fully hidden from the operating system (OS) or application domains. Issues likewhere to place threads if cores have different frequency ranges or architectures, orwhere to perform a task that might be hardware-accelerated cannot be decided on a hardware level. Moreover, performance improvements need to be achieved within a limited power envelope with energy efficiency as a first order design goal. Introduced power saving techniques, however, can contradict OS and applications performance assumptions. Several processor vendors offer heterogeneous processor architectures, such as ARM's big.LITTLE or Apple M1, combining high-performance and power-efficient cores. Intel's first such architecture, Alder Lake, integrates different core architectures and various accelerating components. This work presents an architecture overview of Alder Lake and an in-depth analysis of its power efficiency properties and techniques. For example, this includes frequency scaling of different components, idle states and their latencies, integrated energy measurement capabilities, and recently introduced processor feedback interfaces and OS integration.

The SPEC Power benchmark offers valuable insights into the energy efficiency of server systems, allowing comparisons across various hardware and software configurations. Benchmark results are publicly available for hundreds of systems from different vendors, published since 2007. We leverage this data to perform an analysis of trends in x86 server systems, focusing on power consumption, energy efficiency, energy proportionality and idle power consumption. Through this analysis, we aim to provide a clearer understanding of how server energy efficiency has evolved and the factors influencing these changes.

Energy costs are a critical consideration for operating High-Performance Computing (HPC) systems, with significant efforts dedicated to reducing the energy expenditure of active computations. However, compute nodes of HPC systems also spend a non-negligible amount of time idling, i.e., without performing any useful work. Optimizing idle power consumption presents an impactful and potentially more accessible opportunity for practical energy savings. This paper surveys factors that influence the idle power consumption of HPC systems. We employed various monitoring tools and power measurements in a structured practical approach to optimize idle power consumption during the installation phase of an HPC system. The results highlight the effectiveness of targeted idle power management strategies, demonstrating compound savings of approximately 90 W per node, and 57 kW for the full system in idle. This thoroughly discussed example can serve as a blueprint for similar optimizations in other HPC environments.

Scheduling is critical for achieving high performance for parallel applications executing on high performance computing (HPC) systems. Scheduling decisions can be taken at batch system, application, and operating system (OS) levels. In this work, we investigate the interaction between the Linux scheduler and various OpenMP scheduling options during the execution of three multithreaded codes on two types of computing nodes. When threads are unpinned, we found that OS scheduling events significantly interfere with the performance of compute-bound applications, aggravating their inherent load imbalance or overhead (by additional context switches). While the Linux scheduler balances system load in the absence of application-level load balancing, we also found it decreases performance via additional context switches and thread migrations. We observed that performing load balancing operations both at the OS and application levels is advantageous for the performance of concurrently executing applications. These results show the importance of considering the role of OS scheduling in the design of application scheduling techniques and vice versa. This work motivates further research into coordination of scheduling within multithreaded applications and the OS.

Modern processors, in particular within the server segment, integrate more cores with each generation. This increases their complexity in general, and that of the memory hierarchy in particular. Software executed on such processors can suffer from performance degradation when data is distributed disadvantageously over the available resources. To optimize data placement and access patterns, an in-depth analysis of the processor design and its implications for performance is necessary. This paper describes and experimentally evaluates the memory hierarchy of AMD EPYC Rome and Intel Xeon Cascade Lake SP server processors in detail. Their distinct microarchitectures cause different performance patterns for memory latencies, in particular for remote cache accesses. Our findings illustrate the complex NUMA properties and how data placement and cache coherence states impact access latencies to local and remote locations. This paper also compares theoretical and effective bandwidths for accessing data at the different memory levels and main memory bandwidth saturation at reduced core counts. The presented insight is a foundation for modeling performance of the given microarchitectures, which enables practical performance engineering of complex applications. Moreover, security research on side-channel attacks can also leverage the presented findings.

Processor stress tests target to maximize processor power consumption by executing highly demanding workloads. They are typically used to test the cooling and electrical infrastructure of compute nodes or larger systems in labs or data centers. While multiple of these tools already exists, they have to be re-evaluated and updated regularly to match the developments in computer architecture. This paper presents the first major update of FIRESTARTER, an Open Source tool specifically designed to create near-peak power consumption. The main new features concern the online generation of workloads and automatic self-tuning for specific hardware configurations. We further apply these new features on an AMD Rome system and demonstrate the optimization process. Our analysis shows how accesses to the different levels of the memory hierarchy contribute to the overall power consumption. Finally, we demonstrate how the auto-tuning algorithm can cope with different processor configurations and how these influence the effectiveness of the created workload.

With today’s top supercomputers consuming several megawatts of power, optimization of energy consumption has become one of the major challenges on the road to exascale computing. The EU Horizon 2020 project READEX provides a tools-aided auto-tuning methodology to dynamically tune HPC applications for energy-efficiency. READEX is a two-step methodology, consisting of the design-time analysis and runtime tuning stages. At design-time, READEX exploits application dynamism using the \({readex\_intraphase}\) and the \({readex\_interphase}\) tuning plugins, which perform tuning steps, and provide tuning advice in the form of a tuning model. During production runs, the runtime tuning stage reads the tuning model and dynamically switches the settings of the tuning parameters for different application regions. Additionally, READEX also includes a tuning model visualizer and support for tuning application level tuning parameters to improve the result beyond the automatic version. This paper describes the state of the art used in READEX for energy-efficiency auto-tuning for HPC. Energy savings achieved for different proxy benchmarks and production level applications on the Haswell and Broadwell processors highlight the effectiveness of this methodology.

Within the last years, Python became more prominent in the scientific community and is now used for simulations, machine learning, and data analysis. All these tasks profit from parallelism and offloading. In the domain of High Performance Computing (HPC), we can look back to decades of experience exploiting the different levels of parallelism. By using performance analysis tools like Score-P, we support tuning applications and their parallelism for performance. The HPC community developed approaches to make this analysis as efficient as possible and reduce the overhead these tools introduced. However, the overhead introduced is still significant for complex Python applications, and the traditional compile-time filtering is not available in Python. Therefore, we come up with a selective instrumentation approach. Using some of Pythons concepts, we designed a powerful and intuitive alternative to compile-time filtering for Python. We finish with a small use case from a real-world machine learning application.

This paper discusses an architectural anomaly observed on server processors of the AMD Zen microarchitecture: At a specific operating point, increasing the number of active cores reduces system power consumption while increasing performance more than proportionally to the additional cores. The occurrence of the anomaly is rooted in the hardware control loop for energy management and software-independent. Experiments show a connection to the AMD turbo frequency feature Max Core Boost Frequency (MCBF). In less efficient configurations, this feature could be employed from a processor's perspective, even though it is not necessarily used on any core. Voltage measurements indicate that the availability of MCBF leads to a higher voltage from mainboard voltage regulators, subsequently raising power consumption unnecessarily. We describe the impact of this anomaly on the performance and energy-efficiency of several micro-benchmarks. The reduced power consumption when additional cores are enabled can lead to higher core frequencies and increased per-core-performance. The presented findings can be used to avoid inefficient core configurations and reduce the overall energy-to-solution.

In High Performance Computing, systems are evaluated based on their computational throughput. However, performance in contemporary server processors is primarily limited by power and thermal constraints. Ensuring operation within a given power envelope requires a wide range of sophisticated control mechanisms. While some of these are handled transparently by hardware control loops, others are controlled by the operating system. A lack of publicly disclosed implementation details further complicates this topic. However, understanding these mechanisms is a prerequisite for any effort to exploit the full computing capability and to minimize the energy consumption of today's server systems. This paper highlights the various energy efficiency aspects of the AMD Zen 2 microarchitecture to facilitate system understanding and optimization. Key findings include qualitative and quantitative descriptions regarding core frequency transition delays, workload-based frequency limitations, effects of I/O die P-states on memory performance as well as discussion on the built-in power monitoring capabilities and its limitations. Moreover, we present specifics and caveats of idle states, wakeup times as well as the impact of idling and inactive hardware threads and cores on the performance of active resources such as other cores.