Automatic parameter tuning of parallel codes is ubiquitous in today's HPC environments where the performance portability of said codes is expected to keep pace with the perpetual release of new hardware. With changes in hardware, it is often the case that finding an optimal configuration of these codes is challenging and only further complicated by the high dimensionality or discontinuous topologies of the tuning spaces. Selecting a proper optimization strategy to automatically search these spaces is paramount to minimizing the energy and time spent on exploring sub-optimal configurations. Unfortunately, it is often the case that these optimizers have hyperparameters of their own, which are sensitive and can greatly affect the outcome of quickly converging to, or even finding an optimal code configuration. Much of the existing autotuning literature tends to use particular optimizers without describing their hyperparameter selection, leaving readers to figure out how to configure their optimizer for the best performance. In this work we compare and contrast the popular global optimization strategy of Bayesian Optimization (BO) to Iwo less popular strategies: Particle Swarm Optimization (PSO), and Covariance Matrix Adaptive Evolution Strategy (CMA-ES). We sweep the hyperparameters of these three optimizers in the context of tuning OpenMP hyperparameters of four classic OpenMP programs: BT, FT, HPCG, and Lulesh. Our study compares the long-term search behavior and average time-to-convergence between these three optimization strategies in tuning OpenMP codes. We contribute a detailed study of these strategies and provide deeper insights as to their sensitivities, noting the conditions where each performs well, and hinting at which optimizers require minimal tuning of their hyperparameters for desirable tuning results.

As the scaling of memory density slows physically, a promising solution is to scale memory logically by enhancing the CPU's memory controller to encode and store data more densely in memory. This is known as hardware memory compression. Hardware memory compression decouples OS-managed physical memory from actual memory (i.e., DRAM); the memory controller spends a dynamically varying amount of DRAM on each physical page, depending on the compressibility of the page's content. The newly-decoupled actual memory effectively forms a new layer of memory beyond the traditional layers of virtual, pseudo-physical, and physical memory. We note unlike these traditional memory layers, each with its own specialized allocation interface (e.g., malloc/mmap for virtual memory, page tables+MMU for physical memory), this new layer of memory introduced by hardware memory compression still awaits its own unique memory allocation interface; its absence makes the allocation of actual memory imprecise and, sometimes, even impossible. Imprecisely allocating less actual memory, and/or unable to allocate more, can harm performance. Even imprecisely allocating more actual memory to some jobs can be harmful as it can result in allocating less actual memory to other jobs in highly-occupied memory systems, where compression is useful. To restore precise memory allocation, we design a new memory allocation specialized for this new layer of memory and, subsequently, architect a new MMU-like component in the memory controller and tackle the corresponding design challenges. We create a full-system FPGA prototype of a hardware-compressed memory system with precise memory allocation. Our evaluations using the prototype show that jobs perform stably under colocation. The performance variation is only 1%-2%; in comparison, it is 19%-89% under the prior art.

The Standard Performance Evaluation Corporation (SPEC) CPU benchmark has been widely used as a measure of computing performance for decades. The SPEC is an industry-standardized, CPU-intensive benchmark suite and the collective data provide a proxy for the history of worldwide CPU and system performance. Past efforts have not provided or enabled answers to questions such as, how has the SPEC benchmark suite evolved empirically over time and what micro-architecture artifacts have had the most influence on performance?—have any micro-benchmarks within the suite had undue influence on the results and comparisons among the codes?—can the answers to these questions provide insights to the future of computer system performance? To answer these questions, we detail our historical and statistical analysis of specific hardware artifacts (clock frequencies, core counts, etc.) on the performance of the SPEC benchmarks since 1995. We discuss in detail several methods to normalize across benchmark evolutions. We perform both isolated and collective sensitivity analyses for various hardware artifacts and we identify one benchmark (libquantum) that had somewhat undue influence on performance outcomes. We also present the use of SPEC data to predict future performance.

Although high-performance computing (HPC) systems have been scaled to meet the exponentially growing demand for scientific computing, HPC performance variability remains a major challenge in computer science. Statistically, performance variability can be characterized by a distribution. Predicting performance variability is a critical step in HPC performance variability management. In this article, we propose a new framework to predict performance distributions. The proposed framework is a modified Gaussian process that can predict the distribution function of the input/output (I/O) throughput under a specific HPC system configuration. We also impose a monotonic constraint so that the predicted function is nondecreasing, which is a property of the cumulative distribution function. Additionally, the proposed model can incorporate both quantitative and qualitative input variables. We predict the HPC I/O distribution using the proposed method for the IOzone variability data. Data analysis results show that our framework can generate accurate predictions, and outperform existing methods. We also show how the predicted functional output can be used to generate predictions for a scalar summary of the performance distribution, such as the mean, standard deviation, and quantiles. Our prediction results can further be used for HPC system variability monitoring and optimization. This article has .

Amdahl's Law painted a bleak picture for large-scale computing.The implication was that parallelism was limited and therefore so was potential speedup.While Amdahl's contribution was seminal and important,it drove others vested in parallel processing to define more clearly why large-scale systems are critical to our future and how they funda-mentally provide opportunities for speedup beyond Amdahl's predictions.In the early 2000s,much like Amdahl,we pre-dicted dire consequences for large-scale systems due to power limits.While our early work was often dismissed,the impli-cations were clear to some:power would ultimately limit performance.In this retrospective,we discuss how power-perfor-mance measurement and modeling at scale led to contributions that have driven server and supercomputer design for more than a decade.While the influence of these techniques is now indisputable,we discuss their connections,limits and addi-tional research directions necessary to continue the performance gains our industry is accustomed to.

Although high-performance computing (HPC) systems have been scaled to meet the exponentially-growing demand for scientific computing, HPC performance variability remains a major challenge and has become a critical research topic in computer science. Statistically, performance variability can be characterized by a distribution. Predicting performance variability is a critical step in HPC performance variability management and is nontrivial because one needs to predict a distribution function based on system factors. In this paper, we propose a new framework to predict performance distributions. The proposed model is a modified Gaussian process that can predict the distribution function of the input/output (I/O) throughput under a specific HPC system configuration. We also impose a monotonic constraint so that the predicted function is nondecreasing, which is a property of the cumulative distribution function. Additionally, the proposed model can incorporate both quantitative and qualitative input variables. We evaluate the performance of the proposed method by using the IOzone variability data based on various prediction tasks. Results show that the proposed method can generate accurate predictions, and outperform existing methods. We also show how the predicted functional output can be used to generate predictions for a scalar summary of the performance distribution, such as the mean, standard deviation, and quantiles. Our methods can be further used as a surrogate model for HPC system variability monitoring and optimization.

In fault tolerance for parallel and distributed systems, message logging protocols have played a prominent role in the last three decades. Such protocols enable local rollback to provide recovery from fail-stop errors. Global rollback techniques can be straightforward to implement but at times lead to slower recovery than local rollback. Local rollback is more complicated but can offer faster recovery times. In this work, we study the power and energy efficiency implications of global and local rollback. We propose a power-efficient version of local rollback to reduce power consumption for non-critical, blocked processes, using Dynamic Voltage and Frequency Scaling (DVFS) and clock modulation (CM). Our results for 3 different MPI codes on 2 parallel systems show that power-efficient local rollback reduces CPU energy waste up to 50% during the recovery phase, compared to existing global and local rollback techniques, without introducing significant overheads. Furthermore, we show that savings manifest for all blocked processes, which grow linearly with the process count. We estimate that for settings with high recovery overheads the total energy waste of parallel codes is reduced with the proposed local rollback.

The demand for memory is ever increasing. Many prior works have explored hardware memory compression to increase effective memory capacity. However, prior works compress and pack/migrate data at a small - memory block-level - granularity; this introduces an additional block-level translation after the page-level virtual address translation. In general, the smaller the granularity of address translation, the higher the translation overhead. As such, this additional block-level translation exacerbates the well-known address translation problem for large and/or irregular workloads. A promising solution is to only save memory from cold (i.e., less recently accessed) pages without saving memory from hot (i.e., more recently accessed) pages (e.g., keep the hot pages uncompressed); this avoids block-level translation overhead for hot pages. However, it still faces two challenges. First, after a compressed cold page becomes hot again, migrating the page to a full 4KB DRAM location still adds another level (albeit page-level, instead of block-level) of translation on top of existing virtual address translation. Second, only compressing cold data require compressing them very aggressively to achieve high overall memory savings; decompressing very aggressively compressed data is very slow (e.g., $\gt 800 ns$ assuming the latest Deflate ASIC in industry). This paper presents Translation-optimized Memory Compression for Capacity (TMCC) to tackle the two challenges above. To address the first challenge, we propose compressing page table blocks in hardware to opportunistically embed compression translations into them in a software-transparent manner to effectively prefetch compression translations during a page walk, instead of serially fetching them after the walk. To address the second challenge, we perform a large design space exploration across many hardware configurations and diverse workloads to derive and implement in HDL an ASIC Deflate that is specialized for memory; for memory pages, it is 4X as fast as the state-of-the art ASIC Deflate, with little to no sacrifice in compression ratio. Our evaluations show that for large and/or irregular workloads, TMCC can either improve performance by 14% without sacrificing effective capacity or provide 2.2x the effective capacity without sacrificing performance compared to a state-of-the-art hardware memory compression for capacity.

Many fields of science rely on the collection of samples and estimation of true population distributions from those samples. There are several effective nonparametric methods for approximating a true distribution from empirical data, however it is unclear which methods produce the best approximations in practice. This work presents a case study on the effectiveness of various distribution approximations. Results show that piecewise linear approximations produce the smallest maximum absolute error, while the classic empirical distribution function (EDF) produces the smallest median absolute error as well as the smallest first quartile and minimum absolute error when approximating a distribution from a sample. When building distribution prediction models, the piecewise quintic and cubic approximations produce the lowest absolute error at most error percentiles. This case study encourages more research on the best methods of fitting empirical data with smooth functions to generate accurate distribution approximations.

Lightning talks of EduHPC are a venue where HPC educators discuss work in progress. This paper summarizes the EduHPC 2020 lightning talks, which cover four very different areas: (i) The simulation-based pedagogy of the EduWRENCH project, including motivations for using simulation to teach High Performance Computing, the design principles underlying EduWRENCH modules, a survey of the available modules, a look at a particular module, plus a conclusion including lesson learned thus far and future plans. (ii) The use of the software-tuning component from Student Cluster Competitions in the HPC master’s program at the University of Liverpool. (iii) Steps being taken by the Computer Systems Genome Project at Virginia Tech to foster a community atmosphere among the diverse students working to catalog the lineage of computer system performance over time. (iv) A 3-semester master’s degree program titled Computational Engineering, focused on HPC training, being offered at the University of Warsaw.

Performance variability is an important factor of high-performance computing (HPC) systems. HPC performance variability is often complex because its sources interact and are distributed throughout the system stack. For example, the performance variability of I/O throughput can be affected by factors such as CPU frequency, the number of I/O threads, file size, and record size. In this paper, we focus on the I/O throughput variability across multiple executions of a benchmark program. For a given system configuration, the distribution of throughputs from run to run is of interest. We conduct large-scale experiments and collect a massive amount of data to study the distribution of I/O throughput under tens of thousands of system configurations. Despite normality often being assumed in the literature, our statistical analysis reveals that the performance variability is not normally distributed under most system configurations. Instead, multimodal distributions are common for many system configurations. We propose the use of mixture distributions to describe the multimodal behavior. Various underlying parametric distributions such as normal, gamma, and the Weibull are considered. We apply an expectation–maximization (EM) algorithm for parameter estimation and use the Bayesian information criterion (BIC) for parametric model selections. We also illustrate how to use the estimated mixture distribution to calculate the number of runs needed for future experiments on variability analysis. The paper provides a useful tool set in studying the behavior of performance variability.

DELAUNAYSPARSE contains both serial and parallel codes written in Fortran 2003 (with OpenMP) for performing medium- to high-dimensional interpolation via the Delaunay triangulation. To accommodate the exponential growth in the size of the Delaunay triangulation in high dimensions, DELAUNAYSPARSE computes only a sparse subset of the complete Delaunay triangulation, as necessary for performing interpolation at the user specified points. This article includes algorithm and implementation details, complexity and sensitivity analyses, usage information, and a brief performance study.

Exponential increases in complexity and scale make variability a growing threat to sustaining HPC performance at exascale. Performance variability in HPC I/O is common, acute, and formidable. We take the first step towards comprehensively studying linear and nonlinear approaches to modeling HPC I/O system variability in an effort to demonstrate that variability is often a predictable artifact of system design. Using over 8 months of data collection on 6 identical systems, we propose and validate a modeling and analysis approach (MOANA) that predicts HPC I/Ovariability for thousands of software and hardware configurations on highly parallel shared-memory systems. Our findings indicate nonlinear approaches to I/Ovariability prediction are an order of magnitude more accurate than linear regression techniques. We demonstrate the use of MOANA to accurately predict the confidence intervals of unmeasured I/O system configurations for a given number of repeat runs - enabling users to quantitatively balance experiment duration with statistical confidence.

Time-driven and access-driven attacks are two dominant types of the timing-based cache side-channel attacks. Despite access-driven attacks are popular in recent years, investigating the time-driven attacks is still worth the effort. It is because, in contrast to the access-driven attacks, time-driven attacks are independent of the attackers'cache access privilege. Although cache configurations can impact the time-driven attacks'performance, it is unclear how different cache parameters influence the attacks'success rates. This question remains open because it is extremely difficult to conduct comparative measurements. The difficulty comes from the unavailability of the configurable caches in existing CPU products. In this paper, we utilize the GEM5 platform to measure the impacts of different cache parameters, including Private Cache Size and Associativity , Shared Cache Size and Associativity , Cacheline Size , Replacement Policy , and Clusivity . In order to make the time-driven attacks comparable, we define the equivalent key length (EKL) to describe the attacks' success rates. Key findings from the measurement results include (i) private cache has a key effect on the attacks'success rates; (ii) changing shared cache has a trivial effect on the success rates, but adding neighbor processes can make the effect significant; (iii) the Random replacement policy leads to the highest success rates while the LRU/LFU are the other way around; (iv) the exclusive policy makes the attacks harder to succeed compared to the inclusive policy. We finally leverage these findings to provide suggestions to the attackers and defenders as well as the future system designers.

Parallel I/O performance is crucial to sustaining scientific applications on large-scale High-Performance Computing (HPC) systems. However, I/O load imbalance in the underlying distributed and shared storage systems can significantly reduce overall application performance. There are two conflicting challenges to mitigate this load imbalance: (i) optimizing system-wide data placement to maximize the bandwidth advantages of distributed storage servers, i.e., allocating I/O resources efficiently across applications and job runs; and (ii) optimizing client-centric data movement to minimize I/O load request latency between clients and servers, i.e., allocating I/O resources efficiently in service to a single application and job run. Moreover, existing approaches that require application changes limit wide-spread adoption in commercial or proprietary deployments. We propose iez, an "end-to-end control plane" where clients transparently and adaptively write to a set of selected I/O servers to achieve balanced data placement. Our control plane leverages real-time load information for distributed storage server global data placement while our design model leverages trace-based optimization techniques to minimize I/O load request latency between clients and servers. We evaluate our proposed system on an experimental cluster for two common use cases: synthetic I/O benchmark IOR for large sequential writes and a scientific application I/O kernel, HACC-I/O. Results show read and write performance improvements of up to 34% and 32%, respectively, compared to the state of the art.

The Delaunay triangulation is a fundamental construct from computational geometry, which finds wide use as a model for multivariate piecewise linear interpolation in fields such as geographic information systems, civil engineering, physics, and computer graphics. Though efficient solutions exist for computation of two- and three-dimensional Delaunay triangulations, the computational complexity for constructing the complete Delaunay triangulation grows exponentially in higher dimensions. Therefore, usage of the Delaunay triangulation as a model for interpolation in high-dimensional domains remains computationally infeasible by standard methods. In this paper, a polynomial time algorithm is presented for interpolating at a finite set of points in arbitrary dimension via the Delaunay triangulation. This is achieved by computing a small subset of the simplices in the complete triangulation, such that all interpolation points lie in the support of the subset. An empirical study on the runtime of the proposed algorithm is presented, demonstrating its scalability to high-dimensional spaces.

Performance variability can have a significant impact on many applications of computing. Cloud computing, high performance computing, and computer security communities each exert considerable effort managing and analyzing variability throughout the system stack. This work presents and evaluates a methodology for predicting precise characteristics of the computational performance variability of an input/output (I/O) application over varying system configurations. Results demonstrate that the presented methodology is capable of precisely modeling performance variability, which could allow applications that tighten service level agreements, maximize computational throughput, and obfuscate system configurations against malicious users.

With the dramatic increase in scale expected for Exascale computing, there is a dire need for tuning of hardware configurations and software optimizations such that they are in unison. However, the expected increase in tunable hardware parameters makes searching through the design space for optimal hardware-and-software configurations much more challenging. Towards this end, we propose a composable hardware-software optimization framework called Prometheus. Prometheus uses a combination of analytical and machine-learning techniques to capture application characteristics and subsequently determine the hardware-software configuration for near-optimal performance. We evaluate Prometheus for its efficacy using two widely used proxy applications: LULESH and CoMD. We demonstrate that Prometheus identifies near-optimal hardware-software configurations and verify the results via brute-force search of the design space.

A rapid increase in the quantity of data available is allowing all fields of science to generate more accurate models of multivariate phenomena. Regression and interpolation become challenging when the dimension of data is large, especially while maintaining tractable computational complexity. This paper proposes three novel techniques for multivariate interpolation and regression that each have polynomial complexity with respect to number of instances (points) and number of attributes (dimension). Initial results suggest that these techniques are capable of effectively modeling multivariate phenomena while maintaining flexibility in different application domains.