Using a CPU-GPU hybrid computing framework is becoming a common configuration for supercomputers. The wide deployment of GPUs (as well as other hardware accelerators) brings to the HPC community a big question: Are we using them effectively? Inappropriate use of GPUs can generate incorrect results in certain cases, but more often, will slow down the program instead of speeding it up. This paper describes a tool that satisfies the needs of programmers to analyze the runtime performance of kernels and obtain insights for better GPU utilization. Compared to existing GPU performance tools, ours provides some unique features: data-centric profiling and generating complete GPU call stacks. With the guidance of the tool, we were able to improve the kernel performance of three widely-studied GPU benchmarks by a factor of up to 46.6x with minor code modification.

Parallel programming is hard, and it is even harder to analyze parallel programs and identify specific performance bottlenecks. Chapel is an emerging Partitioned-Global-Address-Space (PGAS) language that provides productive parallel programming. Most established profilers either completely lack the capacity to profile Chapel programs or generate information that cannot provide insightful guidance in a user-level context. To address this issue, we developed ChplBlamer to pinpoint performance losses due to data distribution and remote data accesses. We use a data-centric and code-centric combined approach to help Chapel users quickly identify performance bottlenecks in the source. To demonstrate the utility of ChplBlamer, we studied three multi-locale Chapel benchmarks. For each benchmark, ChplBlamer found the causes of the performance losses. With the optimization guidance provided by ChplBlamer, we significantly improved the performance by up to 4x with little code modification.

We present Purity, a configurable, data-centric, communication profiler for the Chapel language that analyzes memory and communication access patterns in a multi-node PGAS environment. By integrating Purity into the compiler and runtime framework of Chapel we can instrument Chapel programs to capture memory and communication operations and produce both online and fine-grain post execution reporting. Our profiler is equipped with a sampling mechanism for reducing overhead, handles complex data structures, and generates detailed execution profiles that map data motion to the variable, field, loop, and node levels for both distributed and non-distributed instantiations. In a case study, Purity provided valuable insight into task and data locality which allowed us to develop a programmatic solution for reducing nearly 90% of remote operations in SSCA#2.

This project concentrated on various aspects of creating and applying tool infrastructure to make it easier to effectively use large-scale parallel computers. This project was collaborative with Argonne National Laboratory, Lawrence Berkeley National Laboratory, Lawrence Livermore National Laboratory, Oak Ridge National Laboratory, U.C. San Diego, University of Maryland, University of North Carolina, University of Oregon, University Southern California, University of Tennessee, and University of Utah. The research conducted during this project at the University of Maryland is summarized in this report. The complete details of the work are available in the publications listed at the end of the report. Many of the concepts created during this project have been incorporated into tools and made available as freely downloadable software (www.dyninst.org/harmony). It also supported the studies of six graduate students, one undergraduate student, and two post-docs. The funding also provided summer support for the PI and part of the salary of a research staff member.

We developed a data-centric and code-centric combined tool, ChplBlamer, to pinpoint performance loss in multilocale Chapel programs. To evaluate ChplBlamer, we studied three multi-locale Chapel benchmarks. For each one, ChplBlamer found the causes of performance losses. With the optimization guidance provided by ChplBlamer, we significantly improved the performance by up to 4x with little code modification.

Floating-point computation is ubiquitous in high-performance scientific computing, but rounding error can compromise the results of extended calculations, especially at large scales. In this paper, we present new techniques that use binary instrumentation and modification to do fine-grained floating-point precision analysis, simulating any level of precision less than or equal to the precision of the original program. These techniques have an average of 40–70% lower overhead and provide more fine-grained insights into a program’s sensitivity than previous mixed-precision analyses. We also present a novel histogram-based visualization of a program’s floating-point precision sensitivity, as well as an incremental search technique that allows developers to incrementally trade off analysis time for detail, including the ability to restart analyses from where they left off. We present results from several case studies and experiments that show the efficacy of these techniques. Using our tool and its novel visualization, application developers can more quickly determine for specific data sets whether their application could be run using fewer double precision variables, saving both time and memory space.

Autotuning refers to the automatic generation of a search space of possible implementations of a computation that are evaluated through models and/or empirical measurement to identify the most desirable implementation. Autotuning has the potential to dramatically improve the performance portability of petascale and exascale applications. To date, autotuning has been used primarily in high-performance applications through tunable libraries or previously tuned application code that is integrated directly into the application. This paper draws on the authors' extensive experience applying autotuning to high-performance applications, describing both successes and future challenges. If autotuning is to be widely used in the HPC community, researchers must address the software engineering challenges, manage configuration overheads, and continue to demonstrate significant performance gains and portability across architectures. In particular, tools that configure the application must be integrated into the application build process so that tuning can be reapplied as the application and target architectures evolve.

The CASPER system offers a lightweight, multi-disciplinary approach to detect the execution of anomalous code by monitoring the unintended electronic device emissions. Using commodity hardware and a combination of novel signal processing, machine learning, and program analysis techniques, we have demonstrated the ability to detect unknown code running on a device placed 12 '' from the CASPER system by analyzing the devices RF emissions. Our innovations for the sensors subsystem include multi-antenna processing algorithms which allow us to extend range and extract signal features in the presence of background noise and interference encountered in realistic training and monitoring environments. In addition, robust feature estimation methods have been developed that allow detection of device operating conditions in the presence of varying clock frequency and other aspects that may change from device to device or from training to monitoring. Furthermore, a band-scan technique has been implemented to automatically identify suitable frequency bands for monitoring based on a set of metrics including received power, expected spectral feature content (based on loop length and clock frequency), kurtosis, and mode clustering. CASPER also includes an auto-labeling feature that is used to discover the signal processing features that provide the greatest information for detection without human intervention. The system additionally includes a framework for anomaly detection engines, currently populated with three engines based on n-grams, statistical frequency, and control flow. As we will describe, the combination of these engines reduces the ways in which an attacker can adapt in an attempt to hide from CASPER. We will describe the CASPER concept, components and technologies used, a summary of results to-date, and plans for further development. CASPER is an ongoing research project funded under the DARPA LADS program.

The widespread use of computing in the American economy would not be possible without a thoughtful, exploratory research and development (R&D) community pushing the performance edge of operating systems, computer languages, and software libraries. These are the tools and building blocks — the hammers, chisels, bricks, and mortar — of the smartphone, the cloud, and the computing services on which we rely. Engineers and scientists need ever-more specialized computing tools to discover new material properties for manufacturing, make energy generation safer and more efficient, and provide insight into the fundamentals of the universe, for example. The research division of the U.S. Department of Energy’s (DOE’s) Office of Advanced Scientific Computing and Research (ASCR Research) ensures that these tools and building blocks are being developed and honed to meet the extreme needs of modern science. See also http://exascaleage.org/ascr/ for additional information.

Chapel is an emerging PGAS (Partitioned Global Address Space) language whose design goal is to make parallel programming more productive and generally accessible. To date, the implementation effort has focused primarily on correctness over performance. We present a performance measurement technique for Chapel and the idea is also applicable to other PGAS models. The unique feature of our tool is that it associates the performance statistics not to the code regions (functions), but to the variables (including the heap allocated, static, and local variables) in the source code. Unlike code-centric methods, this data-centric analysis capability exposes new optimization opportunities that are useful in resolving data locality problems. This paper introduces our idea and implementations of the approach with three benchmarks. We also include a case study optimizing benchmarks based on the information from our tool. The optimized versions improved the performance by a factor of 1.4x for LULESH, 2.3x for MiniMD, and 2.1x for CLOMP with simple modifications to the source code.

This paper presents a method to design and auto-tune a new parallel 3-D FFT code using the non-blocking MPI all-to-all operation. We achieve high performance by optimizing computation-communication overlap. Our code performs fully asynchronous communication without any support from special hardware. We also improve cache performance through loop tiling. To cope with the complex trade-off regarding our optimization techniques, we parameterize our code and auto-tune the parameters efficiently in a large parameter space. Experimental results from two systems confirm that our code achieves a speedup of up to 1.76x over the FFTW library.

The reviewing process: perhaps no other topic elicits such strong opinions and heated debate among researchers. This is understandable as successfully getting papers through the review process is critical to career advancement in many settings. Likewise, many of us spend significant time reading papers and writing reviews. Given the recent (some might say continual) evolution of paper reviewing at conferences in general, and SC in particular, now seems an apropos time to reflect on the process.

This paper investigates how Chapel performance compares with other parallel frameworks. We provide specific examples of how programmers may improve their single-node (single-locale) Chapel programs to improve performance. We also identify some changes that would be possible to the language to make it easier to get these performance gains. Specifically, we compare the intranode performance of Chapel programs with OpenMP in C/C++ by conducting case studies profiling the LULESH, MiniMD, SSCA#2, and CLOMP benchmarks. Our optimization techniques demonstrate improved runtime performance of Chapel benchmarks by factors of 3x, 5.3x, 6.3x, and 4.8x respectively and outperformed their OpenMP counterparts by factors of 2x for LULESH, 1.6x for SSCA#2, and 4.8x for CLOMP.

This project concentrated on various ways to improve the measurement and tuning large-scale parallel applications. This project was supplement to the project DE-FC0206ER25763 (“Performance Engineering Research Center”). The research conducted during this project is summarized in this report. The complete details of the work are available in the ten publications listed at the end of the report. It also supported the Ph.D. studies of three students and one research scientist.

This paper describes a data-centric profilingtool that provides a way to map performance problemsback to data structures in Chapel programs. Wedescribe the tool's implementation, and illustrate its usewith two simple test programs.

This paper addresses two key parallelization challenges the unstructured mesh-based ocean modeling code, MPAS-Ocean, which uses a mesh based on Voronoi tessellations: (1) load imbalance across processes, and (2) unstructured data access patterns, that inhibit intra- and inter-node performance. Our work analyzes the load imbalance due to naive partitioning of the mesh, and develops methods to generate mesh partitioning with better load balance and reduced communication. Furthermore, we present methods that minimize both inter- and intra- node data movement and maximize data reuse. Our techniques include predictive ordering of data elements for higher cache efficiency, as well as communication reduction approaches. We present detailed performance data when running on thousands of cores using the Cray XC30 supercomputer and show that our optimization strategies can exceed the original performance by over 2×. Additionally, many of these solutions can be broadly applied to a wide variety of unstructured grid-based computations.