BenchTool: a Framework for the Automation and Standardization of HPC Performance Benchmarking

The procurement of large and complex HPC systems necessitates an extensive planning process. This process should ideally include the independent performance evaluation and validation of available solutions to maximize system throughput for expected workloads. This assessment procedure typically requires the arduous task of compiling, configuring and running multiple application benchmarks on every competing architecture, where any combination of hardware and software may produce unique idiosyncrasies and time consuming pitfalls. In addition, the possible lack of standardization and communication within a research group may produce discrepancies in evaluation methodology, affecting consistency between results. The BenchTool utility was developed to provide a framework to automate the compilation of applications, the execution of benchmarks using said applications, and the collection of results including supplementary provenance data.