HPC Workload Analysis Using Distributed Cross-ISA Binary Instrumentation

Developing distributed High Performance Computing (HPC) applications is challenging, with complex interactions between application, runtime environment, processing cores, and network to obtain the highest performance that a given distributed computing system can provide. HPC systems are evolving at a fast pace, so applications must often be ported. Generally, developers natively run their applications on current machines and extrapolate the performances on future ones. However, modern and future HPC machines contain multiple nodes, each with multiple general-purpose processor cores, possibly with an Instruction Set Architecture (ISA) different from the previous generations, as well as new domain-specific accelerators, so simple extrapolations may not be accurate. Instead, we propose an automated approach to execute and non-intrusively characterize distributed HPC applications on a QEMU-based, cross-ISA, distributed simulation platform. As part of this automated approach, we propose a QEMU plugin to extract metrics at run-time during the execution of distributed applications. The approach is demonstrated on a RISC-V-based distributed multi-node architecture. It achieves an average speedup of almost 3.5 x on a single host machine with 16 virtual nodes in comparison with a single node. Using QEMU plugins for collecting Message Passing Interface (MPI) runtime metrics slows the simulation by 1.62 x in average, but overall, our approach remains much faster than other simulation platforms.