An Automated Component-Based Performance Experiment Environment

Performance experiments can involve multiple iterative runs where parameters such as the execution platform, measurement tools, methods of measurement, application parameters, and analysis techniques can vary. In order to manage the layers of complexity involved in setting up and running experiments, data management, and post analysis, a degree of abstraction and automation is necessary for each phase. We present an integrated component-based environment that leverages existing performance measurement and analysis technologies and automates the process of running multiple performance experiments and parameter selection of parallel scientific applications. Our component toolkit will enable application scientists to easily modify the experimental parameters over multiple execution runs and to selectively retrieve the data for analysis and generation of performance models.