Work-in-Progress: Searching Optimal Compiler Optimization Passes Sequence for Reducing Runtime Memory Profile Using Ensemble Reinforcement Learning

The order in which compiler optimization passes are applied has a significant impact on program performance. However, widely used compiler optimization options use handpicked sets of optimization passes, optimized for specific benchmarks. In this paper, we propose an ensemble reinforcement learning (RL) model that optimizes LLVM transform passes sequence to reduce the runtime memory profile, which is an important consideration in resource-constrained embedded systems. We developed an LLVM intermediate representation (IR) analysis pass to extract static program features. The extracted features are processed with PCA for dimension reduction. We also generated datasets using a random program generator, and clustered them according to the PCA results of their extracted features. The ensemble RL model was trained on each clustered dataset. Experiments showed that the proposed model reduced 37% more memory profile than the standard optimization option.