Optimizing Continuous Integration by Dynamic Test Proportion Selection

Continuous integration is widely used in modern software engineering. However, it is an expensive practice. The proposed approaches focus on either intra- or inter-build cost reduction. Test case prioritization and selection (TCPS) techniques, typical intra-build techniques, are designed to save the high cost of CI by identifying failed test cases for failed builds. However, existing works are inadequate to distinguish characteristics of builds, but to apply an identical test selection proportion to different builds. Build-prediction techniques, typical inter-build techniques, are designed to save CI cost at the build level. If a build is deemed likely to pass based on the prediction of machine learning models, the whole test suite is skipped for it. Apparently, build in such a manner may miss some realistic failed test cases, if a machine learning model provides a false negative result. In this paper, we propose a dynamic test proportion selection technique DTPS, which incorporates intra- and inter-build cost reduction techniques. DTPS uses build features to construct machine learning models to predict the probability of a specific build failure and transform the probability into the necessary test proportion, with respect to a selected test case prioritization technique. Based on the output of machine learning model, it thus selects a prioritized test suite and a variable proportion of test cases with respect to a build. We constructed a large-scale dataset with approximately 115,000 builds, and conducted a controlled experiment using the dataset. The experiment shows that DTPS outperforms existing techniques significantly. It detects 19.9% to 32.5% more failed test cases, compared with state-of-the-art techniques evaluated in the experiment. At the same time, DTPS performs better than all three existing peer techniques on approximately 47% of projects. Moreover, the experiment also shows that our failure prediction model has an improvement of 0.15 in Area Under Curve (AUC), compared to prior machine learning models.