Evaluating a Test Automation Decision Support Tool

Goldrich and Flamm developed the MITRE Automated Test Decision Framework (ATDF) to help MITRE government sponsors (and, via sharing on GitHub, development organizations in general) move from manually tested legacy software towards automated test, continuous integration, continuous deployment, and, ultimately, DevOps. Often such legacy systems comprise multiple components with manual test procedures. The objective of the empirical study described in this paper is to determine whether ATDF usefully ranks components with respect to Return on Investment (ROI) when introducing automated tests. ROI is simply the ratio of profit to cost. When adding automated tests, what will be the profit that these tests will carry? What is the cost or level of effort to engineer a sufficient set of automated tests? Our evaluation approach models ROI using static defect counts identified by SonarLint and estimated cost to complete testing. We found positive Pearson correlations between normalized ATDF rankings versus the normalized rankings of our evaluation approach. We reject the null hypothesis that there is no correlation between the two rankings.