Dynamic Test Case Selection in Continuous Integration: Test Result Analysis Using the Eiffel Framework

The popular agile practices of continuous integration and delivery stress the rapid and frequent production of release candidates and evaluation of those release candidates, respectively. Particularly in the case of very large software systems and highly variable systems, these aspirations can come into direct conflict with the need for both thorough and extensive testing of the system in order to build the highest possible confidence in the release candidate. There are multiple strategies to mitigate this conflict, from throwing more resources at the problem to avoiding end-to-end scenario tests in favor of lower-level unit or component tests. Selecting the most valuable tests to execute at any given time, however, plays a critical role in this context: repeating the same static test scope over and over again is a waste that large development projects can ill afford. While a number of alternatives for dynamic test case selection exist - alternatives that may be used interchangeably or even in tandem - many require analysis of large quantities of in situ real-time data in the form of trace links. Generating and analyzing such data is a recognized challenge in industry. In this chapter we investigate one approach to the problem, based on the Eiffel framework for continuous integration and delivery.