Software regression testing: practical experience at the ALMA test facility

The Atacama Large Millimeter Array (ALMA) is a joint project between astronomical organizations in Europe, North America, and Japan. ALMA will consist of at least 50 twelve meter antennas operating in the millimeter and submillimeter wavelength range. It will be located at an altitude above 5000 m in the Chilean Atacama desert. The ALMA Test Facility (ATF), located in New Mexico, USA, is a proving ground for development and testing of hardware, software, commissioning and operational procedure. At the ATF emphasis has shifted from hardware testing to software and operational functionality. The support of the varied goals of the ATF requires stable control software and at the same time flexibility for integrating newly developed features. For this purpose regression testing has been introduced in the form of a semi-automated procedure. This supplements the established offline testing and focuses on operational functionality as well as verifying that previously fixed faults did not re-emerge. The regression tests are carried out on a weekly basis as a compromise between the developers' response- and the available technical time. The frequent feedback allows the validation of submitted fixes and the prompt detection of side-effects and reappearing issues. Results of nine months are presented that show the evolution of test outcomes, supporting the conclusion that the regression testing helped to improve the speed of convergence towards stable releases at the ATF. They also provided an opportunity to validate newly developed or re-factored software at an early stage at the test facility, supporting its eventual integration. Hopefully this regression test procedure will be adapted to commissioning operations in Chile.