Unit Testing Tool Competitions – Lessons Learned

This paper reports about the two rounds of the Java Unit Testing Tool Competition that ran in the context of the Search Based Software Testing (SBST) workshop at ICST 2013 and the first Future Internet Testing (FITTEST) workshop at ICTSS 2013. It describes the main objectives of the benchmark, the Java classes that were selected in both competitions, the data that was collected, the tools that were used for data collection, the protocol that was carried out to execute the benchmark and how the final benchmark scores for each participating tool were calculated. Eventually, we discuss the challenges encountered during the events, what we learned and how we plan to improve our framework for future competitions.