Comparing Model-Based Testing with Traditional Testing Strategies: an Empirical Study

Testing is an important activity on the project life-cycle. Test suites are generated from system models and can be explicit or not. This generation can follow a hand-crafted approach or an automated one, using strategies such as model-based testing (MBT). The literature acknowledges MBT advantages, but to the best of our knowledge most successful reported experiences are case studies. There are few contributions on the empirical comparison of model-based strategies with manual ones. Therefore, in this work we designed and conducted an empirical experiment comparing two approaches: i) manual ad hoc tests and ii) MBT using TaRGeT tool. For such evaluation, we defined metrics for the time spent by tests and the quantity of detected defects. Furthermore, we correlated these metrics with factors such as the testers experience and systems' complexity. The experiment counted on 54 experimental units, testing 27 use cases across the two approaches and detected a total of 82 new defects on a real project under development for the Federal Police of Brazil. From the experiment results, we observe that both approaches are statistically equivalent, but each one has particularities finding certain types of defects. Thus, we discuss major advantages and drawbacks of each approach.