COSMOS: A comprehensive framework for automatically generating domain-oriented test suite

Context: Coverage criteria are satisfied by at least one examination of the test target, while many faults are not revealed by one execution. However, despite executing the faulty statement, the test result is correct in certain circumstances. Such coincidentally passing test cases execute the faulty statement but do not cause failures. Objective: This paper introduces the new concept of domain solver. Domain solvers attempt to detect the domain of inputs rather than a single input satisfying a path constraint. Domain coverage is a new metric to evaluate the relative accuracy of the detected domains. The promising point is that the proposed approach similarly treats nonlinear and linear constraints.Method: Domain solver splits a path constraint into conjunctions of simple conditions comparing two expressions. Such a splitting simplifies the constraint-solving task to detect regions of the input space satisfying a comparison between two expressions. After finding a region, an improved version of an algorithm, update, is used to determine the domain of variables involved in the comparing expressions.Results: Our proposed approach, COSMOS, is implemented using the Roslyn compiler. We compared COSMOS with well-known constraint solvers using various linear/nonlinear constraints. The results show that COSMOS improves the number of supported data types involved in a constraint and solves100% of the instances in which the other solvers fail. Besides, COSMOS achieves the best relative accuracy of 84% compared to the existing domain-oriented test suite generation approaches. Moreover, our experiment results illustrate that COSMOS improves the fault-finding capability of other existing test coverage criteria by detecting coincidentally correct test cases.Conclusion: Combining domain coverage and compiler as a service makes a powerful constraint satisfaction method outperforming the existing constraint solvers regarding the number of solved linear/nonlinear con-straints. Augmenting other structural test coverage criteria with domain coverage reveals the coincidentally correct test cases.