Cyber-physical systems allow digital control systems to interact with the physical world using sensors and actuators. They are increasingly being used to automate critical infrastructure, where software faults can have dire consequences. Due to the complex nature and unpredictability of these systems, their resilience is often tested using a technique called fuzzing, which generates quasi-random sequences of sensor and actuator manipulations with the goal of forcing a system into unsafe states. However, there is currently no way of determining which manipulations of a test case cause a failure without systematically removing each one and re-running the test, which can be extremely time-consuming and expensive. In this work, we present CausalCut , a technique that uses causal inference to estimate the causal contribution of each intervention from pre-existing runtime data, thereby reducing the number of times tests must be re-run. We evaluated CausalCut by applying it to two very different systems: an artificial pancreas and a water treatment plant. CausalCut typically managed to remove more than half of the spurious manipulations using fewer executions than the current state of the art, which represents a saving of up to 18 hours and $6300 per test case.

Apartness is a concept developed in constructive mathematics, which has resurfaced as a powerful notion for separating states in the area of model learning and model-based testing. We identify some fundamental shortcomings of apartness in quantitative models, such as in hybrid and stochastic systems. We propose a closely-related alternative, called strong separability and show that using it to replace apartness addresses the identified shortcomings. We adapt a well-known complete model-based testing method, called the Harmonized State Identifiers (HSI) method, to adopt the proposed notion of strong separability. We prove that the adapted HSI method is complete. As far as we are aware, this is the first work to show how complete test suites can be generated for quantitative models such as those found in the development of cyber-physical systems.

Coverage- and diversity-based test selection are two prominent strategies for accelerating regression testing by deriving a smaller set of tests from an original test suite. While coverage-based approaches maximise fault detection, they require up-front code instrumentation and execution. Diversity-based approaches, on the other hand, select tests based on dissimilarity, but it has been found that they discover fewer faults. However, the specific characteristics of the faults detected by these two strategies remain unexplored. This raises critical questions: Does the superior raw performance of coverage-based approaches stem from merely finding “trivial” faults, and how do both strategies fare against subtle, hard-to-detect bugs? This is vital because optimising for raw fault counts alone may create a false sense of security, masking an inability to detect deeper faults that could present unique challenges to system reliability. Mutation analysis offers a means of studying this behaviour, simulating hard-to-find bugs by generating “stubborn” mutants — mutants that can only be detected by a small number of tests. We empirically investigated how coverage- and diversity-based approaches perform with mutants and real faults of varying stubbornness across seven Java projects. Results show that diversity-based selection often rivals or surpasses coverage for the most difficult mutants, requiring up to $46 \%$ fewer tests. We further show that tests killing only stubborn mutants constitute just $\mathbf{2 - 2 5 \%}$ of a suite yet achieve full mutation adequacy and detect up to 88% of real faults, and that such mutants are disproportionately found in private, void, and deeply nested methods. These findings highlight stubborn mutants as efficient test targets and lightweight diversity-based selection as a scalable alternative to coverage, avoiding pre-execution and instrumentation costs.

There is extensive literature on automated test generation using reactive design models, where control is determined by events. In contrast, the (idealised) simulation paradigm defines control through cycles dictated by the passage of time. Within each cycle, inputs are read and processed, and outputs are provided, all instantaneously, and afterwards time progresses. To exercise a simulation using tests generated from a reactive design model requires changes to the tests to take into account this paradigm shift. This article focuses on automation of the necessary changes and of the use of the resulting tests in a simulation campaign. Based on a notion of conformance that establishes whether a simulation is correct with respect to a reactive design, we (1) identify the reactive tests that are meaningful; (2) define a process to convert those tests; (3) provide an algorithm to execute those tests and (4) prove soundness and completeness of our approach. Our work is described in the context of the RoboStar framework for model-based development of control software for robotics applications, and its process algebraic semantics. The testing approach we propose here represents a significant advancement in the current testing practices within the field of robotics, where simulations are widely used.

In recent years, the rapid development of deep neural networks and their application technologies has propelled autonomous driving systems (ADS) towards commercialisation. However, due to the high safety requirements of ADSs, ensuring their safety and reliability remains a critical challenge in software engineering. Existing testing methods focus on simple driving scenarios with few traffic participants, neglecting the impact of high traffic density on ADS driving performance. This paper presents an empirical study exploring how traffic density affects ADS behaviour. We developed a testing framework using two open-source ADSs to generate scenarios with varying traffic densities in a high-fidelity simulator. Our results indicate that changes in traffic density significantly affect ADS performance. Different traffic densities reveal various types of safety violations and help identify potential design flaws in ADSs. This study highlights the importance of considering traffic density in ADS testing and contributes to a better understanding of ADS performance under different traffic conditions.

Software Testing is normally one of the main forms of verification and validation used in software development but it is often manual and so expensive and error prone. One of the proposed solutions to this is to use model-based testing, in which testing is based on a model of how the system should behave. If the model has a formal semantics, then there is potential to automate systematic test generation. In this paper we consider the case where the semantics of the model is a set of refusal traces, also called failure traces. We show how the notions of fundamental refusal and fundamental refusal trace can be used to derive a normalised transition system, which we call an observation transition system (OTS), from the semantics. We then show how, if this OTS has finitely many states, and we are given a bound m, one can produce a corresponding complete test suite: one that is guaranteed to determine correctness as long as the number of states of the OTS defined by the semantics of the system under test has no more than m states. In practice, the choice of value for m might be based on domain knowledge or a cost-benefit analysis. As far as we are aware, this is the first work to show how a finite complete test suite can be derived when the semantics under consideration is a set of refusal traces.

Software systems with large parameter spaces, nondeterminism and high computational cost are challenging to test. Recently, software testing techniques based on causal inference have been successfully applied to systems that exhibit such characteristics, including scientific models and autonomous driving systems. One significant limitation is that these are restricted to test properties where all of the variables involved can be observed and where there are no interactions between variables. In practice, this is rarely guaranteed; the logging infrastructure may not be available to record all of the necessary runtime variable values, and it can often be the case that an output of the system can be affected by complex interactions between variables. To address this, we leverage two additional concepts from causal inference, namely effect modification and instrumental variable methods. We build these concepts into an existing causal testing tool and conduct an evaluative case study which uses the concepts to test three system-level requirements of CARLA, a high-fidelity driving simulator widely used in autonomous vehicle development and testing. The results show that we can obtain reliable test outcomes without requiring large amounts of highly controlled test data or instrumentation of the code, even when variables interact with each other and are not recorded in the test data.

Regression testing assures software correctness after changes but is resource-intensive. Test Case Prioritisation (TCP) mitigates this by ordering tests to maximise early fault detection. Diversity-based TCP prioritises dissimilar tests, assuming they exercise different system parts and uncover more faults. Traditional static diversity-based TCP approaches (i.e., methods that utilise the dissimilarity of tests), like the state-of-the-art FAST approach, rely on textual diversity from test source code, which is effective but inefficient due to its relative verbosity and redundancies affecting similarity calculations. This paper is the first to study bytecode as the basis of diversity in TCP, leveraging its compactness for improved efficiency and accuracy. An empirical study on seven Defects4J projects shows that bytecode diversity improves fault detection by 2.3-7.8% over text-based TCP. It is also 2-3 orders of magnitude faster in one TCP approach and 2.5-6 times faster in FAST-based TCP. Filtering specific bytecode instructions improves efficiency up to fourfold while maintaining effectiveness, making bytecode diversity a superior static approach.

ABSTRACTThere has been a significant amount of interest regarding the use of DBTtsfull in software testing over the past two decades. Diversity‐based testing (DBT) technique uses similarity metrics to leverage the dissimilarity between software artefacts—such as requirements, abstract models, programme structures or inputs—in order to address a software testing problem. DBT techniques have been used to assist in finding solutions to several different types of problems including generating test cases, prioritizing them and reducing very large test suites. This paper is a systematic mapping study of DBT techniques that summarizes the key aspects and trends of 167 papers that report the use of 79 different similarity metrics with 22 different types of software artefacts, which have been used by researchers to tackle 11 different types of software testing problems. We further present an analysis of the recent trends in DBT techniques and review the different application domains to which the techniques have been applied, giving an overview of the tools developed by researchers in order to do so. Finally, the paper identifies some DBT challenges that are potential topics for future work, such as exploring other diversity artefacts and measuring diversity for complex data input.

The practice of testing software systems modelled as Finite State Machines (FSMs) has garnered significant attention owing to its simplicity. In FSM-based testing, the tester derives a test suite from the FSM model representing the system’s specification. Subsequently, this test suite is executed against the implementation, and the tester uses the output to decide whether the implementation conforms to the specification. Often, a test suite generation technique requires input sequences to check whether the FSM is in the intended state. This task is referred to as state identification and is often carried out using a set of input sequences called a characterising set. Even though the use of characterising sets simplifies testing, they require a reliable reset or reset sequence and additional transfer sequences. Unfortunately, resetting the underlying system can be costly or may entail manual configuration. In addition, transfer sequences do not directly contribute to testing. This work introduces a class of characterising sets (Ordered Characterising Sets (O-WSets)) that avoid using resets or transfers by design. We show that checking the existence of such a characterising set is NP-complete. We introduce the notion of bounded O-WSets (BO-WSets), which are types of O-WSets that limit transfer usage, and give an algorithm that constructs these. In experiments, on average, the proposed approach led to reductions in the number of resets (95% for real FSMs; 99.73% for synthetic FSMs), the number of transfer inputs (53% for real FSMs; 63.3% for synthetic FSMs) and the number of inputs in state identification sequences (50% for real FSMs; 66.6% for synthetic FSMs). Additionally, the proposed algorithm reduced the time and memory required to derive state identification sequences by 85% and 23%, respectively. Finally, the approach led to test suites with 49.3% fewer sequences and 33.3% fewer inputs on average.

Random testing approaches work by generating inputs at random, or by selecting inputs randomly from some pre-defined operational profile. One long-standing question that arises in this and other testing contexts is as follows: When can we stop testing? At what point can we be certain that executing further tests in this manner will not explore previously untested (and potentially buggy) software behaviors? This is analogous to the question in Machine Learning, of how many training examples are required in order to infer an accurate model. In this paper we show how probabilistic approaches to answer this question in Machine Learning (arising from Computational Learning Theory) can be applied in our testing context. This enables us to produce an upper bound on the number of tests that are required to achieve a given level of adequacy. We are the first to enable this from only knowing the number of coverage targets (e.g. lines of code) in the source code, without needing to observe a sample test executions. We validate this bound on a large set of Java units, and an autonomous driving system.

Testing is a crucial phase in the development of complex systems, and this has led to interest in automated test generation techniques based on state-based models. Many approaches use models that are types of finite state machine (FSM). Corresponding test generation algorithms typically require that certain test components, such as reset sequences (RSs) and preset distinguishing sequences (PDSs), have been produced for the FSM specification. Unfortunately, the generation of RSs and PDSs is computationally expensive, and this affects the scalability of such FSM-based test generation algorithms. This paper addresses this scalability problem by introducing a reinforcement learning framework: the Q-Graph framework for MBT. We show how this framework can be used in the generation of RSs and PDSs and consider both (potentially partial) timed and untimed models. The proposed approach was evaluated using three types of FSMs: randomly generated FSMs, FSMs from a benchmark, and an FSM of an Engine Status Manager for a printer. In experiments, the proposed approach was much faster and used much less memory than the state-of-the-art methods in computing PDSs and RSs.

From simulating galaxy formation to viral transmission in a pandemic, scientific models play a pivotal role in developing scientific theories and supporting government policy decisions that affect us all. Given these critical applications, a poor modelling assumption or bug could have far-reaching consequences. However, scientific models possess several properties that make them notoriously difficult to test, including a complex input space, long execution times, and non-determinism, rendering existing testing techniques impractical. In fields such as epidemiology, where researchers seek answers to challenging causal questions, a statistical methodology known as Causal inference has addressed similar problems, enabling the inference of causal conclusions from noisy, biased, and sparse data instead of costly experiments. This article introduces the causal testing framework: a framework that uses causal inference techniques to establish causal effects from existing data, enabling users to conduct software testing activities concerning the effect of a change, such as metamorphic testing, a posteriori . We present three case studies covering real-world scientific models, demonstrating how the causal testing framework can infer metamorphic test outcomes from reused, confounded test data to provide an efficient solution for testing scientific modelling software.

Causal reasoning is becoming an increasingly popular technique for testing software. In this setting, the tester starts from a simple directed graph that captures their underlying understanding of causal relationships between relevant variables in the program, and this knowledge is then used to reason about causal input-output relationships that are observed during testing. One question that has not yet been addressed in this context is how to measure test adequacy: How do we know whether a causal relationship (or set of relationships) has been properly established by a test set? In this paper we present a metric inspired by Weyuker's notion of inference adequacy. For a given causal relationship, we estimate the causal effect from the test data. The basis of our adequacy metric is then an estimate of the convergence of this estimate, which we calculate using statistical bootstrapping. We evaluate our metric on tests for three diverse computational models. The results show a statistically significant correlation between our metric and a test suite's ability to detect mutants, and also that it is a good indicator of whether a sufficient number of system executions have been observed to trust the outcome of the test.

Regression test suites can have a large number of test cases, especially automatically generated ones, and tend to grow in size, making it costly to run the entire test suite. Test suite reduction aims to eliminate some test cases to reduce the test suite size and therefore reduce the cost of running it. In this paper, string distances on the text of the test cases are used as measures of similarity for reduction. A practical benefit of using string distance is that there is no need to run the test cases: the test suite source code is the only requirement, making the approach fast. We reduce test suites generated from Randoop and EvoSuite; two well-known test generation tools of Java programs. We implemented a string-based similarity reduction and compared it against random reduction. In the experiments, mutation scores using reduced test suites based on maximising string dissimilarity of test cases were higher than those for random reduction in over 70% of the test suites generated. Also, the results showed that test suites generated by Randoop can be drastically reduced in one case by 99% using the string-based similarity reduction approach while maintaining the fault-finding capabilities of the original test suite. Finally, on average, the normalised compression distance was found to be the best similarity metric choice in terms of fault-detection.

We explore the problem of finding a minimal complete test suite for a refusal trace (or failure trace) semantics. Since complete test suites are typically infinite, we consider the setting with a bound ℓ on the length of refusal traces of interest. A test suite T is thus complete if it is failed by all processes that contain a disallowed refusal trace of length at most ℓ.The proposed approach is based on generating a minimal complete set of forbidden refusal traces. Our solution utilises several interesting insights into refusal trace semantics. In particular, we identify a key class of refusals called fundamental refusals which essentially determine the refusal trace semantics, and the associated fundamental equivalence relation. We then propose a small but not necessarily minimal test suite based on our theory, which can be constructed with a simple algorithm. Subsequently, we provide an enumerative method to remove all redundant traces from our complete test suite, which comes in two variants, depending on whether we wish to retain the highly desirable uniform completeness (guarantee of shortest counterexamples).A related problem is the construction of a characteristic formula of a process P, that is, a formula ΦP such that every process which satisfies ΦP refines P. Our test generation algorithm can be used to construct such a formula using a variant of Hennessy-Milner logic with recursion.

The existing testing theories for CSP cater for verification of interaction patterns (traces) and deadlocks, but not time. We address here refinement and testing based on a dialect of CSP, called tock -CSP, which can capture discrete time properties. This version of CSP has been of widespread interest for decades; recently, it has been given a denotational semantics, and model checking has become possible using a well established tool. Here, we first equip tock -CSP with a novel semantics for testing, which distinguishes input and output events: the standard models of ( tock -)CSP do not differentiate them, but for testing this is essential. We then present a new testing theory for timewise refinement, based on novel definitions of test and test execution. Finally, we reconcile refinement and testing by relating timed ioco testing and refinement in tock -CSP with inputs and outputs. With these results, this paper provides, for the first time, a systematic theory that allows both timed testing and timed refinement to be expressed. An important practical consequence is that this ensures that the notion of correctness used by developers guarantees that tests pass when applied to a correct system and, in addition, faults identified during testing correspond to development mistakes.

Context: Cyber-physical systems present a challenge to testers, bringing complexity and scale to safety-critical and collaborative environments. Digital twins enhance these systems through data-driven and simulation based models coupled to physical systems to provide visualisation, predict future states and communication. Due to the coupling between digital and physical worlds, digital twins provide a new perspective into cyber-physical system testing.Objective: The objectives of this study are to summarise the existing literature on digital-twin-based testing. We aim to uncover emerging areas of adoptions, the testing techniques used in these areas and identify future research areas.Method: We conducted a systematic literature review which answered the following research questions: What cyber-physical systems are digital twins currently being used to test? How are test oracles defined for cyber- physical systems? What is the distribution of white-box, black-box and grey-box modelling techniques used for digital twins in the context of testing? How are test cases defined and how does this affect test inputs?Results: We uncovered 26 relevant studies from 480 produced by searching with a curated search query. These studies showed an adoption of digital-twin-based testing following the introduction of digital twins in industry as well as the increasing accessibility of the technology. The oracles used in testing are the digital twin themselves and therefore rely on both system specification and data derivation. Cyber-physical systems are tested through passive testing techniques, as opposed to either active testing through test cases or predictive testing using digital twin prediction.Conclusions: This review uncovers the existing areas in which digital twins are used to test cyber-physical systems as well as outlining future research areas in the field. We outline how the infancy of digital twins has affected their wide variety of definitions, emerging specialised testing and modelling techniques as well as the current lack of predictive ability.

Cyber–physical systems present a challenge to testers, bringing complexity and scale to safety-critical and collaborative environments. Digital twins enhance these systems through data-driven and simulation based models coupled to physical systems to provide visualisation, predict future states and communication. Due to the coupling between digital and physical worlds, digital twins provide a new perspective into cyber–physical system testing. The objectives of this study are to summarise the existing literature on digital-twin-based testing. We aim to uncover emerging areas of adoptions, the testing techniques used in these areas and identify future research areas. We conducted a systematic literature review which answered the following research questions: What cyber–physical systems are digital twins currently being used to test? How are test oracles defined for cyber–physical systems? What is the distribution of white-box, black-box and grey-box modelling techniques used for digital twins in the context of testing? How are test cases defined and how does this affect test inputs? We uncovered 26 relevant studies from 480 produced by searching with a curated search query. These studies showed an adoption of digital-twin-based testing following the introduction of digital twins in industry as well as the increasing accessibility of the technology. The oracles used in testing are the digital twin themselves and therefore rely on both system specification and data derivation. Cyber–physical systems are tested through passive testing techniques, as opposed to either active testing through test cases or predictive testing using digital twin prediction. This review uncovers the existing areas in which digital twins are used to test cyber–physical systems as well as outlining future research areas in the field. We outline how the infancy of digital twins has affected their wide variety of definitions, emerging specialised testing and modelling techniques as well as the current lack of predictive ability.