Generating test data is a critical component of software testing, essential for ensuring the proper functioning of software systems. Automated test data generation, in particular, can greatly improve the efficiency and accuracy of the testing process. In 1990, the paper "Automated Software Test Data Generation," published in IEEE Transactions on Software Engineering (Korel, 1990), introduced a novel approach to automated test data generation, focusing on actual program execution, fitness function minimization methods, and dynamic data flow analysis. This paper discusses the impact of the 1990 IEEE TSE publication (Korel, 1990) on automated test data generation and software engineering.

The K-variant is a multi-variant architecture to enhance the security of the time-bounded mission and safety-critical systems. Variants in the K-variant architecture are generated by controlled source program transformations. Previous experimental studies showed that the K-variant architecture might improve the security of systems against memory exploitation attacks. In order to estimate the survivability of K-variant systems, simulation techniques are utilized. However, these techniques are slow and may not be practical for the design of K-variant systems. Therefore, fast and highly accurate estimations of the survivability of K-variant systems are necessary for developers. The neural networks may allow quick and accurate estimation of the survivability of K-variant systems. The developed neural network-based tool can make quick and precise estimations of the survivability of K-variant systems under different conditions. In this paper, the accuracy of the neural network-based tool is investigated in an experimental study. The neural network-based tool estimations are compared with a K-variant attack emulator in three programs for up to ten variant systems under four attack types and three attack durations. The experimental study demonstrates that the neural network-based tool makes fast and accurate estimations of the survivability of K-variant systems under all the conditions investigated.

With the increased adoption of Model-Driven Engineering (MDE), where models are being used as the primary artifact of software, it is apparent that greater attention to the quality of the models is necessary. Traditionally, refactoring is used to enhance the quality of software systems at the source-code level; however, applying refactoring at the model level will have a more significant improvement on the system. After refactoring a model, proving that it still preserves its original behavior is crucial. In this paper, we present a process for applying refactoring transformations to the Extended Finite State Machine (EFSM) models using verified transformation rules that have been proven to preserve the model's original behavior. We provide a simplified three-step verification approach that can be used to prove that a transformation rule will generate a transformed model that is semantically equivalent to the original model. To do this, we formally define semantical equivalence at three different levels of granularity: models, sub-models, and transitions. Additionally, we introduce five model transformation rules and we demonstrate how our verification approach is used to prove the correctness of these rules. Finally, we present two case studies where we apply the proposed transformation process which adopts the five verified transformation rules. Using model testing, we show that applying a sequence of transformations using the verified transformation rules will keep both the original and the transformed model semantically equivalent. Additionally, the case studies show that model transformation can be used to enhance certain pre-defined model characteristics.

Context: Source code transformation techniques can improve the security of systems against memory exploitation attacks. As such, the chance of exploitation of security vulnerabilities can be decreased by using different controlled source code transformation techniques. In K-variant architecture, multiple variants of a program are generated through a controlled source code transformation to improve the security of systems. Objective: To investigate the effectiveness and practicality of source code program transformations in improving the security of time-bounded K-variant systems for memory exploitation attacks. Method: The effectiveness of program transformations in improving the security of time-bounded K-variant systems is experimentally investigated for different memory attacks. Results: The results suggest that generating multiple variants using the presented transformations significantly improves the survivability of time-bounded K-variant systems under memory exploitation attacks. Conclusion: We conclude that generating multi-variants in time-bounded K-variant systems in accordance with the presented program transformations may improve the security of time-bounded K-variant systems significantly for memory exploitation attacks with a reasonable cost and overhead.

Many systems require high security during a limited operational timeframe. These systems can be exposed to many attacks that prevent tasks from being completed. To reduce the chances of a successful attack and increase the security of these systems, many fault tolerance architectures and defense strategies have been developed. N-version architecture is one technique to improve reliability and security. In this architecture, functionally equivalent variants of a program are developed and executed concurrently. However, due to the high cost of developing and maintaining N versions, it is used for large budget projects. In this paper, an alternative approach to improve security is K-variant architecture. By applying safe and automated program transformations, functionally equivalent variants of the original program are generated. The goal is to shift vulnerabilities in variants into different locations. These variants are executed simultaneously to complete a task so that the chance of a successful exploitation attack is decreased. In this paper, several types of memory exploitation attacks are analyzed along with the proposed four defense strategies. The experimental study investigates the effectiveness of defense strategies against these attacks. The results suggest that K-variant architecture with the proposed defense strategies may significantly improve the security of systems.

Software maintenance is becoming more challenging with the increased complexity of the software and the frequently applied modifications. To manage this complexity, systems development is headed towards Model-driven engineering (MDE) and search-based software engineering (SBSE). Additionally, prior to applying a change to these complex systems, change impact analysis is usually performed in order to determine the scope of the change, its feasibility, and the time and resources required to implement the change. The bigger the scope, the riskier the change is on the system. In this paper, we introduce a set of transformation rules for Extended Finite State Machine (EFSM) models of state-based systems. These transformation rules can be used as the basis for search-based model optimization in order to reduce the average impact of a potential change applied to an EFSM model. Assuming that Model-driven development is adopted for the implementation of a state-based system, reducing the change impact at the model level will lead to reducing the impact at the system level. An exploratory study is performed to measure the impact reduction for a given EFSM model when the transformation rules are applied by a search-based algorithm. The initial results show a promising usage of the transformation rules which can lead to a reduction of more than 50% of the initial average change impact of the model.

Software maintenance is becoming more challenging with the increased complexity of the software and the frequently applied changes. Performing impact analysis before the actual implementation of a change is a crucial task during system maintenance. While many tools and techniques are available to measure the impact of a change at the code level, only a few research work is done to measure the impact of a change at an earlier stage in the development process. This work introduces an approach to measure the impact of a change at the model level.

Testing software products is very expensive and time consuming, especially for large software systems with extensive regression testing. During regression testing, a modified system is often re-tested using an existing test suite. Since test suites can be very large, testers are interested in detecting faults in the modified system as early as possible. Test prioritization tries to order test cases for execution in a way that increases the chances of the early detection of faults. Most of the existing test prioritization methods are based on the code of the system under test, but model-based test prioritization has been lately proposed. Most of the existing model-based test prioritization methods can be used only when models are modified during system maintenance. In this paper, we present model-based prioritization for a class of modifications for which models are not modified (only the source code is modified). After identifying the elements of the model related to the modified source code, information collected during the execution of the model is used to prioritize tests for execution. Here, we present and compare existing and new model-based test prioritization methods focused on this class of modifications. The major motivation for presenting these methods is to provide system developers with simple and yet effective test prioritization techniques for early fault detection. Statistical analysis of the empirical study, which compares the effectiveness of the presented methods in terms of early fault detection, show that compared to random ordering of test cases, model-based test prioritization significantly improve the effectiveness of test prioritization with respect to early fault detection.

Software maintenance is becoming more challenging with the increased complexity of the software and the frequently applied changes. Performing impact analysis before the actual implementation of a change is a crucial task during system maintenance. While many tools and techniques are available to measure the impact of a change at the code level, only a few research work is done to measure the impact of a change at an earlier stage in the development process. Measuring the impact of a change at the model level speeds up the maintenance process allowing early discovery of critical components of the system before applying the actual change at the code level. In this paper, we present model-based impact analysis approach for state-based systems such as telecommunication or embedded systems. The proposed approach uses model dependencies to automatically measure the expected impact for a requested change instead of relying on the expertise of system maintainers, and it generates two impact sets representing the lower bound and the upper bound of the impact. Although it can be extended to other behavioral models, the presented approach mainly addresses extended finite-state machine (EFSM) models. An empirical study is conducted on six EFSM models to investigate the usefulness of the proposed approach. The results show that on average the size of the impact after a single modification (a change in a one EFSM transition) ranges between 14 and 38 % of the total size of the model. For a modification involving multiple transitions, the average size of the impact ranges between 30 and 64 % of the total size of the model. Additionally, we investigated the relationships (correlation) between the structure of the EFSM model, and the size of the impact sets. Upon preliminary analysis of the correlation, the concepts of model density and data density were defined, and it was found that they could be the major factors influencing the sizes of impact sets for models. As a result, these factors can be used to determine the types of models for which the proposed approach is the most appropriate.

System reliability has always been a challenging issue for many systems. In order to achieve high reliability, redundancy and voting schemes are often used to tolerate unintentional component failures. For unintentional failures caused by, for instance, normal wear-outs, hardware failures, or software bugs, etc., adding more redundancies often improves a system's reliability. However, when attack-caused failures exist, the number of redundant components and the number of participating voting entities may not be positively proportional to system reliability. In this paper, we study system reliability and system defense strategies when the system is under rational attacks. In particular, we analyze how defense and attack strategies may impact system reliability when both the defender and attacker are given a fixed amount of resources that can only be used for adding camouflaging components or enhancing existing components' cyber protection by defenders, or selecting a subset of components to attack by attackers, respectively. We also present an algorithm to decide the optimal defense strategy in fighting against rational attacks.

This is the second of two volumes of the Special Issue on Software Testing, Verification, and Validation (STVR). Five papers successfully completed the review process. These papers are spread across two special issues of STVR. The previous issue of STVR includes the first three papers. This issue includes the remaining two papers. The rest of this editorial provides a brief overview of these two papers. The first paper, ’Checked Coverage: An Indicator for Oracle Quality’ by David Schuler and Andreas Zeller, proposes an approach to assess oracle quality. The concept of checked coverage is introduced – the dynamic slice of covered statements. The results of the experimental study suggest that the checked coverage is a good indicator for oracle quality, The second paper, ’Handling Test Length Bloat’ by Gordon Fraser and Andrea Arcuri, investigates the effect of test length and bloat in the context of testing object-oriented software, where test cases are sequences of method calls. The bloat occurs when the test length abnormally grows over time during search-based test generation. Techniques to overcome the length bloat are presented. The results of an experimental study demonstrate that these techniques can improve the performance of the search process. In conclusion, we would like to thank the authors of this and the previous special issue for preparing the extended versions of the ICST-2011 conference papers. We are particularly thankful to all reviewers for timely and constructive reviews. Finally, we hope that this special issue will be stimulating for many practitioners and researchers and will help in advancing the knowledge and research in software testing, verification, and validation.

Many long-running computer systems record events as they execute, resulting in a dynamic record of system behavior. In large systems, the event trace may contain thousands of entries and when faced with a problem for analysis, programmers must sort through many disparate events to find those that are related to the system behavior under study and eliminate those that are not. In this research we investigated automatic reduction of event traces to reduce the volume of events and assist in analysis of behavior of large systems. Our approach was to adapt the techniques used in program slicing to compute event trace slices as a means of reduction. Two methods for slicing of event traces were proposed and investigated. The Event Dependence Based method (EDB) uses information available in the event trace to identify dependencies between events and to compute an event trace slice that meets a slicing criterion. The Model Dependence Based method (MDB) incorporates the use of an executable state-based system model to achieve further reduction of traces. The method identifies model-based dependences in the trace to compute trace slices. An experimental study was performed on simulated systems, representative of state-based software systems present in industry to analyze and compare the EDB and MDB slicing methods. Both methods provided significant reduction of event traces, particularly for systems with a low degree of sharing and interaction among resources. However, the MDB method significantly outperformed the EDB method for systems with a high degree of resource sharing.

SUMMARY During regression testing, a modified system is often retested using an existing test suite. Since the size of the test suite may be very large, testers are interested in detecting faults in the modified system as early as possible during this retesting process. Test prioritization attempts to order tests for execution so that the chances of early detection of faults during retesting are increased. The existing prioritization methods are based on the source code of the system under test. In this paper, we present and evaluate two model‐based selective methods and a dependence‐based method of test prioritization utilizing the state‐based model of the system under test. These methods assume that the modifications are made both on the system under test and its model. The existing test suite is executed on the system model and information about this execution is used to prioritize tests. Execution of the model is inexpensive as compared with execution of the system under test; therefore, the overhead associated with test prioritization is relatively small. In addition, we present an analytical framework for evaluation of test prioritization methods. This framework may reduce the cost of evaluation as compared with the framework that is based on observation. We have performed an empirical study in which we compared different test prioritization methods. The results of the empirical study suggest that system models may improve the effectiveness of test prioritization with respect to early fault detection. Copyright © 2011 John Wiley & Sons, Ltd.

Mission critical systems often operate for limit time durations. For these systems, we subscribe to the notion that provisioning of security can be based on the expected duration of a system's mission. In this paper, we present a simple and safe K-variant approach to improve time-based mission critical systems' attack-survivability and provide formal analysis about K-variant system's attack survivability under M memory-based attack attempts. Our theoretical analysis supported by extensive simulations and a case study provide good evidences that the proposed approach may be in improving system's attack-survivability.

Source-code based test data generation is a process of finding program input on which a selected element, e.g., a target statement, is executed. There exist many test generation methods that automatically find a solution to the test generation problem. The existing methods work well for many programs. However, they may fail or are inefficient for programs with complex logic and intricate relationships between program elements. In this thesis we present a testability transformation that transforms programs so that the chances of finding a solution are increased when the existing methods fail using only the original program. In our approach data dependence analysis is used to identify statements in the program that affect computation of the fitness function associated with the target statement. The transformed program contains only these statements, and it is used to explore different ways the fitness function may be computed. These explorations are inexpensive when using the transformed program as compared to explorations using the original program. As a result, executions in the transformed program that lead to the evaluation of the fitness function to the target value are identified. The identified executions are then used to guide the search in the original program to find an input on which the target statement is executed.

During regression testing, a modified system needs to be retested using the existing test suite. Since test suites may be very large, developers are interested in detecting faults in the system as early as possible. Test prioritization orders test cases for execution to increase potentially the chances of early fault detection during retesting. Code-based test prioritization methods are based on the source code of the system, whereas model-based test prioritization methods are based on system models. System modeling is a widely used technique to model state-based systems. Models can be used not only during software development but also during testing. In this paper, we briefly overview code-based and model-based test prioritization. In addition, we present an experimental study in which the code based test prioritization and the model-based test prioritization are compared.

Assertions are used to detect incorrect program behavior during testing and debugging. Assertions when combined with automated test data generation may increase the confidence that certain types of faults are not present in the program. If the test data generation process is not able to violate an assertion, a developer may have confidence that the fault "captured" by the assertion is not present. We refer to this process as an assertion-based validation. Assertion-based validation may be very expensive especially when a large number of assertions are present in a program. During maintenance, after a modification is made to the program, all unchanged assertions need to be revalidated to make sure that certain types of faults are present. In this paper we present an approach that may reduce the cost of assertion-based revalidation after modifications are made to the program by identifying assertions that need to be revalidated or only partially revalidated. The presented approach is based on program dependence analysis and testability transformation. The results of a small case study indicate that the presented approach may significantly reduce the effort during the process of assertion-based revalidation.