Testing plays an important role in the context of software migration as it is used to validate and ensure functional equivalence as a key requirement. As developing new test cases of the migrated system is typically a costly and time-consuming activity, migrating existing test cases for the old system is thus an attractive alternative. Considering that migrated test cases are relied upon to validate an implemented system migration, it is clear that validating the corresponding test case migration is indeed crucial. The solution space involved in validating test case migration, however, is currently not well researched. In this paper, therefore, we analyze the application of mutation analysis as a validation technique for test case migration. Depending on what is mutated, we identify six scenarios which can be used as mutation patterns depending on the situation and the user’s requirements. For each scenario, we provide a discussion of implied assumptions, indications, and limitations in each case. We also present an initial evaluation of the identified mutation analysis scenarios performed in a real-world migration setting.
Traditionally, people abstract away the infrastructure operation, such as power management, network traffic engineering and even the "cloud computing"? layers from software developers. This abstraction brings easier application development and maintenance but leads to complexities and inefficiencies for infrastructure operation. In this talk, we present our recent research into this area. Using data center power management, networking as well as cloud software management as examples.
In recent years, the application of machine learning techniques to software testing has been an active research area. Among the most notable work reported in the literature are those experiments on the uses of supervised and semi-supervised learning techniques to develop test oracles so that the correctness of software outputs and behaviours on new test cases can be predicated. Experiment data show that it seems a promising approach to the test oracle automation problem. In general, software testing is an inductive inference in the course of which the tester attempts to deduce general properties of a software system by observing the behaviours of the system on a finite number of test cases. This talk discusses the theoretical foundation of software testing from the perspective of computational machine learning theories.
Page Object is a design pattern aimed at making web test scripts more readable, robust and maintainable. The effort to manually create the page objects needed for a web application may be substantial and unfortunately existing tools do not help web developers in such task. In this paper we present APOGEN, a tool for the automatic generation of page objects for web applications. Our tool automatically derives a testing model by reverse engineering the target web application and uses a combination of dynamic and static analysis to generate Java page objects for the popular Selenium WebDriver framework. Our preliminary evaluation shows that it is possible to use around 3/4 of the automatic page object methods as they are, while the remaining 1/4 need only minor modifications.
The test oracle problem is regarded as one of the most challenging problems in software testing. Metamorphic testing has been developed to alleviate this problem, which is done using the relations involving relevant inputs and their outputs. This keynote speech will provide a summary of the state-of-the-art of metamorphic testing.
In recent years, software testing research has produced notable advances in the area of automated test data generation, but the corresponding oracle problem (a mechanism for determine the (in)correctness of an executed test case) is still a major problem. In this paper, we present a preliminary study which investigates the application of anomaly detection techniques (based on clustering) to automatically build an oracle using a system's input/output pairs, based on the hypothesis that failures will tend to group into small clusters. The fault detection capability of the approach is evaluated on two systems and the findings reveal that failing outputs do indeed tend to congregate in small clusters, suggesting that the approach is feasible and has the potential to reduce by an order of magnitude the numbers of outputs that would need to be manually examined following a test run.
Testing of Web service orchestrations brings new challenges to developers of Service-Oriented Applications (SOA) due to their dynamic nature and missing control over underlying infrastructure. The current testing approaches and tools cover single testing activities such as test path analysis, test case generation, Web service emulation, fault injection, etc. There is still a lack of solutions that provide complete testing of such applications. In order to fill this gap, this paper proposes a framework, called Testing as a Service Software Architecture (TASSA), for testing web service orchestrations, described with Business Process Execution Language for Web Services (WS-BPEL). It automates a complete set of activities for design time testing of functional and non-functional behavior of composed Web services. The feasibility of the framework is proved through a case study that demonstrates its features while testing a sample business process.
In this paper we investigate the effect of test case length on the performance of test suites for automated graphical user interface (GUI) testing. Research has been done on the effect of test length for traditional white-box testing; however, these findings do not necessarily extend to GUI testing. We run experiments on five subject applications using a custom-built dynamic test generator. We measure the effect that test length has on statement, branch, and event-pair coverage, as well as the fault-finding abilities for each test suite. Our findings indicate that longer test suites perform better than shorter test suites in terms of coverage, fault-finding, and cost. However, the effect eventually diminishes with greater increase in length. As well, the results vary depending on the application under test.
In the context of structural testing, automatic test-pattern generation (ATPG) may fail to provide suites covering 100% of the testing requirements for grey-box programs, i.e., applications wherein source code is available for some parts (white-box), but not for others (black-box). Furthermore, test suites based on abstract models may elicit behaviors on the actual program that diverge from the intended ones. In this paper, we present a new ATPG methodology to reduce divergence without increasing manual effort. This is achieved by ( i ) learning models of black-box components as finite-state machines, and ( ii ) composing the learnt models with the white-box components to generate test-suites for the grey-box program. Experiments with a prototypical implementation of our methodology show that it yields measurable improvements over two comparable state-of-the-art solutions.
Dynamic symbolic execution or concolic testing has been proposed recently to effectively generate test inputs for real-world programs. Unfortunately, dynamic symbolic execution techniques do not scale well for large realistic programs, because often the number of feasible execution paths of a program increases exponentially with the increase in the length of an execution path. We believe that programmers usually have a good understanding of programs they write and could help concolic testing to guide path exploration effectively so that concolic testing can achieve the desired testing goal quickly. In this paper, we propose a simple annotation mechanism, called Guide SE, that enables a programmer to guide concolic testing by annotating the program under test. We have currently identified three mechanisms for providing these annotations: control annotations, equivalence annotations, and data annotations. Controls annotations allow a programmer to specify the paths of interest using regular expressions. Equivalence annotations enable a programmer to prune paths that result in user defined equivalent states. Data annotations allow a programmer to abstract portions of the program under test and to explore the reduced path space of the abstract program.
Detection of infeasible paths is required in many areas including test coverage analysis, test case generation, security vulnerability analysis, etc. Existing approaches typically use static analysis coupled with symbolic evaluation, heuristics, or path-pattern analysis. This paper is related to these approaches but with a different objective. It is to analyze code of real systems to build patterns of unsatisfiable constraints in infeasible paths. The resulting patterns can be used to detect infeasible paths without the use of constraint solver and evaluation of function calls involved, thus improving scalability. The patterns can be built gradually. Evaluation of the proposed approach shows promising results.
Multithreaded programs are subject to data races. Data race detectors find such defects by static or dynamic inspection of the program. Current race detectors suffer from high numbers of false positives, slowdown, and false negatives. Because of these disadvantages, recent approaches reduce the false positive rate and the runtime overhead by applying race detection only on a subset of the whole program. To achieve this, they make use of parallel test cases, but this has other disadvantages: Parallel test cases have to be engineered manually, cover code regions that are affected by data races, and execute with input data that provoke the data races. This paper introduces an approach that does not need additional parallel use cases to be engineered. Instead, we take conventional unit tests as input and automatically generate parallel test cases, execution contexts and input data. As can be observed, most real-world software projects nowadays have high test coverages, so a large information base as input for our approach is already available. We analyze and reuse input data, initialization code, and mock objects that conventional unit tests already contain. With this information, no further oracles are necessary for generating parallel test cases. Instead, we reuse the knowledge that is already implicitly available in conventional unit tests. We implemented our parallel test case generation strategy in a tool called TestMerge. To evaluate these test cases we used them as input for the dynamic race detector CHESS that evokes all possible thread interleavings for a given program. We evaluated TestMerge using six sample programs and one industrial application with a high test case coverage of over 94%. For this benchmark, TestMerge identified all previously known data races and even revealed previously unknown ones.
Software testability is the degree of difficulty to test a program. Code visibility is important to support design principles, such as information hiding. It is widely believed that code visibility has effects on testability. However, little empirical evidence has been shown to clarify whether and how software testability is influenced by code visibility. We have performed an empirical study to shed light on this problem. Our study focuses on test code coverage, in particular that of automatic testing tools. Code coverage is commonly used for various purposes, such as evaluating test adequacy, assessing test quality, and analyzing testability. Our study uses code coverage as the concrete measurement of testability. By analyzing code coverage of two state-of-the-art tools, in comparison with that of developer-written tests, we have discovered that code visibility does not necessarily have effects on its code coverage, but significantly affects automatic testing tools. Low code visibility often leads to low code coverage for automatic tools. In addition, different treatments on code visibility can result in significant differences in overall code coverage for automatic tools. Using a tool enhancement specific to code visibility, we demonstrate the great potential to improve existing tools.
Buffer overflow exploits form a substantial portion of input manipulation attacks as they are commonly found and are easy to exploit. Despite existence of many detection solutions, buffer overflow bugs are widely being reported in multitude of applications suggesting either inherent limitations in current solutions or problems with their adoption by the end-users. To address this, we propose a novel light-weight rule-based test case generation approach for detecting buffer overflows. The proposed approach uses information collected from static program analysis and pre-defined rules to generate test cases. Since the proposed approach uses only static analysis information and does not involve any constraint solving it is termed as light-weight. Our experimental evaluation on benchmark programs shows that the test inputs generated by the proposed approach are effective in detecting known bugs along with reporting some new bugs.
Metamorphic testing is an advanced technique to test programs without a true test oracle such as machine learning applications. Because these programs have no general oracle to identify their correctness, traditional testing techniques such as unit testing may not be helpful for developers to detect potential bugs. This paper presents a novel system, KABU, which can dynamically infer properties of methods' states in programs that describe the characteristics of a method before and after transforming its input. These Metamorphic Properties (MPs) are pivotal to detecting potential bugs in programs without test oracles, but most previous work relies solely on human effort to identify them and only considers MPs between input parameters and output result (return value) of a program or method. This paper also proposes a testing concept, Metamorphic Differential Testing (MDT). By detecting different sets of MPs between different versions for the same method, KABU reports potential bugs for human review. We have performed a preliminary evaluation of KABU by comparing the MPs detected by humans with the MPs detected by KABU. Our preliminary results are promising: KABU can find more MPs than human developers, and MDT is effective at detecting function changes in methods.
Graphical User Interface (GUI) application is a kind of typical event-driven software (EDS) that transforms state according to input events invoked through a user interface. It is time consuming to test a GUI application since there are a large number of possible event sequences generated by the permutations and combinations of user operations. Although some GUI test case prioritization techniques have been proposed to determine "which test case to execute next" for early fault detection, most of them use random ordering to break tie cases, which has been proved to be ineffective. Recent research presents the opinion that using hybrid criteria can be an effective way for tie-breaking, but few studies focus on seeking a new criterion cooperating well with other criteria when breaking tie cases. In this paper, we propose a state-distance-based method using state coverage as a new criterion to prioritize GUI test cases. An empirical study on three GUI programs reveals that the state-distance-based method is really suitable for GUI test case prioritization and can cooperate well with the (additional) event length criterion.
In software engineering, error impact analysis consists in predicting the software elements (e.g. Modules, classes, methods) potentially impacted by a change. Impact analysis is required to optimize the testing effort. In this paper we present a new protocol to analyze the accuracy of impact analysis. This protocol uses mutation testing to simulate changes that introduce errors. To this end, we introduce a variant of call graphs we name the "use graph" of a software which may be computed efficiently. We apply this protocol to two open-source projects and correctly predict the impact of 30% to 49% of changes.
Fault Injection (FI) is an established testing technique to assess the fault-tolerance of computer systems. FI tests are usually highly automated for efficiency and to prevent human error from affecting result reliability. Most existing FI automation tools have been built for a specific application domain, i.e., A certain system under test (SUT) and fault types to test the SUT against, which significantly restricts their reusability. To improve reusability, generalist fault injection tools have been developed to decouple SUT-independent functionality from SUT-specific code. Unfortunately, existing generalist tools often embed subtle and implicit assumptions about the target system that affect their reusability. Furthermore, no assessments have been conducted how much effort the SUT-specific adaptation of generalist tools entails in comparison to reimplementation from scratch. In this paper, we present GRINDER, an open-source, highly-reusable FI tool, and report on its applicability in two very different systems (the Android OS in an emulated environment, and a real-time AUTOSAR system) under four different FI scenarios.
Instrumentation and monitoring plays an important role in measurement-based performance analysis of software systems. However, in practice the performance overhead of extensive instrumentation is not negligible. Experiment-based performance analysis overcomes this problem through a series of experiments on selectively instrumented code, but requires additional manual effort to adjust required instrumentation and hence introduces additional costs. Automating the experiments and selective instrumentation can massively reduce the costs of performance analysis. Such automation, however, requires the capability of dynamically adapting instrumentation instructions. In this paper, we address this issue by introducing AIM, a novel instrumentation and monitoring approach for automated software performance analysis. We apply AIM to automate derivation of resource demands for an architectural performance model, showing that adaptable instrumentation leads to more accurate measurements compared to existing monitoring approaches.
This paper presents a research activity which is being performed with the aim of contributing to the development of an effective prognostics and health management (PHM) system for the electrohydraulic servoactuators of primary flight controls. One of the key objectives of the research work is to develop a PHM system without adding new sensors, but exploiting in the most appropriate way all information already available in the present EHSAs. This will extend the possibility of providing the PHM capability to the EHSAs of legacy aircraft and not limit the applicability of PHM to new platforms. The PHM system which is being developed is based on a neural network approach in which the health indexes defining the EHSA significant features are processed by appropriate algorithms. The neural network acts as a reasoner that performs data fusion eventually detecting and classifying an anomalous condition, if it exists. The paper outlines the research work, describing approach, methodology, structure of the PHM system, results of test cases and assessment of its effectiveness, with the evaluation of the probability of missed failures and false alarms
