Non-deterministically behaving test cases cause developers to lose trust in their regression test suites and to eventually ignore failures. Detecting flaky tests is therefore a crucial task in maintaining code quality, as it builds the necessary foundation for any form of systematic response to flakiness, such as test quarantining or automated debugging. Previous research has proposed various methods to detect flakiness, but when trying to deploy these in an industrial context, their reliance on instrumentation, test reruns, or language-specific artifacts was inhibitive. In this paper, we therefore investigate the prediction of flaky tests without such requirements on the underlying programming language, CI, build or test execution framework. Instead, we rely only on the most commonly available artifacts, namely the tests' outcomes and durations, as well as basic information about the code evolution to build predictive models capable of detecting flakiness. Furthermore, our approach does not require additional reruns, since it gathers this data from existing test executions. We trained several established classifiers on the suggested features and evaluated their performance on a large-scale industrial software system, from which we collected a data set of 100 flaky and 100 non-flaky test- and code-histories. The best model was able to achieve an F1-score of 95.5% using only 3 features: the tests' flip rates, the number of changes to source files in the last 54 days, as well as the number of changed files in the most recent pull request.

Mutation testing can be used to measure the quality of a given test suite. But two flaws prevent wide acceptance. First, there are equivalent mutants - mutants that are semantically equivalent to the unmodified version and therefore unkillable. Manually identifying those mutants is time-consuming and error-prone. Second, initially there are often too few test cases. Mutation testing detects missing cases. But it is too time-consuming to manually write all the required test cases. This paper shows how to use symbolic execution to tackle both problems, i.e., to detect equivalent mutants and exclude them from further analysis, and to automatically generate test cases that kill the remaining mutants. Our evaluation uses a set of 252 publicly available mutants for which it is known that they are hard to classify. Despite the fact that detecting equivalent mutants is an undecidable problem in general, our fully automatic tool MutantDistiller correctly classifies all of them (13 equivalent, 239 non-equivalent). MutantDistiller also generates test cases that kill the non-equivalent mutants.

Structural (manual or automated) testing today often overlooks typical pro- gramming faults because of inherent flaws in the simple criteria applied (e.g. branch or all-uses). Dedicated testing strategies that address such faults (e.g. mutation testing) are not specifically designed for smart automatic test case generation. In this paper we present a new coverage criterion and its implementation that accomplishes both: it detects more faults and integrates easily into automated test case generation. The criterion is targeted towards unveiling faults that originate from shifts in the equiva- lence classes that are caused by small coding errors (inspired by mutation testing). On benchmark codes from the Java-API and from an open-source project we improve the fault detection capability by up to 41% compared to branch and all-use coverage.

Token Expression OperatorPriorityLev el AbstractRuntimeEnvironment StandaloneRuntimeEnv ironment RuntimeEnv ironment extends extends extends extends Figure 2.6: Primary classes in Afra Expression Parser The whole procedure of using Afra Expression Parser in AFRA CASE is described in stages as follows: • Importing UML diagrams are imported with the help of Eclipse Modeling Framework (EMF) into the program AFRA CASE. Expressions in UML diagrams are read as strings into the class Expression, for example an Afra Script expression 3*(a+2) will be read as a string into the Afra Expression Parser. • Lexing The class OperatorPriorityLevel plays the roll of lexical analysis. That defines the tokens of all the operands and operators which are used in the Afra Script expressions. OperatorPriorityLevel defines also the priority of the operands and operators in various levels. According to the priority levels, the Afra Script expression 3*(a+2) will be represented as * (Number 3) (+ (Variable a) (Number 2)). CHAPTER 2 EXTENSION 15 • Parsing Afra Expression Parser parses the tokens after the process lexing. At the same time of parsing an Afra Script expression will be built in a syntax tree and ordered from left to right. The syntax tree according to * (Number 3) (+ (Variable a) (Number 2)) looks like in figure 2.7. The class AbstrackToken in Afra Expression Parser memorizes the interim results of parsing and the syntax trees of expressions. Figure 2.7: Syntax tree of the expression 3*(a+2) • Checking data types Afra Expression Parser supports the primitive data types in Java, such as boolean, int, float, string etc. AFRA GmbH has advanced an own data type system which is used to define UML objects in a number of data types, such as UnITeDType, UnITeDInstanceType, UnITeDValueType, etc. AFRA CASE uses such data types to define and check the correctness of expression semantics through accessing the class RuntimeEnvironment which is shown in the left of figure 2.6. After checking the correctness of semantics the Afra Script expressions are further represented and stored in Afra Expression objects. • Evaluating An Afra Script expression can be once parsed and then evaluated as often as necessary. AFRA CASE generates the value for the variables in expressions, and the expressions are then evaluated. Figure 2.8 illustrates the complete procedure of current Afra Expression Parser that is extracted to show the work flow of using Afra Expression Parser so far in AFRA CASE. Afra Expression Parser reads the Afra Script expressions from all the UML objects and parses them into the syntax trees which are then encapsulated in the Afra Expression Objects and finally used as parameters in the genetic algorithm to evaluate the values for fitness. CHAPTER 2 EXTENSION 16 Figure 2.8: Procedure of current Afra Expression Parser 2.2 Alternatives for the extension This section describes primarily the extension of Afra Expression Parser named Python Expression Parser which can parse the Python expressions in the UML diagrams. Figure 2.9 shows the procedure of the Python Expression Parser that will be extended in the program AFRA CASE. Three alternatives are described in the following paragraph how a Python parser in AFRA CASE works. 1. Direct translator Idea of a direct translator is to generate temporary Afra Script expressions (e.g. with ANTLR) from Python expressions directly. The translated Afra Script expressions are then further parsed in the original Afra Expression Parser module which could be reused as much as possible in this alternative. CHAPTER 2 EXTENSION 17 2. Creating dedicated parser with ANTLR Idea of this alternative is to create dedicated parser with a parser generator ANTLR and further using this dedicated parser to parse the Python expressions as what the Afra Expression Parser did, finally evaluating the parsed Python expressions dynamically. 3. Using Jython Another possibility is to reference Jython, an implementation of Python in Java, in the program AFRA CASE, further using Jython to parse and evaluate the Python expressions in AFRA CASE. Figure 2.9: Procedure of the extended Python Expression Parser 2.2.1 Alternative 1: Direct translator Direct translator focuses on the stage of lexical analysis in language compiler. The purpose of direct translation from Python expressions to Afra Script expressions is to reuse the original source code of current Afra Expression Parser as much as possible. CHAPTER 2 EXTENSION 18 A translator means a program in a computer language that translates source code into some other represented code. In order to build a direct translator from Python expressions to Afra Script expressions the difference of syntax between the Python expressions and the Afra Script expressions must be found for the process of translating. Afra Script expression is a C-like scripting language that defines just the primary data types for the expressions. Python is an objectoriented, extensible programming language that has much richer syntax than the Afra Script expression. Figure 2.10: Work flow of direct translator CHAPTER 2 EXTENSION 19 The process of translating from Python expressions to Afra Script expressions needs the help with parsing procedure of Python expression (e.g. using ANTLR) and generating the temporary Afra Script expressions, and then the original module Afra Expression Parser will be reused to parse and evaluate the expressions as what it did. It is seemly easier to develop for this direct translator, but the low extension problems of development will come in the future which will be discussed in the following paragraph “Disadvantages of direct translator” below. Figure 2.10 illustrates the work flow of direct translator. Advantages of direct translator • Easy to implement Direct translator is easy to implement in AFRA CASE. It will need just a new component Python Expression Parser which will be inserted as a translator module before the Afra Expression Parser in AFRA CASE. • Saving time and labour of implementation Direct translator can save time and labour of implementation. The Python lexer can be freely available from the internet. And the translator needs just a program to restrict the used Python token types on the Afra token types and translate the different operator representations from Python to Afra Script expression. • Code reused as much as possible Direct translator can reduce the cost of rewriting the source code of AFRA CASE as much as possible. It changes just the stage of lexical analysis and reuses further the Afra Expression Parser module to parse the expressions. Disadvantages of direct translator • 1:1 conversion Direct translator is an obvious 1:1 translator. It depends highly on the extended language syntax. That means, much more various expressions of Python could not be translated into the Afra Script expressions correctly which could be understood further by Afra Expression Parser. • Low extension The purpose of this work is to extend the Afra Expression Parser, i.e. to support the diversity of parsed expressions from UML models into AFRA CASE, but direct translator cannot extend the Afra Expression Parser since Afra Expression Parser is still reused. CHAPTER 2 EXTENSION 20 2.2.2 Alternative 2: Creating dedicated parser with ANTLR Idea of this alternative bases on the stages of lexical analysis and syntactical analysis in language compiler and the evaluating procedure in AFRA CASE. For the stage of lexical analysis it will be required a lexer and for syntactical analysis required a parser with which ANTLR can deal effectively. ANTLR is the acronym for “Another Tool for Language Recognition” that has been developed since 1989 by Terence Parr, a professor of computer science at the University of San Francisco. The information about ANTLR in this work bases on the book [24] of Terence Parr. “ANTLR is a parser generator: a program that generates code to translate a specified input language into a nice, tidy data structure.” -Terence Parr ANTLR can generate the lexer, the tokens and the parser according to the grammar of the input language. The input language in this case is Python. A grammar is a formal, text based language specification, which describes the syntax of a language, or in other words, which describes a language, what it looks like. Grammar of a widely used programming language can be freely available from the internet. Python grammar has been completed by Terence Parr and Loring Craymer since 2004, which is also a free source file in the internet. Figure 2.11 shows the work flow of creating dedicated lexer and dedicated parser using the tool ANTLR and the Python grammar. The main processes of using ANTLR are described as follows: • Executing Python grammar Python grammar is a text based specification described in the file Python.g. The step is adding the file Python.g in ANTLR and executing ANTLR plugin on the Python grammar. • Creating the Python lexer With the help of Python grammar in the file Python.g ANTLR generates in the program a dedicated lexer for Python, which is written in Java classes. The generated file is PythonLexer.java. • Generating the Python tokens With the help of Python grammar ANTLR generates also the tokens for Python in the program. The file Python.tokens contains the list of token-name and token-type assignments. CHAPTER 2 EXTENSION 21 • Creating the Python parser With the help of Python grammar ANTLR generates at the same time the dedicated parser for Python. The generated file is PythonParser.java, in which declares a number of methods for every rule defined in the Python grammar. • Building Python ASTs ANTLR's grammar offers a set of options, such as language, output, rewrite, etc. The output option controls the generated data structure. The Python expressions can be outputted from the Python parser in the form of abstract syntax trees (ASTs). Using output = AST in the Python grammar allows to use tree construction operators. It mus

This article presents two different tools automating the generation of optimized test data for unit, model-based and integration testing by maximizing the coverage and minimizing the number of test cases required. To cope with these conflicting goals, hybrid self-adaptive and multi-objective evolutionary algorithms were applied. The efficiency was demonstrated by evaluating fault detection capability by mutation testing. Thanks to the effort reduction offered, the approach is particularly suitable for the verification of complex, safety-relevant software systems.

The importance of software in nearly all of today's engineering disciplines demands for development and validation techniques ensuring high dependability of complex software systems. Component-based software development as the ultimate approach for dealing with complexity shifts the focus of verification from unit to integration tests addressing the correctness of component interactions. As the current state-of-the-art does not include a systematic and tool-supported approach to interface testing, this paper presents a procedure for the automatic generation of integration test data based on genetic algorithms.

Mit der immer groseren Verbreitung modellgetriebener Softwareent- wicklung kommt dem modellbasierten Test eine immer wichtigere Rolle zu. Dieser Artikel prasentiert einen neuen Ansatz zur Optimierung des Regressionstests mit- tels automatischer Erkennung wieder verwendbarer Testfalle und automatischer Generierung einer minimalen Anzahl zusatzlich zu uberprufender Ablaufe.

Kurzfassung Nach einer einführenden Klassifikation wesentlicher Softwarequalitätsmerkmale und zuverlässigkeitserhöhender Maßnahmen plädiert dieser Beitrag für einen intensiven Einsatz struktureller Testverfahren und bietet hierzu neuartige, vollautomatische Unterstützung an. Insbesondere werden zwei Werkzeuge vorgestellt, die mittels multikriterieller Heuristiken sowohl auf Code- als auch auf Modellebene mit möglichst wenigen, automatisch generierten Testfällen (inklusive zugehöriger Daten) eine möglichst hohe Testobjektüberdeckung zu erzielen erlauben. Dadurch können unterschiedliche Testphasen (Komponententest, Integrationstest, Systemtest) weit über den Stand der Technik hinaus automatisch und kosteneffizient unterstützt werden. Abschließend illustriert der Artikel auch den Einsatz neuerer Verfahren zum quantitativen Nachweis erzielter Zuverlässigkeitskenngrößen und empfiehlt, die Auswertung der mit wiederverwendbarer Software bereits gewonnenen Betriebserfahrung beim statistischen Zuverlässigkeitsnachweis mit zu berücksichtigen.

In this article, the problem of assessing software trustworthiness is considered from a holistic perspective addressing both safety- and security-critical application domains. In particular, the importance of achieving high structural coverage during component and integration testing phases is stressed. In view of the immense effort required by manual testing activities, the present article suggests novel automatic test case generation techniques, capable of maximizing test coverage and minimizing test amount. The tools developed on the basis of these approaches were successfully applied to achieve high control flow, data flow and interface coverage by means of a low number of test cases.

Novel heterocyclic derivatives of (4-aryloxymethyl-1,3-dioxolan-2-yl)methyl-1H-imidazoles and 1H-1,2,4-triazoles, useful as antifungal and antibacterial agents.

Even assuming exhaustive tests at component level, interaction faults can only be avoided by thorough testing of component interfaces during integration testing. In view of the relevance of the integration testing phase for modern componentübased systems, this article presents an extensive set of interface coverage criteria, which are not restricted to mere operation call sequences, but include messageübased as well as stateübased information. In order to support rational decisionümaking in identifying the best affordable integration coverage criterion for a given industrial system, a tool was implemented providing conservative estimates of the testing effort expected to be required by a given stateübased integration testing strategy considered, supporting in addition the automatic visualisation of the interface entities needed to be covered.

Modellbasierte Testfallgenerierungsansatze sind inzwischen zwar weit verbreitet, meist jedoch auf die Erzeugung von Testszenarien beschrankt. Im Allgemeinen ist allerdings daruber hinaus ein nicht unerheblicher manueller Aufwand zur Ermittlung zugehoriger Eingabedaten notwendig. Dieser Artikel prasentiert ein Verfahren, das die vollautomatische Generierung vollstandiger Testfallinformation aus Zustandsmaschinen ermoglicht und auf evolutionaren Algorithmen sowie Modellsimulation basiert. Erste experimentelle Erfahrungen bei der Anwendung dieses Verfahrens werden berichtet.

In this work a technique for the automated generation of test data is presented, which can be equally applied to both procedural and object-oriented sof tware. During the generation the test cases are optimised in a way to achieve maximised structural code coverage with a minimised number of test cases. In order to cope with these inherently c onfli tive goals, self-adaptive multiobjective metaheuristics (among others evolutionary algo rithms) are applied. The approach is based on a preliminary phase comprising an automatic instru mentation of the source code, aiming at recording relevant information about controlflow an d dataflow during runtime. Using the insight gained hereby, the test sets are successively impro ved until the given testing goals have been reached. In a concluding phase the quality of the genera t d test data is assessed in terms of its fault detection capability by means of mutation testi ng. Additionally, the actual coverage (expressed as percentage of the entities to be covered) is de termined by means of static analysis of the controlflow and the dataflow. The technique presented h ere allows to considerably reduce the effort required for verification and validation of compl ex, safety-critical software. The document is structured in seven chapters, followed by se veral appendices. In the beginning the goals of the research project, the result of which is presented here, are motivated in chapter 1. In the following chapter 2 the technique and its in tended purpose are classified according to the software development process and in the context o f preliminary work. In chapter 3 the basics of structural testing strategies supported by th e presented technique are introduced, while in chapter 4 the multi-objective metaheuristics appl ied are outlined. Subsequently, both topics are brought together in chapter 5, where the (two-ste p) procedure is described in detail. In chapter 6 experimental results are presented and discussed , as gained by means of a prototypical implementation of the technique in a tool named •gEAr for the programming language J AVA TM. Finally, chapter 7 gives an outlook on possible extensions t the approach presented.

This paper presents a technique for automated test data generation applicable to both procedural and object-oriented programs. During the generation, the test cases are optimised such as to maximise structural code coverage by minimising at the same time the number of test cases required. To cope with these two inherently conflicting goals, hybrid self-adaptive and multi-objective evolutionary algorithms are applied. Our approach is based on a preliminary activity that provides support for the automatic instrumentation of source code in order to record the relevant data flow information at runtime. By exclusively utilising the insight gained hereby, test data sets are successively enhanced towards the goals mentioned above. Finally, the efficiency of the test set generated is evaluated in terms of its fault detection capability by means of mutation testing. In addition, the actual coverage percentage achieved is determined by taking into account the results of a static data flow analysis of the system under test. Thanks to the dramatic decrease of effort required for generating and verifying test cases, the technique presented here allows to substantially improve the V&V-phase of complex, safety-relevant software. Preliminary experimental results gained so far are reported in the paper.

In this research paper, an approach to fully automating the generation of test data for object-oriented programs fulfilling dataflow-based testing criteria and the subsequent evaluation of its fault-detection capability are presented. The underlying aim of the generation is twofold: to achieve a given dataflow coverage measure and to minimize the effort to reach this goal in terms of the number of test cases required. In order to solve the inherent conflict of this task, hybrid self-adaptive and multiobjective evolutionary algorithms are adopted. Our approach comprises the following steps: a preliminary activity provides support for the automatic instrumentation of source code in order to record the relevant dataflow information. Based on the insight gained hereby, test data sets are continuously enhanced towards the goals mentioned above. Afterwards, the generated test set is evaluated by means of mutation testing. Progress achieved so far in our ongoing project will be described in this paper.

In this paper we present an approach to an automated support for generating test cases satisfying data flow criteria for testing object-oriented software. We describe our implementation of the source code instrumentation tool needed therefor. After an overview of the main data flow coverage criteria, we motivate why data flow testing should be done in practice pinpointing at some fault scenarios detectable by data flow coverage and reveal some pitfalls of the latter.

Hybrid Genetic Algorithms apply so called hybrid or repair operators or include problem specific knowledge about the problem domain in their mutation and crossover operators. These operators use local search to repair or avoid illegal or unsuitable assignments or just to improve the quality of the solutions already found.Those Hybrid Genetic Algorithms have been successfully applied to different constraint satisfaction and timetabling problems such as the travelling salesman problem, scheduling problems, employee timetabling or high school timetabling.In this paper we describe a Genetic Algorithm for solving the German school timetabling problem. The Genetic Algorithm uses direct representation of the problem and applies an adapted mutation operator as well as several specific repair operators. We redecode the computed improvements to the genotype which establishes a kind of Lamarckian evolution.One of the problems utilising these hybrid operators is how and when to apply them, i.e. how to set the parameters right to achieve the best results. Different approaches have been started to adjust these parameters in an optimal way, but in most cases these adjustments require additional computing time and consequently are quite costly. We tackled this problem by an adaptation mechanism for the repair operators which can be applied without additional computing time. These operators are switched on when the normal Genetic Algorithm does not yield any more improvements. When the Genetic Algorithm then converges again, a reconfiguration step for the operator parameters guides the search out of the local optimum.