Digital twins (DT) of industrial processes have become increasingly important. They aim to digitally represent the physical world to help evaluate, optimize, and predict physical processes and behaviors. Therefore, DT is a vital tool to improve production automation through digitalization and becomes more sophisticated due to rapidly evolving simulation and modeling capabilities, integration of IoT sensors with DT, and high-capacity cloud/edge computing infrastructure. However, the fidelity and reliability of DT software are essential to represent the physical world. This paper shows an automated and systematic test architecture for DT that correlates DT states with real-time sensor data from a production line in the forging industry. Our evaluation shows that the architecture can significantly accelerate the automatic DT testing process and improve its reliability. A systematic online DT testing method can significantly detect the performance shift and continuously improve the DT's fidelity. The snapshot creation methodology and testing agent architecture can be an inspiration and can be generally applicable to other industrial processes that use DT to generalize their automated testing.
HTML sanitization is an essential security feature for web services that incorporate user-generated content, removing the ability of malicious actors to perform injection attacks such as cross-site scripting. Exercising additional scrutiny when choosing such filters is prudent, as any insufficiencies in their design or implementation may lead to widespread exploitation of vulnerabilities. This work presents an approach for applying combinatorial security testing, which offers the unique capability to provide mathematically guaranteed coverage of the language defined by an attack grammar, to HTML sanitizers.
Test case prioritisation (TCP) is a critical task in regression testing to ensure quality as software evolves. Machine learning has become a common way to achieve it. In particular, learning-to-rank (LTR) algorithms provide an effective method of ordering and prioritising test cases. However, their use poses a challenge in terms of explainability, both globally at the model level and locally for particular results. Here, we present and discuss scenarios that require different explanations and how the particularities of TCP (multiple builds over time, test case and test suite variations, etc.) could influence them. We include a preliminary experiment to analyse the similarity of explanations, showing that they do not only vary depending on test case-specific predictions, but also on the relative ranks.
This paper presents a novel method for locating combinational hardware Trojans (HT) based on fault location approaches used in combinatorial testing. This method relies exclusively on the combinatorial properties of the executed test vectors and the results of test execution. Under specific assumptions, the method is guaranteed to locate all combinational HTs with trigger patterns of length $\ell $ or less, with the location process itself consuming negligible time. We give a description of our method by devising suitable algorithms and provide the links to combinatorial fault location. Furthermore, we demonstrate our approach in a concrete case study where we locate HTs embedded in a circuit that implements the AES symmetric-key encryption algorithm with 128 bits key length. In these experiments, we demonstrate how any HT that is activated by a trigger pattern of length $\ell \leq 8$ can be located in an effective way. Our method compares particularly well against randomized approaches. Although instantiated for a specific circuit in our case study, the proposed approach is generic, due to its algorithmic description, and can be applied for testing other (cryptographic) circuits. We believe that our work presents an important first step in the development of more general logic testing methodologies for HT location using combinatorial testing methods.
Mutation analysis is being extensively used for testing from a finite state machine (FSM); it consists in seeding a fault in the model using a mutation operator. There is still a lot that we need to know about mutation fault for FSM so that we can adequately use them in software testing or in software testing experiments. In this paper, we report on results from an empirical experiment during which we compared different mutation operators used to generate mutants from an FSM. We randomly generated multiple FSMs along with all the possible mutants from a relatively complete set of mutation operators. We then generated test suites using transition trees which are then executed on the FSM and its mutants to measure the mutation score for each mutation operator using different types of oracles. In doing so we report on how easy or difficult mutants generated from specific mutation operators can be. We have developed a tool that automates the whole process of this experiment.
Over the past decade, predictive language modeling for code has proven to be a valuable tool for enabling new forms of automation for developers. More recently, we have seen the ad-vent of general purpose "large language models", based on neural transformer architectures, that have been trained on massive datasets of human written text, which includes code and natural language. However, despite the demonstrated representational power of such models, interacting with them has historically been constrained to specific task settings, limiting their general applicability. Many of these limitations were recently overcome with the introduction of ChatGPT, a language model created by OpenAI and trained to operate as a conversational agent, enabling it to answer questions and respond to a wide variety of commands from end users.The introduction of models, such as ChatGPT, has already spurred fervent discussion from educators, ranging from fear that students could use these AI tools to circumvent learning, to excitement about the new types of learning opportunities that they might unlock. However, given the nascent nature of these tools, we currently lack fundamental knowledge related to how well they perform in different educational settings, and the potential promise (or danger) that they might pose to traditional forms of instruction. As such, in this paper, we examine how well ChatGPT performs when tasked with answering common questions in a popular software testing curriculum. We found that given its current capabilities, ChatGPT is able to respond to 77.5% of the questions we examined and that, of these questions, it is able to provide correct or partially correct answers in 55.6% of cases, provide correct or partially correct explanations of answers in 53.0% of cases, and that prompting the tool in a shared question context leads to a marginally higher rate of correct answers and explanations. Based on these findings, we discuss the potential promises and perils related to the use of ChatGPT by students and instructors.
Model-based testing (MBT) offers the possibility of automatic generation and execution of tests. However, it is not yet widely used in industry due to the difficulty in creating and maintaining models. On the other hand, Behavior Driven Development (BDD) is becoming more popular in the agile development process to achieve a common understanding of the system under development among stakeholders and to automate testing. However, BDD scenarios are written in human language and are usually not precise enough. Moreover, tests extracted from BDD scenarios are too short and incomplete; they only cover a very small part of the system. Our goal is to combine these two approaches to benefit from the usability of BDD and the test automation capabilities of MBT. In this paper, we first define a formal model of scenarios that we call BDD Transition Systems, second, we create more complete tests by composing scenarios (model composition), and finally, we generate and execute tests automatically. We demonstrate the applicability of this approach in a real-world example: an industrial printer.
Data is a crucial component in machine learning. However, many datasets contain sensitive information such as personally identifiable health and financial data. Access to these datasets must be restricted to avoid potential security concerns. Synthetic data generation addresses this problem by generating artificial data that are similar to, and thus could be used in place of, the original real-world data. This research introduces a synthetic data generation approach called CT-VAE that uses Combinatorial Testing (CT) and Variational Autoencoder (VAE). We first use VAE to learn the distribution of the real-world data and encode it in a latent, lower-dimensional space. Next, we use CT to sample the latent space by generating a t-way set of latent vectors, each of which represents a data point in the latent space. A synthetic dataset is generated from the t-way set by decoding each latent vector in the set. Our experimental evaluation suggests that machine learning models trained with synthetic datasets generated using our approach could achieve performance that is very similar to those trained with real-world datasets. Furthermore, our approach performs better than several state-of-the-art synthetic data generation approaches.
Flaky tests are tests that non-deterministically pass and fail in unchanged code. These tests can be detrimental to developers' productivity. Particularly when tests run in continuous integration environments, the tests may be competing for access to limited computational resources (CPUs, memory etc.), and we hypothesize that resource (un)-availability may be a significant factor in the failure rate of flaky tests. We present the first assessment of the impact that computational resources have on flaky tests, including a total of 52 projects written in Java, JavaScript and Python, and 27 different resource configurations. Using a rigorous statistical methodology, we determine which tests are RAFTs (Resource-Affected Flaky Tests). We find that 46.5% of the flaky tests in our dataset are RAFTs, indicating that a substantial proportion of flaky-test failures happen depending on the resources available when running tests. We report RAFTs and configurations to avoid them to developers, and received interest to either fix the RAFTs or to improve the specifications of the projects so that tests would be run only in configurations that are unlikely to encounter RAFT failures. Although most test suites in our dataset are executed quite quickly (under one minute) in a baseline configuration, our results highlight the possibility of using this methodology to detect RAFT to reduce the cost of cloud infrastructure for reliably running larger test suites.
Well-trained software development personnel, in the art and science of software testing, will effectively and efficiently develop quality software products with potentially fewer, less-critical defects. Thus software testing education is considered to be an important part of curricula for a university degree in Computer Science or Information Systems. The objective of this paper is to determine how much dedicated knowledge in the field of software testing is taught within Swedish universities. To achieve this objective, a systematic search of syllabi for software testing-related courses was done. From 25 Swedish universities offering Computer Science (or related) degrees, 14 currently offer dedicated courses in software testing. Some findings include: 32% of the individual courses were offered at the undergraduate level; 28% of the universities offer courses for specialised testing training; and, for the vast majority of the universities, dedicated software testing courses account for about 5% of the total degree credits. While some universities fare better than others, the overall state of academic software testing education in Sweden is limited but promising.
Machine learning-based regression test selection (MLRTS) has typically arisen in major cloud companies with full CI/CD pipelines. Various approaches for RTS are researched except for deep semantics of test cases, i. e. test objectives. Test objectives are essential but implicit information hard to calculate. In this paper, we propose a test architecture of a test objective-based MLRTS (TOMLRTS). First, word vectors are converted from explicitly written words in all regression test cases by Word2Vec. Second, semantic test objective clusters are formed by K-means++ to express all test objectives in the regression test cases. Third, distance vectors are constituted, whose elements are distances from each test case to the test objective clusters. Priorities of regression test cases are then calculated by MLRTS additionally according to the distance vectors. We additionally evaluate TOMLRTS compared to Facebook's MLRTS for commercial software. TOMLRTS selected test cases to detect bugs more rapidly.
Teaching software testing can be challenging due to low student interest, high cognitive load, and lack of alignment with industry needs. Previous research has attempted to address these challenges by using mutation testing, which involves intentionally introducing faults into the code to measure the ability of a test suite to detect faults. Although this method has been proven effective in teaching software testing, it can sometimes be difficult for a novice to write a test to kill some mutants because they are too subtle and there are no hints. In contrast, extreme mutation testing involves more evident changes (e.g., removing a method body) that may be easier for novice testers to identify. This paper investigates extreme mutation testing as an alternative to teaching software testing by comparing it to regular mutation testing in an empirical evaluation with two undergraduate classes. Our results show that both can help teach software testing, with regular mutation testing slightly more effective, and both types of reports were considered clear by a similar number of students.
Fault detection often depends on the specific order of inputs that establish states which eventually lead to a failure. However, beyond basic structural coverage metrics, it is often difficult to determine if the code has been exercised sufficiently to ensure confidence in its functions. Measures are needed to ensure that relevant combinations of input values have been tested with adequate diversity of ordering to ensure correct operation. Combinatorial testing and combinatorial coverage measures have been applied to many types of applications but have some deficiencies for verifying and testing state-based systems where the response depends on both input values and the current system state. In such systems, internal states change as input values are processed. Examples include network protocols, which may be in listening, partial connection, full connection, disconnected, and other states depending on the values of packet fields and the order of packets received. This paper discusses definitions for ordered t-way combination arrays and proves results regarding the construction of adequate blocks of test inputs, including consecutive ordering of combinations and with interleaving allowed. The application of the results to verify and test state- based systems is also illustrated.
Performance is a general quality attribute of every software system that developers always want to improve. A performance fuzzer helps developers in this task by automatically generating inputs hitting performance bottlenecks. However, a developer must still manually localize the root causes of these bottlenecks. In this study, we perform grammar-based performance fuzzing on an example System Under Test (SUT), focusing on response time for determining problematic grammar constructs with the highest likelihood of causing bottlenecks. We show that replacing these constructs creates an average of 40.53x speedup on 24 bottleneck cases out of 50. Furthermore, avoiding the problematic constructs in the input generation provides an average of 1.46x speedup. These preliminary results suggest a measurable link between grammar constructs and performance bottlenecks, opening up the possibility of high-level categorization and analysis.
Autonomous systems are supposed to automatically plan their actions and execute the plan without human intervention. In this paper, we propose a model-based two-layer framework for policy synthesis and test-case generation for autonomous systems. At the high-level layer of the framework, we have two kinds of methods for synthesising policies whose correctness is guaranteed by model checking. The autonomous system's controller executes synthesised policies at the low-level layer. As the kinematics of autonomous systems is often nonlinear and the environment may influence the results of their actions, formally verifying the controllers is extremely difficult. We propose a novel method for generating test cases for the controllers at the low-level layer. The method employs reinforcement learning for test-case generation and model checking to ensure that the test cases faithfully realise the execution of the policy. The framework is designed in Uppaal Stratego, which integrates model checkers and algorithms for policy synthesis. Therefore, the framework separates concerns and seamlessly interchanges the information between two layers.
Code coverage criteria define test objectives and provide actionable stopping conditions for creating test cases. They are also widely accepted as indicators of a test suite's fault detection effectiveness. Several factors influence the relationship between code coverage and test suite effectiveness. However, previous empirical studies investigating this relationship tend to fail to control some of these factors, resulting in contradictory results. Through a systematic review of 417 previous studies investigating the effectiveness of test suites from various venues, such as journals, conferences, workshops, and book chapters, we have identified several factors that can impact experimental studies' results and even threaten their validity. Some of these factors are well known, such as the test suite size (the number of test cases in the test suite), while some others are relatively unknown, such as the variety of execution traces (how different are test cases within a test suite in terms of structural coverage level). The list of factors we describe should be of interest to researchers and practitioners alike.
Goals - This whitepaper describes Context-aware adaptive mobile systems (CAAMS) experience testing as a crucial aspect of ensuring the functionality and user experience of mobile apps. It involves testing how a mobile app responds and adapts to different contexts, such as the user’s location, time of day, activity, or environment. This type of testing is essential for location-based services, hyper-personalization, adaptation to the environment, and ambient information. From an industry perspective, CAAMS experience testing is becoming increasingly important as mobile apps continue to integrate more advanced features and technologies. It is essential to ensure that they can provide relevant and accurate information and services to users in different contexts. This requires rigorous testing against various use cases and conditions to ensure that the mobile app can adapt and respond effectively. Furthermore, CAAMS experience testing is also important for ensuring the security and privacy of users. The industry-standard tools and techniques that are supporting context-aware adaptive mobile systems/application experience testing are a critical aspect of mobile app development and testing platforms are essential for ensuring a positive user experience and maintaining trust in mobile apps. When it comes to CAAMS testing, it is a semi-automated cognitive AI-driven approach that captures device key vitals covering architecture, design, UI/UX, performance, security, and more. CAAMS testing use cases with results benchmarked with industry-specific standards are demonstrated to unearth architectural, performance experience issues and more.
This study examines to what extent the testing of traditional software components and machine learning (ML) models fundamentally differs or not. While some researchers argue that ML software requires new concepts and perspectives for testing, our analysis highlights that, at a fundamental level, the specification and testing of a software component are not dependent on the development process used or on implementation details. Although the software engineering/computer science (SE/CS) and Data Science/ML (DS/ML) communities have developed different expectations, unique perspectives, and varying testing methods, they share clear commonalities that can be leveraged. We argue that both areas can learn from each other, and a non-dual perspective could provide novel insights not only for testing ML but also for testing traditional software. Therefore, we call upon researchers from both communities to collaborate more closely and develop testing methods and tools that can address both traditional and ML software components. While acknowledging their differences has merits, we believe there is great potential in working on unified methods and tools that can address both types of software.
Test-driven development is an effective industrial practice. As educators of future software engineers, we need to effectively teach students test-driven development. Unfortunately, this topic is usually taught in a theoretical way. Students hardly use this approach in their projects. In this paper, we present a simple software development workflow that seamlessly integrates requirements refinement, object-oriented design, and test-driven development. A step-by-step live-coding demonstration which resembles test-driven development tasks students will face in the industry is also provided. By adopting the workflow and exemplary project in a programming class, we hope this provides students with some hands-on experience, thus helping develop good habit of programming.
The concept of software test architecture (STA) has been discussed within industry meetings with little universal agreement on its usage. This paper proposes the use of Test Architecture (TA) as defined in the OMG UTP in place of STA and refines some earlier paper definitions and usage of STA to align with UTP and ISO concepts. Further, the paper considers TA styles of test environments and how they can use UTP modeling, which can lead to test automation as well as to better testing outcomes.
