ABSTRACT With the advancement of wireless technology, smart spaces such as homes and offices are increasingly populated with smart devices, including sensors and actuators. In such environments, user activities generate time‐series data that can be analysed to derive rules for the autonomous activation of actuators such as appliances, lights and switches. Since time‐series data in smart spaces grows continuously, manual annotation becomes impractical, making unsupervised learning techniques a practical choice for supporting automation. However, verifying the outcomes of these unsupervised methods is challenging due to the absence of ground‐truth labels. Moreover, the internal decision‐making processes of most unsupervised algorithms are complex and opaque, making their outputs difficult for end users to validate. These challenges give rise to the ‘oracle problem’, which complicates verification and validation. In this paper, we propose a metamorphic testing approach for verifying a novel sensor grouping technique in smart spaces. The technique employs a Spectral clustering algorithm with graph‐based feature representations derived from time‐series data. Our approach defines 10 metamorphic relations encompassing both verification and validation perspectives. Test cases are generated based on these relations, enabling the sensor grouping technique to be evaluated against variations in input data and clustering parameters. Experimental results demonstrate that the proposed approach effectively identifies implementation flaws in sensor relationship inference methods, assesses the structural consistency of clustering results in the absence of ground‐truth labels and evaluates the robustness of the technique across diverse smart space scenarios.
ABSTRACT Software reliability growth models (SRGMs) are widely used to predict software reliability based on defect discovery data collected during testing or operational phases. However, their predictive accuracy often degrades in data‐scarce environments, such as early‐stage testing or safety‐critical systems. Although cross‐project transfer learning has been explored to mitigate this issue by leveraging data from past projects, its applicability remains limited due to the scarcity and confidentiality of real‐world datasets. To overcome these limitations, we propose deep synthetic cross‐project SRGM (DSC‐SRGM), a novel approach that integrates synthetic data generation with cross‐project transfer learning. Synthetic datasets are generated using widely validated traditional SRGMs to represent defect discovery trends. A cross‐correlation‐based clustering method is applied to identify synthetic datasets with patterns similar to the target project. These datasets are then used to train a deep learning model for reliability prediction. The proposed method is evaluated on 60 real‐world datasets, and its performance is compared with both traditional SRGMs and cross‐project deep learning models trained on real‐world datasets. DSC‐SRGM achieves improvements of up to 98.3% over traditional SRGMs and up to 41.3% over cross‐project deep learning models trained on real‐world datasets across evaluation metrics. However, excessive use of synthetic data or a naive combination of synthetic and real‐world data may degrade prediction performance, highlighting the importance of maintaining an appropriate data balance. These findings indicate that DSC‐SRGM is a promising approach for software reliability prediction in data‐scarce environments.
ABSTRACT Code review is a well‐established practice to ensure code quality, identify potential bugs, promote knowledge sharing and maintain coding standards within a team or organization. This study aims to investigate the specific practices, challenges and information needs encountered when reviewing GUI‐based test artefacts for web applications, which remain poorly understood. We conducted a qualitative interview study with 14 software testing professionals from six different companies to explore the distinct aspects of reviewing GUI‐based test artefacts. We identified four practices, six challenges and four information needs related to reviewing GUI‐based test artefacts. The foremost challenge is the validation of GUI‐based tests under review. Furthermore, challenges concerning levels of abstraction and test robustness were not addressed in related studies. Additionally, participants proposed six potential improvements for tools and practices to better support the code review process. Notably, the absence of standardized practices and the need to run tests locally were common themes across participants. The code review process for GUI‐based test artefacts differs from that of production code, highlighting the need for practices and tools tailored specifically to the unique demands of GUI‐based testing.
ABSTRACT Smartphones have become the most commonly used electronic devices, realizing most of the functions of desktop computers and providing a variety of applications that meet user needs. However, constrained by various factors such as hardware and software, the nature of the mobile environment differs from that of personal computer (PC) and server environments. Traditional models based on the PC environment cannot accurately assess and predict the reliability of mobile applications. This paper proposes a new software reliability growth model (SRGM) that uses the extended Weibull function as the distribution of the total number of faults in the mobile environment. Meanwhile, considering the characteristics of dynamic outbreaks of software faults in the mobile environment, a multiple change‐points (CPs) scheme is proposed to improve the model. Relevant experiments were designed using entropy measures to verify the fitting and prediction performance of the proposed model. In addition, due to the large number of model parameters, a metaheuristic optimization algorithm called the Snake Optimizer is used to estimate the model parameters, and a sensitivity analysis of the model is conducted at the end. Experimental results indicate that the performance of the proposed model is superior to that of other models. The proposed model can assist developers and testers in evaluating the reliability of mobile software.
Assessing the robustness of safety-critical deep learning (DL) systems is of utmost importance, as these systems can cause harm when deployed in the real world. Metamorphic testing (MT) is one commonly used method to evaluate the robustness of DL systems, as it does not require expensive labelled ground truth data. This paper tackles two challenges: (1) One challenge in regulated domains such as the automotive industry is to provide a traceable argumentation of why a certain metamorphic relation (MR) was chosen. We adopt the idea of defect-based testing to MT and argue that an MR is traceable if it can be linked to a defect hypothesis. We demonstrate how to assess the robustness of safety-critical DL systems using the example of LiDAR object detectors. To this end, we create three new MRs for the LiDAR domain and identify five MR that can be reused by adapting them from related domains. Our experiments on the nuScenes dataset with three different object detectors produce 3.9 million test verdicts, of which 0.7 million are test failures. This shows that our defect-based MR effectively uncover failures. (2) A second challenge resulting from executing numerous metamorphic test cases is that MT can lead to the generation of an impractically high number of failures. We show how to prioritize the most critical failures, such as failures that occur close to the ego vehicle. By prioritizing, we reduced the observed 685,000 failures to 5397 safety-critical failures corresponding to a 127-fold reduction.
Vulnerability detection in software code is essential across multiple programming languages. However, existing approaches often introduce security vulnerabilities, as they rely only on token-based analysis, and most existing models were developed for specific programming languages. This study develops an enhanced vulnerability detection framework that incorporates a graph-based representation and a tokenized representation for effective vulnerability detection across multiple programming languages. An enhanced vulnerability detection framework is proposed in this study that leverages code property graphs (CPGs) and graph attention networks (GATs) to capture structural code relationships, while byte pair encoding (BPE) and CodeT5+ perform token-level vulnerability detection. The fusion of graph-based and token-based representations improves accuracy in identifying security flaws such as buffer overflows, injection attacks and memory leaks. To evaluate the efficacy of the proposed vulnerability detection framework, we conducted an experimental study on three vulnerability datasets. The three datasets detect the vulnerabilities with an accuracy of 96.71%, 98.80% and 95.84%. This shows that the proposed model effectively detects vulnerability across multiple programming languages.
The opacity of deep neural networks (DNNs) poses challenges in understanding the causes of their misbehaviours. Illumination search characterizes the inputs of a DNN by means of relevant features and explores the resulting feature map extensively. This facilitates the interpretation of misbehaviour-inducing inputs based on the regions they occupy in the feature map. However, current illumination-based approaches necessitate human expert involvement for the definition of the features, limiting broad applicability. In this paper, we address these limitations with DeepTheia, our fully automated illumination-based test generator that automatically extracts the features and explores the feature space using cutting-edge diffusion models. Experimental results show that DeepTheia consistently extracts highly discriminative features. Independent human assessors certified that DeepTheia is able to group misbehaviour-inducing inputs in a way that is understandable to humans in over 78% of the cases. Moreover, the inputs generated by DeepTheia were useful in significantly improving the ability of the original DL systems to handle inputs with critical feature combinations through fine-tuning.
Intelligent software built on Deep Neural Networks (DNNs) has been widely used in many fields, especially in Natural Language Processing (NLP). However, even the most advanced NLP models can also be attacked by delicately designed adversarial perturbations in many NLP application scenarios. To improve the reliability and robustness of such Intelligent software, adversarial test cases are generated to evaluate their performance. Existing adversarial text test case generation methods can be divided into two categories: heuristic and greedy. However, few methods can deceive DNNs while preserving the syntactic structure and semantic information of the original text, and most heuristic generation methods have a high time overhead. To address these challenges, this paper proposes a black-box NLP adversarial test case generation method—HGA, which optimizes the G enetic A lgorithm by adopting a H euristic multi-point crossover and cross-generational elite strategy to improve search efficiency, and designs a new fitness function based on part-of-speech and semantic similarity so as to guarantee the quality of test cases. We evaluate HGA upon four widely used text classification models and three real user datasets. The results show that HGA outperforms the baselines in the term of attack success rate, as well as human evaluation on test cases. In addition, the time overhead of HGA is reduced by nearly half compared to existing heuristic generation methods. Through training with the adversarial test cases generated by the HGA, the robustness of the threat model can be improved by up to 5%.
System logs play a vital role in recording operational behaviours and internal states. However, most of the research mainly focused on the log template features, without considering other features such as components and levels. Facing this challenge, we propose MFLog, a log sequence anomaly detection approach via multifeature extraction. Firstly, a log sequence feature extraction method based on the optimized BERT model and Bi-LSTM is proposed, which adopts the idea of out-of-manifold regularization to optimize the BERT model, employs Mixup as a form of out-of-manifold regularization to impose linear constraints on the out-of-manifold input space of the model, and extracts the sequence features of the log templates by using the optimized BERT model. Then, we utilize Bi-LSTM to analyse log components and levels, aiming to detect workflow-related anomalies and distinguish logs across different severity levels. Secondly, we propose a method for log sequence anomaly detection using dynamic weights. This method adaptively integrates log template, component, and level features using dynamic weights, effectively addressing the issue of feature redundancy and reducing the computational complexity caused by direct feature concatenation. Comparative experiments conducted on two publicly available datasets demonstrate that MFLog outperforms several state-of-the-art approaches in Precision, Recall and F1-score.
Search-based test generation typically targets structural coverage of source code. Past research suggests that targeting coverage alone is insufficient to yield tests that achieve common testing goals (e.g., discovering situations where a class-under-test throws exceptions) or detect faults. A suggested alternative is to perform multi-objective optimization targeting both coverage and additional objectives directly related to the goals of interest. However, it is not fully clear how coverage and goal-based objectives interact during the generation process and what effects this interaction will have on the generated test suites. In this study, we assess five hypotheses about multi-objective test generation and the relationships between coverage-based and goal-based objectives, focusing on the effects on coverage, goal attainment, fault detection, test suite size, test case length and the impact of the search budget. We generate test suites using the EvoSuite framework targeting Branch Coverage, three testing goals-Exception Count, Output Coverage and Execution Time-and combinations of coverage and goal-based objectives. Ultimately, we find that targeting multiple objectives does not reduce code coverage, yields no or minor reductions in goal attainment, but-at the same time-detects more faults compared with single-target configurations. In addition, it produces larger test suites, but test case length is not increased. The benefits of multi-objective optimization are often more limited than hypothesized in past research, but improved fault detection is still sufficient to recommend multi-objective optimization over targeting coverage or testing goals alone. Our study offers insights and guidance into how coverage and goal-based objectives interact during multi-objective test generation.
Integrating services in service-based architectures is a major concern and challenge to their developers. A key problem is that today's compilers cannot ensure syntactic interoperability of web APIs. Without further help, invalid calls surface only at runtime. Microservice-based architectures exacerbate this problem due to their use of polyglot software stacks and independent deployments. As a result, maintaining API compatibility with consumers has become increasingly complex. This study presents a systematic literature review on consumer-driven contract testing, a testing technique that ensures syntactic compatibility between microservices through isolated test execution. We develop a theory on when and how to use consumer-driven contract testing to address the problem of syntactic interoperability. We build out our theory with the insights of an action research study, contributing rare empirical data to the field. Our theory posits that consumer-driven contract testing can ensure syntactic interoperability between microservices and complement the testing strategy of such systems. The action research study confirmed this and revealed that introducing consumer-driven contract testing can promote the design and development of higher-quality APIs and code.
Mutation testing has been applied successfully to several programming languages. Despite its benefits for software testing, its high computational cost has prevented it from being widely used. Several refinements have been proposed to reduce its cost by decreasing the number of generated mutants, one of which is Evolutionary Mutation Testing. This refinement aims to generate a reduced set of mutants with an Evolutionary Algorithm, which searches for potentially equivalent and difficult to kill mutants that help improve the test suite. This study presents Tempus, a system that includes a Profile-Based individual generation version of Evolutionary Mutation Testing. Tempus has been applied to four case studies that process information in real time from Internet of Things systems. This huge volume of information arrives as events that need to be monitored and processed in real time: the case studies manage the events through Esper Event Processing Language queries. Given that the events used as input in the Internet of Things systems may need a reaction in a specific period of time, it is crucial to test that the system can trigger the expected responses on demand within the expected period. After applying Tempus to our test suite, 62 of the 80 experiments showed a reduction in cost over Evolutionary Mutation Testing.
Mutation testing is a software quality assurance technique that introduces small changes (i.e., 'mutants') into the code to help assess the effectiveness of test suites. In mutation testing, equivalent mutants are a special type of mutants that have the same behaviour as the original program for all the possible tests. A significant challenge of mutation testing is called the equivalent mutant problem, which leads to the very expensive cost of mutation testing, in terms of extensive manual work in the identification of equivalent mutants. As a result, mutation testing research efforts have focused on proposing approaches to aid testers in better identifying equivalent mutants. To address this challenge, this paper proposes leveraging new machine learning techniques to automatically classify equivalent mutants. Specifically, this research develops and compares a bidirectional GRU (Bi-GRU) based RNN model, an Elman recurrent neural network (RNN) model and a long short-term memory (LSTM) model, all based on the code representation of the abstract syntax tree (AST). The three machine learning models are trained separately, and the hyperparameters are tuned to obtain the best-performing values for each model. First, an experimental study was conducted using four different mutation operators and 690 equivalent and nonequivalent mutants from 24 C programs, to understand the effectiveness of the proposed approach. The experimental results indicate that the proposed machine learning-based approach can automatically classify equivalent mutants with an average F1-Score of 87.4% across all three models. For each mutation operator, the best performing models achieve an average F1-Score of 94%. The proposed approach can reduce the time and effort on manually identifying equivalent mutants, improve the set of generated mutants and eventually enable mutation testing to be more accessible to industry practitioners. A second set of experiments with seven traditional machine learning algorithms was run to compare against the proposed AST-based machine learning models. These traditional algorithms include decision trees (DT), Gaussian naive Bayes (GNB), K-nearest neighbor (KNN), linear discriminant analysis (LDA), linear regression (LR), random forest (RF) and support vector machine (SVM). The comparison results indicate our proposed AST-based Bi-GRU, LSTM and RNN models achieved the highest F1-Scores, at 84.61%, 87.99% and 88.88%, respectively. In addition, to demonstrate the effectiveness of our proposed models in a more realistic scenario where one could expect stubborn mutants, we developed a sub-dataset consisting of a mix of equivalent, nonequivalent and stubborn mutants. Testing results reflect our proposed models' capabilities at identifying equivalent mutants when stubborn mutants are present, as all metrics scored above 90%. The combination of the key performance metrics across three different experiments shows the AST-based models' effectiveness in terms of the identification of equivalent mutants.
Modern computer games have become very complex, so they can benefit from automated testing. However, their huge and fine grained interaction space makes them very challenging for automated testing algorithms. Having a model of a system would greatly improve the effectiveness of a testing algorithm. However, manually constructing a model is expensive and time-consuming. This paper proposes an online agent-based search approach to solve common testing tasks for computer games, in particular games that involve elements of world navigation and exploration. On the fly, the approach also constructs a model of the system, which is then exploited to solve the given testing task. The effectiveness of the approach is studied via a case study called Lab Recruits and its simulation of another game called Dungeons and Dragons Online. The study showed that the approach is superior in its ability to complete testing tasks and its completion time compared to evolutionary algorithm, Q-learning and MCTS. This paper extends a previous work presented in ATEST by including evaluation on large game levels, evaluation of the achieved coverage and fault detection and the aforementioned comparison with other algorithms.
Metamorphic testing (MT) is an effective software quality assurance method; it uses metamorphic relations (MRs) to examine the inputs and outputs of multiple test cases. Metamorphic exploration (ME) and metamorphic robustness testing (MRT) are advanced additions to the MT literature: ME can enable better understanding, and thus testing, of systems; MRT can test software robustness without an oracle. Metamorphic relation patterns (MRPs) are abstractions or templates for multiple concrete MRs. The performance of MT/ME/MRT relies on the quality of MRs, but the identification of MRs is often a manual task requiring some knowledge of MRPs, creative thinking and a good understanding of the system. In other words, the effective identification of MRs remains a challenge and requires further study. In this paper, we propose formal definitions to describe the relationship between multiple MRPs in order to classify them. We also identify a series of new MRPs and propose a new MT framework that can guide the identification and application of MRPs. Based on our definitions, we present the notion of MRP family trees and further propose two trees by classifying previously published MRPs and our proposed MRPs, in order to make it easier for users to find their target MRPs for reuse, reference or inference. Finally, we report on case studies using the MRPs and our framework to guide the identification of MRs for MT/ME/MRT and successfully detect multiple violations.
The video gaming industry has evolved into an industry that drives high revenue and tangentially faces some of the toughest problems in the software engineering domain. With more games being released with either remote multiplayer components or cloud streaming foundations, maintaining an engaging player experience that lives up to a high-quality bar is a problem that many game studios must overcome. Traditionally, studios have relied on brute-force manual testing of their software to find impacting bugs. However, as studios and publishers focus on shortening the delivery timeline of their products to consumers, we have seen a rise in the use of testing tools, automation and enhanced processes that focus on achieving a high level of quality. In this systematic literature review, we provide a review of the tools and processes and discuss how these tools enhance the quality experience or help decrease the cost of testing while allowing for faster releases. We conclude by observing some potential areas of opportunity, such as suggesting an approach to help validate the severity of bugs found through community validation and providing a mechanism to create a way to track bugs across heterogeneous devices.
ABSTRACT The recent growth in popularity of wearable medical devices has improved the quality of life of people with medical conditions. Testing such devices may require users to configure these systems using physical trials, putting themselves in potentially dangerous scenarios. Misconfiguration of such devices has caused disease misdiagnoses and incorrect drug prescriptions. Digital twins have been proposed as an opportunity to reduce such risks of testing system configurations in simulated environments, decoupling the user from the system under test. In this paper, we perform an evaluative case study to assess the use of a digital twin for configuration testing of an artificial pancreas system (APS) control algorithm. These systems regulate the blood glucose levels in people with type 1 diabetes mellitus, and so misconfigurations can cause severe hypoglycaemia or hyperglycaemia, which can be life‐threatening. We tested the OpenAPS control algorithm against 156 people's clinical data. We found that our digital twin provided an accurate simulation environment to perform configuration testing and accurately predict blood glucose–insulin behaviour. We evaluated different APS configurations, identifying a potentially unsafe configuration without the risks associated with a physical trial. We identified the challenges associated with modelling clinical data, which could lead to misinterpretations in configuration testing and the reduction of test reliability when modelling stochastic body dynamics.
Nowadays, more and more IT services are being hosted on cloud systems, which render cloud systems to grow into a huge complex with millions of physical servers, multi-layer software stacks and the processing of cloud service requests across many servers and software layers. It is highly demanded for cloud service providers to have the capability of getting the knowledge on cloud service behaviour directly from the service execution instead of from people's expertise. This paper studies the problem of tracing cloud service's processing of requests across components in cloud environments and proposes cloud tracing mechanisms for this purpose. We also developed model-based studies of our proposed mechanisms for analysing certain designs of the mechanisms. The implementation of the proposed cloud tracing is deployed onto the environments of OpenStack, Kubernetes and Hadoop, and the experiments on these environments demonstrate that our mechanisms effectively trace cloud service behaviour and generate a single complete request execution path, while without our mechanisms the cloud tracing either fails to work or results in thousands of path segments. Our mechanisms have a low performance overhead (2.3%) in the experiments.
ABSTRACTThere has been a significant amount of interest regarding the use of DBTtsfull in software testing over the past two decades. Diversity‐based testing (DBT) technique uses similarity metrics to leverage the dissimilarity between software artefacts—such as requirements, abstract models, programme structures or inputs—in order to address a software testing problem. DBT techniques have been used to assist in finding solutions to several different types of problems including generating test cases, prioritizing them and reducing very large test suites. This paper is a systematic mapping study of DBT techniques that summarizes the key aspects and trends of 167 papers that report the use of 79 different similarity metrics with 22 different types of software artefacts, which have been used by researchers to tackle 11 different types of software testing problems. We further present an analysis of the recent trends in DBT techniques and review the different application domains to which the techniques have been applied, giving an overview of the tools developed by researchers in order to do so. Finally, the paper identifies some DBT challenges that are potential topics for future work, such as exploring other diversity artefacts and measuring diversity for complex data input.
In the realm of software development, selecting the appropriate Java application programming interfaces (APIs) from a vast pool remains a significant challenge for developers. This research addresses this complexity by tackling the limitations of current API recommendation methods, which often struggle to align API suggestions with the specific queries and development contexts. In this paper, we introduce a novel prompt method named DSKIPP (Development Scenario, key Knowledge and Intention's Progressive Prompt), designed to enhance the efficiency of large language models (LLMs) in Java API recommendations. Firstly, we devise an overview of DSKIPP which conducts LLMs through a sequential process: first, inferring the package level, followed by the class level, and ultimately the method level as an API comprises three distinct components at varying levels-package, class and method. Secondly, at each level, DSKIPP assists LLMs in deducing the development scenario associated with a query and the essential key knowledge relevant to that scenario. This approach enables LLMs to gain a more profound contextual understanding of the query's intention. Moreover, during the inference process at the class and method level, we implement a self-check mechanism enabling LLMs to validate the results and ensure a more reasoned and reliable outcome. To validate the efficiency of DSKIPP, comparison and ablation experiments are both conducted within Java programming environment. The comparison results affirm that our method outperforms the current state-of-the-art technologies in API recommendation tasks, while the ablation results shed light on why DSKIPP can enhance the reliability of API recommendations in LLMs. This research contributes to the field by offering a more reliable and context-sensitive solution for API recommendation in software development.
