Metamorphic testing (MT) is widely adopted for testing image processing applications. Although a variety of metamorphic relations (MRs) have been proposed, using all of them for testing will cost a large amount of computational resources. In addition, complex transformation operations are not well supported when generating follow-up test images based on MRs. To overcome these limitations, this study proposes a general MT framework for image processing applications, which employs CycleGAN to generate images that are very close to the realistic scenarios and leverages MRs for various categories of image processing applications. Two optimization strategies called EquivalentMR and SSampling are further proposed to reduce MRs and test images, respectively. A prototype tool called MT4I was developed. The experimental results showed that the proposed framework was capable of effectively testing various categories of image processing applications, while optimization strategies can reduce the amounts of MRs and test images without significantly jeopardizing the fault detection effectiveness.
This paper explores the educational and practical impacts of integrating metamorphic testing (MT) into a software engineering project conducted by an intercultural group of students. The students designed a Steer-by-Wire (SBW) system to control the steering of a model vehicle and tested using a hybrid approach that combined unit testing and MT. Four metamorphic relations (MRs) were generated and two significant violations were encountered during the testing phase. The first violation, related to steering angle consistency reported by the system, as a case of metamorphic exploration (ME), revealed a common coding mistake where the system failed to maintain consistent steering angles for equivalent inputs in opposite directions, illustrating how ME can enhance comprehension of the system and the testing process itself. It not only deepened the testers' understanding of the integration between software and mechanical systems but also represented valuable insights for others engaged in similar tasks. The second MR violation revealed issues with interruptions and delays when the system switched between manual and automated control modes, demonstrating MT's effectiveness in identifying defects and highlighting MT's importance in real-world software development scenarios. Additionally, the project examined the effectiveness of aligning MT roles to team members based on their Myers-Briggs Type Indicator (MBTI) personalities, suggesting that such alignments can enhance team dynamics and overall project efficiency. This study provides insights into the benefits of using MT in educational settings, the implications of personality-based task assignments, and the enhancement of software reliability and team performance in an intercultural context. The findings of this research reinforce the value of MT in software engineering education and support for the integration of psychological analysis in managing complex projects.
The Category Partition (CP) functional testing method has proven to be useful in various contexts. It begins by identifying parameters and environment conditions on the basis of the function's behaviour. The characteristics/categories of these parameters/environment conditions are identified and partitioned into choices. The choices of a category are mutually exclusive and can be based on input partitioning and boundary value analysis. Thereafter, the choices are combined on the basis of a selection criterion to form test frames. Once input values satisfying the conditions of a test frame's choices are identified, one is equipped with a test case. This paper suggests and demonstrates that those test frames, once equipped with characterizations of output values, i.e., with categories and choices for outputs, can be considered Metamorphic Relations to be used in Metamorphic Testing.
When applying metamorphic testing to a deep neural network (DNN), the DNN could have an "acceptable" level of accuracy but it performs poorly against some metamorphic relations (MRs). Such a DNN is considered not robust against these MRs. Improving both accuracy and robustness of a DNN is non-trivial because improving one aspect may adversely affect the other aspect. To alleviate this trade-off problem, we proposed a regularization-based method, in which an optimization function is designed to balance a DNN's accuracy and robustness. Then, we designed a reinforcement-learning-based algorithm to optimize this function. We tested our training method with two datasets (SVHN and CIFAR10), and each dataset with two DNN models. When comparing ours with the other six benchmark methods, we found the DNNs trained with our method have a better balance between accuracy and robustness.
The prediction of human trajectories is important for planning in autonomous systems that act in the real world, e.g. automated driving or mobile robots. Human trajectory prediction is a noisy process, and no prediction does precisely match any future trajectory. It is therefore approached as a stochastic problem, where the goal is to minimise the error between the true and the predicted trajectory. In this work, we explore the application of metamorphic testing for human trajectory prediction. Metamorphic testing is designed to handle unclear or missing test oracles. It is well-designed for human trajectory prediction, where there is no clear criterion of correct or incorrect human behaviour. Metamorphic relations rely on transformations over source test cases and exploit invariants. A setting well-designed for human trajectory prediction where there are many symmetries of expected human behaviour under variations of the input, e.g. mirroring and rescaling of the input data. We discuss how metamorphic testing can be applied to stochastic human trajectory prediction and introduce the Wasserstein Violation Criterion to statistically assess whether a follow-up test case violates a label-preserving metamorphic relation.
As the complexity of autonomous driving systems (ADSs) increases, the question of how to organize testing in an efficient manner has become a serious issue. This study investigates the potential for metamorphic testing (MT) to evaluate the perception-camera module of an open-source autonomous driving system (ADS), namely Baidu’s Apollo ADS. The experiments revealed inconsistent obstacle identification results when increasing the brightness of a specific region of the driving scenarios, both in individual and sequential frames, demonstrating the ability of MT to address the oracle problem when testing the perception module of ADSs. Furthermore, this paper presents an MT harness to facilitate ADS testing, which would increase efficiency and help testers to better organize the testing procedure. We also present an industry case study to demonstrate its use in actual production phases.
Simulation testing is considered the supplement to ensure the safety of Autonomous driving (AD) and advanced driving assistance (ADAS's) systems in terms of time and costs. However, it is very difficult and challenging when the simulation results are unexpected. This work presents a simulation-based metamorphic testing (MT) approach to test the ADAS system, implementing the European new car assessment program (Euro NCAP) standards on OpenStreetMap (OSM). We first defined input patterns and relations related to autonomous driving, following the principles of MT. To assess the approach, we executed three tests in two steps at both the design and system levels. Our results show that none of the three (source) tests detected any collisions. However, for follow-up test cases, the ego vehicle failed to apply brakes to avoid a collision when the speed changed. A real-life issue in the system was immediately revealed and confirmed by the development team. We then designed a mechanism and continued the test to check whether the recorded collisions were avoidable. Our results (rate of 5.8%) indicate the fault detection effectiveness when testing the ADAS system. Although we applied the approach to testing the ADAS driving performance, it can be applied to other AD systems. This research, therefore, provides a systematic way to design and test autonomous driving technologies and integrate testing standards with metamorphic testing.
Autonomous vehicles (AV), at their highest potentials, will provide a greater mobility, an increased traffic efficiency and, more importantly, safer trips for millions of people every day. While assuring their safety and reliability is, thus, of great importance, it is also a huge challenge. Metamorphic testing (MT) has been shown to be one of the most successful testing techniques to validate automated driving systems (ADS) underpinning the AV. Having said that, there are still lots of rooms for further improving the ADS testing with MT. On one hand, the non-determinism in ADS' behaviors poses great challenges for precisely judging their correctness. On the other hand, the testing scenarios used in the existing studies are still not very much complex for mimicking various realistic traffic conditions. In this study, we propose a new framework which takes into account the hypothesis testing to provide a more solid way for judging the non-deterministic behaviors of test outcomes. On top of that, we develop a new toolkit to implement more complex and realistic ADS testing scenarios. To demonstrate its practicability, we design complex traffic scenarios and pay attention to examine the ADS' behaviors in non-collision cases which are often unable to be detected by conventional testing methods. It is then applied to test Autoware, a state-of-the-art modular ADS using the Carla simulator. An analysis of results with the Mann-Whitney-Wilcoxon test and Cohen's d values reveals a large number of consistencies and reliability issues of Autoware. The findings highlight the flexibility and capability of our MT-based framework in validating the AV using a non-deterministic measure and realistic scenarios that can work in the absence of ground truth datasets.
Deep Learning (DL) is one of the most popular research topics in machine learning and DL-driven image recognition systems have developed rapidly. Recent research has employed metamorphic testing (MT) to detect misclassified images. Most of them discuss metamorphic relations (MR), with limited attention given to which regions should be transformed. We focus on the fact that there are sensitive regions where even small transformations can easily change the prediction results and propose an MT framework that efficiently tests for regions prone to misclassification by transforming these sensitive regions. Our evaluation demonstrated that the sensitive regions can be specified by Explainable AI (XAI) and our framework effectively detects faults.
Hand pose estimation (HPE) is a task that predicts and describes the hand poses from images or video frames. When HPE models estimate hand poses captured in a laboratory or under controlled environments, they normally deliver good performance. However, the real-world environment is complex, and various uncertainties may happen, which could degrade the performance of HPE models. For example, the hands could be occluded, the visibility of hands could be reduced by imperfect exposure rate, and the contour of hands prone to be blurred during fast hand movements. In this work, we adopt metamorphic testing to evaluate the robustness of HPE models and provide suggestions on the choice of HPE models for different applications. The robustness evaluation was conducted on four state-of-the-art models, namely MediaPipe hands, OpenPose, BodyHands, and NSRM hand. We found that on average more than 80% of the hands could not be identified by BodyHands, and at least 50% of hands could not be identified by MediaPipe hands when diagonal motion blur is introduced, while an average of more than 50% of strongly underexposed hands could not be correctly estimated by NSRM hand. Similarly, applying occlusions on only four hand joints will also largely degrade the performance of these models. The experimental results show that occlusions, illumination variations, and motion blur are the main obstacles to the performance of existing HPE models. These findings may pave the way for researchers to improve the performance and robustness of hand pose estimation models and their applications.
In this paper, we consider the testing of software whose input involves a data stream, such as in video analysis applications. To address the oracle problem in testing this kind of software, existing metamorphic testing (MT) and metamorphic exploration (ME) techniques typically generate both source and follow-up input streams. In contrast, we propose a different strategy: an in-place method. Instead of examining relations among multiple input streams, our method works on a finer granularity by investigating relations among the data frames within a single input stream --- such as the relations among the image frames within a video stream --- treating a frame, rather than a stream, as a test case. Because our method does not generate any follow-up input stream, or change any frame values, it is also suitable for runtime error detection. We have applied this in-place method to testing video streams in Baidu Apollo, a real-life autonomous driving system. Our study identified previously-unknown obstacle perception failures in the camera perception module, including both undetected and incorrectly-detected objects. The empirical results show that our approach is practical, and readily applicable to industrial-scale systems that include (but are not limited to) the computer vision domain.
Fairness of deepfake detectors in the presence of anomalies are not well investigated, especially if those anomalies are more prominent in either male or female subjects. The primary motivation for this work is to evaluate how deepfake detection model behaves under such anomalies. However, due to the black-box nature of deep learning (DL) and artificial intelligence (AI) systems, it is hard to predict the performance of a model when the input data is modified. Crucially, if this defect is not addressed properly, it will adversely affect the fairness of the model and result in discrimination of certain sub-population unintentionally. Therefore, the objective of this work is to adopt metamorphic testing to examine the reliability of the selected deepfake detection model, and how the transformation of input variation places influence on the output. We have chosen MesoInception-4, a state-of-the-art deepfake detection model, as the target model and makeup as the anomalies. Makeups are applied through utilizing the Dlib library to obtain the 68 facial landmarks prior to filling in the RGB values. Metamorphic relations are derived based on the notion that realistic perturbations of the input images, such as makeup, involving eyeliners, eyeshadows, blushes, and lipsticks (which are common cosmetic appearance) applied to male and female images, should not alter the output of the model by a huge margin. Furthermore, we narrow down the scope to focus on revealing potential gender biases in DL and AI systems. Specifically, we are interested to examine whether MesoInception-4 model produces unfair decisions, which should be considered as a consequence of robustness issues. The findings from our work have the potential to pave the way for new research directions in the quality assurance and fairness in DL and AI systems.
Deep Learning (DL) components are increasing their presence in mission and safety-critical systems, such as autonomous vehicles. The verification process of such systems needs to be rigorous, for which automated solutions are paramount. To allow test automation, test oracles are necessary. In the context of DL systems, metamorphic test oracles have found to be effective. However, such oracles require the execution of multiple tests, which makes testing more expensive. Metamorphic relation composition can reduce the cost of metamorphic testing. However, its effectiveness has found mixed answers. This paper reports the preliminary results of our study on measuring the cost-effectiveness of composite metamorphic relations for testing DL systems. To this end, we empirically evaluate the cost-effectiveness of composite metamorphic relations within a DL model for object classification. Our results suggest that composite metamorphic relations reduce the failure revealing capability when compared to their component metamorphic relations.
Simulation is widely adopted by the research community to analyze and study complex systems. It is based on the idea of creating a model representing the target system under study, so that the experiments can be executed over the model instead of the target system. However, since the model is a simplification of a real-world system, the obtained results entail an accuracy loss, which makes determining the reliability of the experiments a complex task. Testing can be applied to check the correctness of systems. Thus, an oracle is used to determine if a test is correct or not. In the field of simulation, an oracle can be applied to determine the reliability of the results, but in most cases, the oracle is not available or is computationally too expensive to be applied. In this work, we propose to use metamorphic testing to detect faults in simulated distributed systems. In essence, we use metamorphic relations - representing the relevant properties of the system under study - as an oracle. Thus, the results provided are contrasted against these relations to determine their reliability. In order to show the applicability of this approach, we have modelled different distributed systems architectures using the SIMCAN simulator and a high performance application that is executed over the models.
Speech recognition (SR) systems are systems that convert speech signals into text and are widely used in mobile devices, wearable devices, and intelligent living room devices. When SR software recognizes speech in the laboratory or another quiet environment, the recognition accuracy is high. However, when SR software is applied in a complex real-world environment, there may be background noise from birds, machines, and soon, the influence of other speakers, or other adverse factors, which will reduce the recognition accuracy of the SR software. In this work, we propose SR-MT, which is a metamorphic testing (MT) approach to test the robustness of SR software. In SR-MT, we adopt four criteria to analyze the MT results and rank the robustness of the target SR software for speech affected by noise interference and speech variations from level 1 to level 5. SR-MT was evaluated on three real industrial applications: iFLYTEK speech-to-text(1), Baidu speech-to-text(2), and Google speech-to-text(3). We found that on average, 13.5% of the words in the speech could not be recognized correctly when the signal-to-noise ratio reached 10 dB. Similarly, changes in speech speed and tone will also reduce the recognition accuracy of SR software.
MetaCompass is a metage nomic assembly tool developed by the Pop Lab at the University of Maryland, College Park [2]. Such tools are vital for studying the DNA of the variety of microorganisms present in mixed environmental samples, but they can be difficult to test due to the infeasibility of assessing the correctness of the output of a given test case. In this paper, we demonstrate a simple metamorphic test that detects inconsistencies in the popular assem-bly tools MEGAHIT [7] and Pilon [11], upon which MetaCompass relies. These inconsistencies were not detected during the development process by standard testing procedures, despite the fact that they may be introducing undesirable noise into MetaCompass's results. Our results suggest that metamorphic testing is a valuable tool for finding otherwise subtle defects in metagenomic assembly software.
Advancing ocean science has a significant impact to the development of the world, from operating a safe navigation for vessels to maintaining a healthy and diverse ocean ecosystem. Various ocean software systems have been extensively adopted for different purposes, for instance, predicting hourly sea level elevation across shorelines, simulating large-scale ocean circulations, as well as integrating into Earth system models for weather forecasts and climate projections. Regardless of their significance, guaranteeing the trustworthiness of ocean software and modelling systems is a long-standing challenge. The testing of ocean software suffers a lot from the so-called oracle problem, which refers to the absence of test oracles mainly due to the nonlinear interactions of multiple physical variables and the high complexity in computation. In the ocean, observed tidal signals are distorted by non-deterministic physical variables, hindering us from knowing the "true" astronomical tidal constituents existing in the timeseries. In this paper, we present how to test tidal analysis and prediction (TAP) software based on metamorphic testing (MT), a simple yet effective testing approach to the oracle problem. In particular, we construct metamorphic relations from the periodic property of astronomical tide, and then use them to successfully detect a real-life defect in an open-source TAP software. We also conduct a series of experiments to further demonstrate the applicability and effectiveness of MT in the testing of TAP software. Our study not only justifies the potential of MT in testing more complex ocean software and modelling systems, but also can be expanded to assess and improve the quality of a broader range of scientific simulation software systems.
Searching and displaying data based on user queries is a pervasive feature of most software applications such as information systems, web portals, and web APIs. The large volume of data managed by these types of systems, henceforth called query-based systems (QBSs), makes them extremely hard to test due to the difficulty to assess whether the output of a query is correct, the so-called test oracle problem. Metamorphic testing has proved to be a very effective approach to alleviate the oracle problem in QBSs, by exploiting the relations among multiple executions of the QBS under test, so-called metamorphic relations (MRs). However, the identification of MRs mostly remains a manual and creative task, limiting the applicability of the approach. In this paper, we propose a method for the automated generation of MRs in QBSs starting from a lightweight specification of the query parameters of the system. Evaluation results show that hundreds of MRs can be automatically identified in real-world systems like IMDb, SkyScanner, or YouTube in just a few seconds.
Metamorphic testing requires the availability of a suitable set of metamorphic relations (MRs) for the application domain of interest. A software testing practitioner interested in using metamorphic testing is thus blocked unless they can devise a suitable set of MRs. In this paper we offer some practical advice on sources of inspiration for MRs, based on our experience building three fuzzing tools based on metamorphic testing: MF++, which supports automated testing of C++11 libraries, C4, which tests concurrency support in C11 compilers, and spirv-fuzz, which aims to find bugs in compilers for the SPIR-V programming language (mainly used in computer graphics). The MRs we have devised have taken inspiration from three main sources: (1) careful study of specification documents related to the libraries and programming languages that these tools target, (2) consultation of prior work and discussion with domain experts, and (3) manual inspection of the results of automated code coverage analysis on the systems under test. We describe these sources of inspiration in detail, giving a range of concrete examples for each. We hope that this experience report will help to inform developers of future metamorphic testing tools as to the steps they can take to discover MRs in their domains of interest.
Deep Neural Network based models are widely used for perception and control in autonomous driving. Recent work leverages metamorphic testing to improve defect detection but is limited to using only an equality-based metamorphic relation. Thus, it does not provide sufficient expressiveness for users to define custom metamorphic relations nor means to automatically generate meaningful inputs based on such expressive metamorphic relations that reflect real-world traffic behaviors. In this paper, we preliminarily design and evaluate a declarative Behaviour-Driven Development (BDD)-based metamorphic testing framework BMT, which enables domain experts to specify custom traffic behaviors—a car shall decelerate by x% when a bicycle is in front, etc. It then automatically translates a human-written behavior to a corresponding metamorphic relation and synthesizes meaningful test inputs using a variety of image and graphics processing techniques. Our preliminary evaluation shows that BMT can detect a significant number of erroneous predictions of three driving models for speed predictions. These detected erroneous predictions are manually examined and confirmed by six human judges as meaningful traffic violations. By automating test generation from custom behaviors, BMT enables experts to easily express domain-specific constraints and finds violations of such constraints.
