Metamorphic testing (MT) is widely adopted for testing image processing applications. Although a variety of metamorphic relations (MRs) have been proposed, using all of them for testing will cost a large amount of computational resources. In addition, complex transformation operations are not well supported when generating follow-up test images based on MRs. To overcome these limitations, this study proposes a general MT framework for image processing applications, which employs CycleGAN to generate images that are very close to the realistic scenarios and leverages MRs for various categories of image processing applications. Two optimization strategies called EquivalentMR and SSampling are further proposed to reduce MRs and test images, respectively. A prototype tool called MT4I was developed. The experimental results showed that the proposed framework was capable of effectively testing various categories of image processing applications, while optimization strategies can reduce the amounts of MRs and test images without significantly jeopardizing the fault detection effectiveness.
This paper explores the educational and practical impacts of integrating metamorphic testing (MT) into a software engineering project conducted by an intercultural group of students. The students designed a Steer-by-Wire (SBW) system to control the steering of a model vehicle and tested using a hybrid approach that combined unit testing and MT. Four metamorphic relations (MRs) were generated and two significant violations were encountered during the testing phase. The first violation, related to steering angle consistency reported by the system, as a case of metamorphic exploration (ME), revealed a common coding mistake where the system failed to maintain consistent steering angles for equivalent inputs in opposite directions, illustrating how ME can enhance comprehension of the system and the testing process itself. It not only deepened the testers' understanding of the integration between software and mechanical systems but also represented valuable insights for others engaged in similar tasks. The second MR violation revealed issues with interruptions and delays when the system switched between manual and automated control modes, demonstrating MT's effectiveness in identifying defects and highlighting MT's importance in real-world software development scenarios. Additionally, the project examined the effectiveness of aligning MT roles to team members based on their Myers-Briggs Type Indicator (MBTI) personalities, suggesting that such alignments can enhance team dynamics and overall project efficiency. This study provides insights into the benefits of using MT in educational settings, the implications of personality-based task assignments, and the enhancement of software reliability and team performance in an intercultural context. The findings of this research reinforce the value of MT in software engineering education and support for the integration of psychological analysis in managing complex projects.
The Category Partition (CP) functional testing method has proven to be useful in various contexts. It begins by identifying parameters and environment conditions on the basis of the function's behaviour. The characteristics/categories of these parameters/environment conditions are identified and partitioned into choices. The choices of a category are mutually exclusive and can be based on input partitioning and boundary value analysis. Thereafter, the choices are combined on the basis of a selection criterion to form test frames. Once input values satisfying the conditions of a test frame's choices are identified, one is equipped with a test case. This paper suggests and demonstrates that those test frames, once equipped with characterizations of output values, i.e., with categories and choices for outputs, can be considered Metamorphic Relations to be used in Metamorphic Testing.
When applying metamorphic testing to a deep neural network (DNN), the DNN could have an "acceptable" level of accuracy but it performs poorly against some metamorphic relations (MRs). Such a DNN is considered not robust against these MRs. Improving both accuracy and robustness of a DNN is non-trivial because improving one aspect may adversely affect the other aspect. To alleviate this trade-off problem, we proposed a regularization-based method, in which an optimization function is designed to balance a DNN's accuracy and robustness. Then, we designed a reinforcement-learning-based algorithm to optimize this function. We tested our training method with two datasets (SVHN and CIFAR10), and each dataset with two DNN models. When comparing ours with the other six benchmark methods, we found the DNNs trained with our method have a better balance between accuracy and robustness.
The prediction of human trajectories is important for planning in autonomous systems that act in the real world, e.g. automated driving or mobile robots. Human trajectory prediction is a noisy process, and no prediction does precisely match any future trajectory. It is therefore approached as a stochastic problem, where the goal is to minimise the error between the true and the predicted trajectory. In this work, we explore the application of metamorphic testing for human trajectory prediction. Metamorphic testing is designed to handle unclear or missing test oracles. It is well-designed for human trajectory prediction, where there is no clear criterion of correct or incorrect human behaviour. Metamorphic relations rely on transformations over source test cases and exploit invariants. A setting well-designed for human trajectory prediction where there are many symmetries of expected human behaviour under variations of the input, e.g. mirroring and rescaling of the input data. We discuss how metamorphic testing can be applied to stochastic human trajectory prediction and introduce the Wasserstein Violation Criterion to statistically assess whether a follow-up test case violates a label-preserving metamorphic relation.
