Currently, enormous amounts of data are continuously processed to support our daily activities, such as managing bank accounts, streaming movies, or interacting on social networks. In recent years, cloud infrastructures have proven to be a reliable solution, not only for processing this data but also for enabling users worldwide to access it remotely. However, this processing demands vast computing resources, leading to significant energy consumption. In this paper, we present a strategy to address this problem by combining multi-objective optimization techniques with Metamorphic Testing (MT) and simulation tools to optimize cloud systems, focusing on both performance and energy consumption. To achieve this, several multi-objective genetic algorithms (MOGAs) have been integrated into the MT-EA4Cloud framework, a solution that previously applied single-objective evolutionary algorithms with MT. To determine the suitability of the proposed approach, an empirical study was conducted to analyze the behavior of the different MOGAs included in the framework. In this study, various test sets and two distinct workloads-inspired by big data analytics operations-were created to represent multiple cloud scenarios. The results clearly demonstrate that MOGAs can be effectively combined with MT to optimize cloud systems while considering multiple objectives-in this case, performance and energy consumption. A careful analysis of the results indicates that increasing the mutation rate leads to the best outcomes. In general, the NSGA-II algorithm has produced the best results in the experiments conducted in this study.

The Internet of Things (IoT) paradigm has experienced exponential growth in recent years, becoming a key component in the business strategies of leading global technology companies. However, IoT systems face critical challenges such as high mobility demands, low latency requirements, and significant bandwidth consumption. Fog computing has emerged as a viable solution to alleviate these challenges. This paradigm introduces intermediate layers between edge devices and centralised cloud systems, which reduces latency and alleviates bandwidth bottlenecks inherent in traditional computing models. Despite its potential, fog systems often require significant financial investment, either for proprietary infrastructure or pay-per-use services, which constrains their study to theoretical analyses.To address these difficulties, we present Simcan2Fog, a discrete-event simulation platform for modelling and analysing fog computing environments. Built on OMNeT++ and INET – widely adopted frameworks for discrete-event simulation and network protocol modelling, respectively – Simcan2Fog provides highly detailed communication network models and enhanced capabilities to model sensors, actuators, controllers, applications, distribution algorithms, and interconnected fog devices. Additionally, it inherits cloud computing related functionalities such as virtualisation, data centres, cloud provider allocation policies, and user management from the Simcan2Cloud simulator. These features enable Simcan2Fog to simulate realistic IoT scenarios, offering detailed insights into performance metrics such as latency and resource utilisation.

Testing is essential for assessing the correctness of software systems. Metamorphic testing (MT) is an approach especially suited when the system under test lacks oracles, or they are expensive to compute. However, creating an MT environment for a specific domain (e.g., cloud simulation, model transformation, machine learning) requires substantial effort. To alleviate these difficulties, we present a model-driven tool that automates the construction of MT environments. Starting from a meta-model with the domain concepts, and a description of the domain execution environment, our tool produces an MT environment featuring comprehensive support for the MT process. This includes the definition of domain-specific metamorphic relations, their evaluation, detailed reporting of the testing results, and the automated search-based generation of follow-up test cases. This paper illustrates the tool on a case-study in the domain of video streaming APIs. A video showcasing the tool is available at https://youtu.be/DeuIW6V4LaQ.

Cloud computing is an evolving paradigm whose adoption has been increasing over the last few years. This fact has led to the growth of the cloud computing market, together with fierce competition for the leading market share, with an increase in the number of cloud service providers. Novel techniques are continuously being proposed to increase the cloud service provider’s profitability. However, only those techniques that are proven not to hinder the service agreements are considered for production clouds. Analysing the expected behaviour and performance of the cloud infrastructure is challenging, as the repeatability and reproducibility of experiments on these systems are made difficult by the large number of users concurrently accessing the infrastructure. To this, must be added the complications of using different provisioning policies, managing several workloads, and applying different resource configurations. Therefore, in order to alleviate these issues, we present Simcan2Cloud, a discrete-event-based simulator for modelling and simulating cloud computing environments. Simcan2Cloud focuses on modelling and simulating the behaviour of the cloud provider with a high level of detail, where both the cloud infrastructure and the interactions of the users with the cloud are integrated in the simulated scenarios. For this purpose, Simcan2Cloud supports different resource allocation policies, service level agreements (SLAs), and an intuitive and complete API for including new management policies. Finally, a thorough experimental study to measure the suitability and applicability of Simcan2Cloud, using both real-world traces and synthetic workloads, is presented.

Mutation testing is a valuable technique for measuring the quality of test suites in terms of detecting faults. However, one of its main drawbacks is its high computational cost. For this purpose, several approaches have been recently proposed to speed-up the mutation testing process by exploiting computational resources in distributed systems. However, bottlenecks have been detected when those techniques are applied in large-scale systems. This work improves the performance of mutation testing using large-scale systems by proposing a new load distribution algorithm, and parallelising different steps of the process. To demonstrate the benefits of our approach, we report on a thorough empirical evaluation, which analyses and compares our proposal with existing solutions executed in large-scale systems. The results show that our proposal outperforms the state-of-the-art distribution algorithms up to 35% in three different scenarios, reaching a reduction of the execution time of-at best-up to 99.66%.

Simulation is widely adopted by the research community to analyze and study complex systems. It is based on the idea of creating a model representing the target system under study, so that the experiments can be executed over the model instead of the target system. However, since the model is a simplification of a real-world system, the obtained results entail an accuracy loss, which makes determining the reliability of the experiments a complex task. Testing can be applied to check the correctness of systems. Thus, an oracle is used to determine if a test is correct or not. In the field of simulation, an oracle can be applied to determine the reliability of the results, but in most cases, the oracle is not available or is computationally too expensive to be applied. In this work, we propose to use metamorphic testing to detect faults in simulated distributed systems. In essence, we use metamorphic relations - representing the relevant properties of the system under study - as an oracle. Thus, the results provided are contrasted against these relations to determine their reliability. In order to show the applicability of this approach, we have modelled different distributed systems architectures using the SIMCAN simulator and a high performance application that is executed over the models.

During the last years, the fast evolution of computers and networks has led to the creation of a wide variety of services that have changed the way we live, like video streaming, online gaming and online shopping. These services are supported by complex systems, which require not only high computational power but high-speed and low-latency networks to fulfil the expected quality requirements. However, a misleading configuration in one of the thousand components that compose these systems may cause performance bottlenecks and functioning disruptions. Unfortunately, conventional testing methods are not adequate for checking these systems since, on many occasions, there is no mechanism to determine if the behaviour of a system is the expected one. Fortunately, Metamorphic Testing is a valuable and promising testing technique that alleviates the two fundamental problems of testing: the oracle problem and the reliable test set problem. In this paper, we combine Metamorphic Testing and simulation techniques for validating communication network configurations in HPC systems. For this, we rely on a catalogue of Metamorphic Relations, based on network communications knowledge, for checking their correctness. In addition, we have conducted an experimental study for analysing the communication networks of HPC systems. The results show that Metamorphic Testing is appropriate for checking the correctness of communication networks supported by complex topologies in HPC systems.

During the last decade, the research community has developed different simulation tools to model and study cloud systems. However, current cloud simulators focus on specific features that typically do not fully cover all aspects of the cloud infrastructure. The ever-growing number of existing simulators increases the difficulty to properly choose the most appropriate one. Moreover, in certain situations, these simulators must be combined to analyze the features required by the user, which leads to investing a considerable time and effort for their selection. In this paper, we propose CloudExpert, an intelligent system based on metamorphic testing that selects the most appropriate simulator covering the features of interest for the user. In contrast to our previous work, where metamorphic testing is applied to improve models representing a cloud, in this work we analyse the underlying features of several well-known cloud simulators to generate metamorphic rules, which are applied to represent the properties of the simulator. To show the applicability of CloudExpert, we conducted an empirical study where the adequacy of six well-known cloud simulators was analyzed. In this experiment, CloudExpert recommended the most appropriate simulator for eight scenarios involving different aspects of the cloud (energy, storage, network, memory, CPU) and simulator performance; and could also identify strengths and weaknesses of these simulators. Then, we further validated CloudExpert in two different ways. Firstly, the effectiveness of CloudExpert was measured using different faulty cloud simulators. Secondly, we designed a questionnaire based on the results provided by CloudExpert for some of the scenarios of the first experiment. The questionnaire was answered by eight experts in cloud simulation, confirming the usefulness of the tool.

Nowadays, cloud‐native software architectures have a significant relevance due to the speed and agility they provide. These properties lead relevant organizations in different industries, like video streaming (Netflix), car‐sharing (Uber, Cabify), banking (BBVA, HSBC), and governmental agencies (NASA, FBI, CERN, ESA) to heavily rely on cloud‐native software to run their business‐critical applications. Additionally, including fault injection actions in the production infrastructure allows companies to have consistent environments, to improve applications dependability against unexpected failures, to provide better user experience, and to improve the overall system quality. Thus, cloud computing technologies allow development teams to rapidly create complex systems and to continuously deploy them, at a global scale. This work describes Pystol, a novel fault injection platform—represented as a Software Product Line—to analyze the effects caused by a wide spectrum of adverse conditions. Pystol is designed to be executed on top of cloud‐native environments, either in private or public clouds. The proposed architecture shows a way for representing feature models based on Unified Model Language (in short, UML) component diagrams. Furthermore, we present a thorough empirical study carried out in real‐world environments, providing promising results.

Two crucial aspects for the trustworthy utilization of domain-specific languages (DSLs) are their semantic correctness, and proper testing support for their users. Testing is frequently used to verify correctness, but is often done informally -- which may yield unreliable results -- and requires substantial effort for creating suitable test cases and oracles. To alleviate this situation, we propose an automated technique for building metamorphic testing environments for DSLs. Metamorphic testing identifies expected relationships between the outputs of two consecutive tests, reducing the effort in specifying oracles and creating test cases manually. This new ideas paper presents the overarching concepts, the architecture and a prototype implementation. We illustrate our proposal using a DSL to model and simulate data centres.

In this paper, we model cloud systems and the user interactions with the cloud provider using the UML2Cloud profile.In general, users request virtual machines according to their needs, but they can also subscribe to the cloud provider and wait to be notified when the requested resources are not available.In this case, users indicate a maximum subscription time, so once this time elapses without being notified, users leave the system unattended.In this paper, then, we present an exhaustive research study to measure how the user subscription times affect the overall system responsiveness.In this study, three different cloud configurations are analyzed.Each cloud processes several workloads, which are generated using two distribution functions for the user arrivals, namely a normal and a cyclic normal distribution.The purpose of this study is to find out the inflection point for the waiting time of the users, from which the cloud responsiveness and its performance do not improve.The obtained information is therefore useful for the cloud provider to improve the configuration of the cloud.

Cloud providers face the challenge of managing large amounts of heterogeneous resources in real time. It is usually very costly to conduct experiments with real cloud systems. Therefore, tools to analyze and evaluate cloud scenarios and experimental studies are very useful for them. In this paper, we model cloud systems and the user interactions with the cloud provider using the UML2Cloud profile. In general, users request virtual machines according to their needs, but they can also subscribe to the cloud provider and wait to be notified when the requested resources are not available. In this case, users indicate a maximum subscription time, so once this time elapses without being notified, users leave the system unattended. Thus, we present an exhaustive experimental study to measure how the user subscription times affect the overall system responsiveness. To this end, four different cloud configurations are analyzed, and the workloads for these studies are produced by using three distribution functions for the user arrivals, namely, a uniform, a normal, and a cyclic normal distribution. Furthermore, we also analyze the cloud performance with a workload obtained from a real trace. The purpose of this study is to find out the inflection point for the waiting time of the users, from which the cloud responsiveness and its performance do not improve. The obtained information is, therefore, useful for the cloud provider to improve the configuration of the cloud.

Cloud computing has emerged as a cutting-edge technology which is widely used by both private and public institutions, since it eliminates the capital expense of buying, maintaining, and setting up both hardware and software. Clients pay for the services they use, under the so-called Service Level Agreements (SLAs), which are the contracts that establish the terms and costs of the services. In this paper, we propose the CloudCost UML profile, which allows the modeling of cloud architectures and the users’ behavior when they interact with the cloud to request resources. We then investigate how to increase the profits of cloud infrastructures by using price schemes. For this purpose, we distinguish between two types of users in the SLAs: regular and high-priority users. Regular users do not require a continuous service, so they can wait to be attended to. In contrast, high-priority users require a constant and immediate service, so they pay a greater price for their services. In addition, a computer-aided design tool, called MSCC (Modeling SLAs Cost Cloud), has been implemented to support the CloudCost profile, which enables the creation of specific cloud scenarios, as well as their edition and validation. Finally, we present a complete case study to illustrate the applicability of the CloudCost profile, thus making it possible to draw conclusions about how to increase the profits of the cloud infrastructures studied by adjusting the different cloud parameters and the resource configuration.

In this paper, we investigate how to improve the profits in cloud infrastructures by using price schemes and analyzing the user interactions with the cloud provider. For this purpose, we consider two different types of client behavior, namely regular and high-priority users. Regular users do not require a continuous service, and they can wait to be attended to. In contrast, high-priority users require a continuous service, e.g., a 24/7 service, and usually need an immediate answer to any request. A complete framework has been implemented, which includes a UML profile that allows us to define specific cloud scenarios and the automatic transformations to produce the code for the cloud simulations in the Simcan2Cloud simulator. The engine of Simcan2Cloud has also been modified by adding specific SLAs and price schemes. Finally, we present a thorough experimental study to analyze the performance results obtained from the simulations, thus making it possible to draw conclusions about how to improve the cloud profit for the cloud studied by adjusting the different parameters and resource configuration.

Currently, using conventional techniques for checking and optimising the energy consumption in cloud systems is unpractical, due to the massive computational resources required. An appropriate test suite focusing on the parts of the cloud to be tested must be efficiently synthesised and executed, while the correctness of the test results must be checked. Additionally, alternative cloud configurations that optimise the energetic consumption of the cloud must be generated and analysed accordingly, which is challenging. To solve these issues we present MT-EA4Cloud, a formal approach to check the correctness - from an energy-aware point of view - of cloud systems and optimise their energy consumption. To make the checking of energy consumption practical, MT-EA4Cloud combines metamorphic testing, evolutionary algorithms and simulation. Metamorphic testing allows to formally model the underlying cloud infrastructure in the form of metamorphic relations. We use metamorphic testing to alleviate both the reliable test set problem, generating appropriate test suites focused on the features reflected in the metamorphic relations, and the oracle problem, using the metamorphic relations to check the generated results automatically. MT-EA4Cloud uses evolutionary algorithms to efficiently guide the search for optimising the energetic consumption of cloud systems, which can be calculated using different cloud simulators. (C) 2020 Elsevier Inc. All rights reserved.

The validation of a cloud system can be complicated by the size of the system, the number of users that can concurrently request services, and the virtualization used to give the illusion of using dedicated machines. Unfortunately, it is not feasible to use conventional testing methods with cloud systems. This article proposes a framework, called TEA-Cloud, that integrates simulation with testing methods for validating cloud system designs. Testing is applied on both functional and nonfunctional aspects of the cloud, like performance and cost. The aim of the framework is to provide a complete methodology to help users to model both software and hardware parts of cloud systems and automatically test the validity of these clouds using a cost-effective approach. Metamorphic testing is used to overcome the lack of an oracle that checks whether the behavior observed in testing is allowed. Metamorphic testing is based on metamorphic relations (MRs). We define three families of MRs, which target issues such as performance, resource provisioning, and cost. TEA-Cloud was evaluated through an empirical study that used fault seeding (mutation) and ten MRs for testing different cloud configurations. The results were promising, with TEA-Cloud finding all seeded faults.

During the last few years, computer performance has reached a turning point where computing power is no longer the only important concern. This way, the emphasis is shifting from an exclusive focus on the optimisation of the computing system to optimising other systems, like the memory system. Broadly speaking, testing memory systems entails two main challenges: the oracle problem and the reliable test set problem. The former consists in deciding if the outputs of a test suite are correct. The latter refers to providing an appropriate test suite for determining the correctness of the system under test. In this paper we propose an expert system for checking the correctness of memory systems. In order to face these challenges, our proposed system combines two orthogonal techniques - simulation and metamorphic testing - enabling the automatic generation of appropriate test cases and deciding if their outputs are correct. In contrast to conventional expert systems, our system includes a factual database containing the results of previous simulations, and a simulation platform for computing the behaviour of memory systems. The knowledge of the expert is represented in the form of metamorphic relations, which are properties of the analysed system involving multiple inputs and their outputs. Thus, the main contribution of this work is two-fold: a method to automatise the testing process of memory systems, and a novel expert system design focusing on increasing the overall performance of the testing process. To show the applicability of our system, we have performed a thorough evaluation using 500 memory configurations and 4 different memory management algorithms, which entailed the execution of more than one million of simulations. The evaluation used mutation testing, injecting faults in the memory management algorithms. The developed expert system was able to detect over 99% of the critical injected faults, hence obtaining very promising results, and outperforming other standard techniques like random testing. (C) 2019 Elsevier Ltd. All rights reserved.

We introduce a probabilistic extension of our previous work SPLA: a formal framework to specify and analyze software product lines. We use probabilistic information to identify those features that are more frequently used. This is done by computing the probability of having a feature in a specific software product line, from now on SPLAP. We redefine the syntax of SPLA to include probabilistic operators and define new operational and denotational semantics. We prove that the expected equivalence between these two semantic frameworks holds. Our probabilistic framework is supported by a set of scripts to show the model behavior. We briefly comment on the characteristics of the scripts and discuss the advantages of using probabilities to quantify the likelihood of having features in potential software product lines.

Nowadays, cloud computing technologies are commonly used both by the public and private institutions, so as to save money and computing efforts, by using a network of remote servers hosted on the Internet to manage their services. Thus, cloud clients must interact with cloud providers in order to make use of the services deployed in the cloud, and the cloud infrastructure must be designed taking into account both the physical devices and equipment required (racks, switches, and so on) and also the user demands, expressed in terms of the services they need. This paper focuses on this latter aspect, in which we analyze how cloud clients interact with a cloud provider, providing a Unified Modeling Language-based framework to model a cloud system, including the underlying infrastructure, the user resource requirements, and their interactions with the cloud provider.