Performance testing with the aim of generating an efficient and effective workload to identify performance issues is challenging. Many of the automated approaches mainly rely on analyzing system models, source code, or extracting the usage pattern of the system during the execution. However, such information and artifacts are not always available. Moreover, all the transactions within a generated workload do not impact the performance of the system the same way, a finely tuned workload could accomplish the test objective in an efficient way. Model-free reinforcement learning is widely used for finding the optimal behavior to accomplish an objective in many decision-making problems without relying on a model of the system. This paper proposes that if the optimal policy (way) for generating test workload to meet a test objective can be learned by a test agent, then efficient test automation would be possible without relying on system models or source code. We present a self-adaptive reinforcement learning-driven load testing agent, RELOAD, that learns the optimal policy for test workload generation and generates an effective workload efficiently to meet the test objective. Once the agent learns the optimal policy, it can reuse the learned policy in subsequent testing activities. Our experiments show that the proposed intelligent load test agent can accomplish the test objective with lower test cost compared to common load testing procedures, and results in higher test efficiency.

Test automation has been an acknowledged software engineering best practice for years. However, the topic involves more than the repeated execution of test cases that often comes first to mind. Simply running test cases using a unit testing framework is no longer enough for test automation to keep up with the ever-shorter release cycles driven by continuous deployment and technological innovations such as microservices and DevOps pipelines. Now test automation needs to rise to the next level by going beyond mere test execution.

Testing resource allocation is the problem of planning the assignment of resources to testing activities of software components so as to achieve a target goal under given constraints. Existing methods build on software reliability growth models (SRGMs), aiming at maximizing reliability given time/cost constraints, or at minimizing cost given quality/time constraints. We formulate it as a multiobjective debug-aware and robust optimization problem under uncertainty of data, advancing the state-of-the-art in the following ways. Multiobjective optimization produces a set of solutions, allowing to evaluate alternative tradeoffs among reliability, cost, and release time. Debug awareness relaxes the traditional assumptions of SRGMs-in particular the very unrealistic immediate repair of detected faults-and incorporates the bug assignment activity. Robustness provides solutions valid in spite of a degree of uncertainty on input parameters. We show results with a real-world case study.

The importance of efficient software testing procedures is driven by an ever increasing system complexity as well as global competition. In the particular case of manual test cases at the system integration level, where thousands of test cases may be executed before release, time must be well spent in order to test the system as completely and as efficiently as possible. Automating a subset of the manual test cases, i.e, translating the manual instructions to automatically executable code, is one way of decreasing the test effort. It is further common that test cases exhibit similarities, which can be exploited through reuse when automating a test suite. In this paper, we investigate the potential for reducing test effort by ordering the test cases before such automation, given that we can reuse already automated parts of test cases. In our analysis, we investigate several approaches for prioritization in a case study at a large Swedish vehicular manufacturer. The study analyzes the effects with respect to test effort, on four projects with a total of 3919 integration test cases constituting 35,180 test steps, written in natural language. The results show that for the four projects considered, the difference in expected manual effort between the best and the worst order found is on average 12 percentage points. The results also show that our proposed prioritization method is nearly as good as more resource demanding meta-heuristic approaches at a fraction of the computational time. Based on our results, we conclude that the order of automation is important when the set of test cases contain similar steps (instructions) that cannot be removed, but are possible to reuse. More precisely, the order is important with respect to how quickly the manual test execution effort decreases for a set of test cases that are being automated.

In testing, engineers want to run the most useful tests early (prioritization). When tests are run hundreds or thousands of times, minimizing a test set can result in significant savings (minimization). This paper proposes a new analysis technique to address both the minimal test set and the test case prioritization problems. This paper precisely defines the concept of mutant stubbornness, which is the basis for our analysis technique. We empirically compare our technique with other test case minimization and prioritization techniques in terms of the size of the minimized test sets and how quickly mutants are killed. We used seven C language subjects from the Siemens Repository, specifically the test sets and the killing matrices from a previous study. We used 30 different orders for each set and ran every technique 100 times over each set. Results show that our analysis technique performed significantly better than prior techniques for creating minimal test sets and was able to establish new bounds for all cases. Also, our analysis technique killed mutants as fast or faster than prior techniques. These results indicate that our mutant stubbornness technique constructs test sets that are both minimal in size, and prioritized effectively, as well or better than other techniques.

In this scenario, the adoption of Mutation Analysis increasingly demands the definition of specific mutation operators for these new features. It is desirable that these new operators can be added to the mutation environment for reproducing those new errors that appear over time.The main contributions of this paper are:(1) the development of a hierarchical architecture for mutation operators that minimizes the dependence of external libraries and that facilitates the implementation of new operators;(2) BacterioWeb, a new mutation tool that performs all the mutation tasks in the web;(3) some mutation operators specifically designed for reproducing some common context-aware errors reported by mobile applications developers;(4) a comparison of the proposed architecture with those in other tools.

Mutation analysis has proven to be a strong technique for software testing. Unfortunately, it is also computationally expensive and researchers have therefore proposed several different approaches to reduce the effort. None of these reduction techniques however, focuses on non-functional properties. Given that our goal is to create a strong test suite for testing a certain non-functional property, which mutants should be used? In this paper, we introduce the concept of targeted mutation, which focuses mutation effort to those parts of the code where a change can make a difference with respect to the targeted non-functional property. We show how targeted mutation can be applied to derive efficient test suites for estimating the Worst-Case Execution Time (WCET). We use program slicing to direct the mutations to the parts of the code that are likely to have the strongest influence on execution time. Finally, we outline an experimental procedure for how to evaluate the technique.

Managing a software architecture after the deployment phase is a very complex task due to frequent changes in the software requirements and environment. The software architecture must evolve in order to tackle such changes. The goal of this paper is to provide support for the decisions that software architects make after deployment. This approach is based on an optimization model whose solution suggests the “best” actions to be taken according to a given change scenario (i.e., a set of new requirements that induce changes in the structural and behavioral aspects of the software architecture). The model aims to minimize the costs while keeping the reliability and the performance of the software architecture within certain thresholds. The approach has been implemented as a framework named SHEPhERd, which is composed of a UML case tool, a model builder and a model solver. We show how SHEPhERd works on a smartphone mobile application example, and we highlight its potential to drive architectural decisions through sensitivity analysis. The achieved results are compared with those obtained by two groups of (human) maintainers composed of experts and non-experts with respect to the system and the execution environment, and we show that SHEPhERd outperforms the human judgment-based approaches.

In Web services (WS), software applications are dynamically built by assembling over a network existing, loosely coupled, distributed, and heterogeneous services. Reliability is one of the most important quality dimensions for Web services, since predicting their reliability is fundamental to appropriately drive the selection and the assembly of services. This chapter presents two approaches to predict the reliability of a Web service architecture. The first one is based on the Business Process Execution Language (BPEL), the de facto standard executable language for specifying actions within business processes with Web services. The second one is based on the SCA-ASM, a lightweight formal language for modeling service-oriented applications, which is based on the OASIS (Organization for the Advancement of Structured Information Standards) standard Service Component Architecture for heterogeneous service assembly and on the formal method abstract state machines (ASMs) for modeling service behavior, interactions, and orchestration in an abstract but executable way. Through a set of experimental results, we show how the two models work on a smartphone mobile application example, and we discuss the effectiveness of the SCA-ASM approach in comparison with the BPEL-based approach.

This work analyzes the challenges that quality decisions represent to software project managers. Projects’ goals are normally determined by the paradigm of the Iron Triangle of project management. Managers need to know which are the effects of a quality assurance (QA) decision on the three axis: which effects in quality they can get but at what cost and which effects may appear in terms of schedule. This decision problem is clearly related to existing disciplines like SBSE, multi-objective optimization and methods for ROI calculation and value-based software engineering. This survey paper critically reviews the contributions of these disciplines to support QA decisions together with basic information from a pilot survey carried out as part of the developments of the Iceberg project funded by EU Programme Marie Curie.

Software Product Lines (SPL) approach has been widely developed in academia and successfully applied in industry. Based on the selection of features, stakeholders can efficiently derive tailor-made programs satisfying different requirements. While SPL was very successful at building products based on identified features, achievements and preservation of many nonfunctional properties (NFPs) remain challenging. A knowledge how to deal with NFPs is still not fully obtained. In this paper, we present a systematic literature review of NFPs analysis for SPL products, focusing on runtime NFPs. The goal of the paper is twofold: (i) to present an holistic overview of SPL approaches that have been reported regarding the analysis of runtime NFPs, and (ii) to categorize NFPs treated in the scientific literature regarding development of SPLs. We analyzed 36 research papers, and identified that system performance attributes are typically the most considered. The results also aid future research studies in NFPs analysis by providing an unbiased view of the body of empirical evidence and by guiding future research directions.

Project ICEBERG investigated a novel approach to improving understanding of the real cost impacts of poor quality software and supporting the suite of management decisions required to take corrective action across the entire software development cycle.

Service-oriented applications may require adaptation to tackle changing user needs, system intrusions or faults, changing operational environment, resource variability, etc. In order to achieve the right trade off among the functional requirements, software qualities (such as performance and reliability) and the adaptation cost itself, the adaptation decisions should involve the (a priori) evaluation of new alternatives to the current application design. However, the generation and evaluation of design alternatives is often time-consuming, it can be error-prone and can lead to suboptimal design decisions, especially if carried out manually by system maintainers.This article proposes an automatic optimization process for adaptation space exploration of service-oriented applications based on trade-offs between functional and extra-functional requirements. The proposed method combines the use of metaheuristic search techniques and functional/extra-functional patterns (i.e., architectural design patterns and tactics). Besides, the proposed methodology relies on the standard Service Component Architecture (SCA) for heterogeneous service assembly and its runtime platforms. As a proof-of-concept, this article provides also an example of instantiation of the process together with an experimentation on a sample application and a numerical evaluation of the scalability of the approach. We propose a process for the adaptation of service-oriented applications.It is based on an optimization method.It is based on trade-offs between functional and extra-functional requirements.It uses metaheuristic search techniques and functional/extra-functional patterns.

In service-oriented computing, software applications are dynamically built by assembling existing, loosely-coupled, distributed, and heterogeneous services. Predicting their reliability is important to appropriately drive the selection and assembly of services. This paper presents an approach to predict the reliability of a service component architecture. We adopt a lightweight formal component model, SCA-ASM, as core modeling technique for both architecture and behavior, supported by a run-time platform. This component model is based on the OASIS standard Service Component Architecture for heterogeneous service assembly and on the formal method Abstract State Machines for modeling service behavior, interactions, and orchestration in an abstract but executable way. The proposed reliability prediction method exploits ideas from architecture-based and path-based reliability models.

Growing advances in hardware technologies are enabling significant improvements in application performance by the deployment of components to dedicated executable units. This is particularly valid for Cyber Physical Systems in which the applications are partitioned in HW and SW execution units. The growing complexity of such systems, and increasing requirements, both project- and product-related, makes the partitioning decision process complex. Although different approaches to this decision process have been proposed during recent decades, they lack the ability to provide relevant decisions based on a larger number of requirements and project/business constraints. A sound approach to this problem is taking into account all relevant requirements and constraints and their relations to the properties of the components deployed either as HW or SW units. A typical approach for managing a large number of criteria is a multicriteria decision analysis. This, in its turn, requires uniform definitions of component properties and their realization in respect to their HW/SW deployment. The aim of this paper is twofold: a) to provide an architectural metamodel of component-based applications with specifications of their properties with respect to their partitioning, and b) to categorize component properties in relation to HW/SW deployment. The metamodel enables the transition of system requirements to system and component properties. The categorization provides support for architectural decisions. It is demonstrated through a property guideline for the partitioning of the System Automation and Control domain. The guideline is based on interviews with practitioners and researchers, the experts in this domain.

The analysis of non-functional properties of Service-Based Systems (SBSs) is a complex task, mostly because it requires models that encompass the composition of service properties into architectural properties. For example, the reliability of a SBS is given by the composition of service and interconnection reliabilities. Although several approaches have been introduced in the last few years to address these issues, the tradeoff analysis among non-functional properties of software services has not yet been studied enough. The goal of this paper is to introduce a set of optimization models that allow quantifying the costs of service failure repair/mitigation actions aimed at keeping the whole SBS reliability over a certain threshold. On the basis of our previous work in this area, we first introduce an optimization model aimed at selecting either in-house built or provided services with the goal of minimizing the SBS cost while guaranteeing a certain level of reliability. Thereafter we strengthen the reliability constraints, and we build two different optimization models that aim to solve the same problem under new constraints, where one model starts from the solution obtained in the original model and tries to improve it, while the other one looks for an optimal solution in the whole search space. Finally, we introduce a fourth model, based on stochastic optimization, with the goal of rather searching for solutions that explicitly take into account the stochastic nature of the problem and search for new repair/mitigation actions cheaper than the ones identified by the other models. Each optimization model has been experimented on about 300 variations of a nominal model. The experimental results show the efficacy of our optimization models to quantify the costs of different failure repairing/mitigation actions in different contexts.