A/B testing, also referred to as online controlled experimentation or continuous experimentation, is a form of hypothesis testing where two variants of a piece of software are compared in the field from an end user's point of view. A/B testing is widely used in practice to enable data-driven decision making for software development. While a few studies have explored different facets of research on A/B testing, no comprehensive study has been conducted on the state-of-the-art in A/B testing. Such a study is crucial to provide a systematic overview of the field of A/B testing driving future research forward. To address this gap and provide an overview of the state-of-the-art in A/B testing, this paper reports the results of a systematic literature review that analyzed primary studies. The research questions focused on the subject of A/B testing, how A/B tests are designed and executed, what roles stakeholders have in this process, and the open challenges in the area. Analysis of the extracted data shows that the main targets of A/B testing are algorithms, visual elements, and workflow and processes. Single classic A/B tests are the dominating type of tests, primarily based in hypothesis tests. Stakeholders have three main roles in the design of A/B tests: concept designer, experiment architect, and setup technician. The primary types of data collected during the execution of A/B tests are product/system data, user-centric data, and spatio-temporal data. The dominating use of the test results are feature selection, feature rollout, continued feature development, and subsequent A/B test design. Stakeholders have two main roles during A/B test execution: experiment coordinator and experiment assessor. The main reported open problems are related to the enhancement of proposed approaches and their usability. From our study we derived three interesting lines for future research: strengthen the adoption of statistical methods in A/B testing, improving the process of A/B testing, and enhancing the automation of A/B testing.

A/B testing is a common approach used in industry to facilitate innovation through the introduction of new features or the modification of existing software. Traditionally, A/B tests are administrated manually and conducted sequentially, with each experiment targeting the entire population that uses the corresponding application. This approach can be time-consuming and costly, particularly when the experiments are not relevant to the entire population. To tackle these problems, we present a new self-adaptive approach called AutoPABS, short for Automated Pipelines of A/B tests using Self-adaptation. AutoPABS contributes: (1) a self-adaptive architecture that enables automated execution of A/B testing pipelines, (2) a population split component that is managed by the managing system of the self-adaptive architecture to enable more efficient A/B testing pipeline execution, and (3) a notation to specify A/B testing pipelines that can be executed using the self-adaptive architecture. We started the evaluation with a small survey to probe the appraisal of the notation and infrastructure of AutoPABS. Then we performed a series of tests to measure the gains obtained by applying a population split in an automated A/B testing pipeline, using an extension of the SEAByTE artifact. The survey results show that the participants appraise the usefulness of automating A/B testing pipelines and population split. The test results show that automatically executing pipelines of A/B tests using the self-adaptation architecture with population split accelerates the identification of statistically significant results by exploiting parallel A/B test execution compared to traditional sequential execution of A/B tests.

Despite considerable research efforts on handling uncertainty in self-adaptive systems, a comprehensive understanding of the precise nature of uncertainty is still lacking. This paper summarises the findings of the 2023 Bertinoro Seminar on Uncertainty in Self- Adaptive Systems, which aimed at thoroughly investigating the notion of uncertainty, and outlining open challenges associated with its handling in self-adaptive systems. The seminar discussions were centered around five core topics: (1) agile end-toend handling of uncertainties in goal-oriented self-adaptive systems, (2) managing uncertainty risks for self-adaptive systems, (3) uncertainty propagation and interaction, (4) uncertainty in self-adaptive machine learning systems, and (5) human empowerment under uncertainty. Building on the insights from these discussions, we propose a research agenda listing key open challenges, and a possible way forward for addressing them in the coming years.

Internet of Things (IoT) networks consist of small devices that use a wireless communication to monitor and possibly control the physical world. A common threat to such networks are jamming attacks, a particular type of denial of service attack. Current research highlights the need for the design of more effective and efficient anti-jamming techniques that can handle different types of attacks in IoT networks. In this paper, we propose DeMiJA, short for Detection and Mitigation of Jamming Attacks in IoT, a novel approach to deal with different jamming attacks in IoT networks. DeMiJA leverages architecture-based adaptation and the MAPE-K reference model (Monitor-Analyze-Plan-Execute that share Knowledge). We present the general architecture of DeMiJA and instantiate the architecture to deal with jamming attacks in the DeltaIoT exemplar. The evaluation shows that DeMiJA can handle different types of jamming attacks effectively and efficiently, with neglectable overhead.

When a self-adaptive system detects that its adaptation goals may be compromised, it needs to determine how to adapt to ensure its goals. To that end, the system can analyze the possible options for adaptation, i.e., the adaptation space, and pick the best option that achieves the goals. Such analysis can be resource and time consuming, in particular when rigorous analysis methods are applied. Hence, exhaustively analyzing all options may be infeasible for systems with large adaptation spaces. This problem is further complicated as the adaptation options typically include uncertainty parameters that can only be resolved at runtime. In this paper, we present a machine learning approach to tackle this problem. This approach enhances the traditional MAPE-K feedback loop with a learning module that selects subsets of adaptation options from a large adaptation space to support the analyzer with performing efficient analysis. We instantiate the approach for two concrete learning techniques, classification and regression, and evaluate the approaches for two instances of an Internet of Things application for smart environment monitoring with different sizes of adaptation spaces. The evaluation shows that both learning approaches reduce the adaptation space significantly without noticeable effect on realizing the adaptation goals.

Artifacts support evaluating new research results and help comparing them with the state of the art in a field of interest. Over the past years, several artifacts have been introduced to support research in the field of self-adaptive systems. While these artifacts have shown their value, it is not clear to what extent these artifacts support research on problems in self-adaptation that are relevant to industry. This paper provides a set of guidelines for artifacts that aim at supporting industry-relevant research on self-adaptation. The guidelines that are grounded on data obtained from a survey with practitioners were derived during working sessions at the 17th International Symposium on Software Engineering for Adaptive and Self-Managing Systems. Artifact providers can use the guidelines for aligning future artifacts with industry needs; they can also be used to evaluate the industrial relevance of existing artifacts. We also propose an artifact template.

Micro-services are a common architectural approach to software development today. An indispensable tool for evolving micro-service systems is A/B testing. In A/B testing, two variants, A and B, are applied in an experimental setting. By measuring the outcome of an evaluation criterion, developers can make evidence-based decisions to guide the evolution of their software. Recent studies highlight the need for enhancing the automation when such experiments are conducted in iterations. To that end, we contribute a novel artifact that aims at enhancing the automation of an experimentation pipeline of a micro-service system relying on the principles of self-adaptation. Concretely, we propose SEAByTE, an experimental framework for testing novel self-adaptation solutions to enhance the automation of continuous A/B testing of a micro-service based system. We illustrate the use of the SEAByTE artifact with a concrete example.

Many software systems today face uncertain operating conditions, such as sudden changes in the availability of resources or unexpected user behavior. Without proper mitigation these uncertainties can jeopardize the system goals. Self-adaptation is a common approach to tackle such uncertainties. When the system goals may be compromised, the self-adaptive system has to select the best adaptation option to reconfigure by analyzing the possible adaptation options, i.e., the adaptation space. Yet, analyzing large adaptation spaces using rigorous methods can be resource- and time-consuming, or even be infeasible. One approach to tackle this problem is by using online machine learning to reduce adaptation spaces. However, existing approaches require domain expertise to perform feature engineering to define the learner and support online adaptation space reduction only for specific goals. To tackle these limitations, we present “Deep Learning for Adaptation Space Reduction Plus”—DLASeR+ for short. DLASeR+ offers an extendable learning framework for online adaptation space reduction that does not require feature engineering, while supporting three common types of adaptation goals: threshold, optimization, and set-point goals. We evaluate DLASeR+ on two instances of an Internet-of-Things application with increasing sizes of adaptation spaces for different combinations of adaptation goals. We compare DLASeR+ with a baseline that applies exhaustive analysis and two state-of-the-art approaches for adaptation space reduction that rely on learning. Results show that DLASeR+ is effective with a negligible effect on the realization of the adaptation goals compared to an exhaustive analysis approach and supports three common types of adaptation goals beyond the state-of-the-art approaches.

With the increasing ubiquity and scale of self-adaptive systems, there is a growing need to decentralize the functionality that realizes self-adaptation. Our focus is on architecture-based self-adaptive systems where one or more functions for monitoring, analyzing, planning, and executing are realized by multiple components that coordinate with one another. While some earlier studies have shed light on existing work on the decentralization of self-adaptive systems, there is currently no clear overview of the state of the art in decentralization of self-adaptive systems. Yet, having a precise view on the state of the art in decentralized self-adaptive systems is crucial for researchers to understand existing solutions and drive future research efforts. To address this gap, we conducted a mapping study. The study focused on papers published at 24 important venues that publish research on self-adaptation. The study focused on the motivations for choosing a decentralized approach to realize self-adaptation, the adaptation functions that are decentralized, the realization of the coordination, and the open challenges in the area.

Recently, we have been witnessing an increasing use of machine learning methods in self-adaptive systems. Machine learning methods offer a variety of use cases for supporting self-adaptation, e.g., to keep runtime models up to date, reduce large adaptation spaces, or update adaptation rules. Yet, since machine learning methods apply in essence statistical methods, they may have an impact on the decisions made by a self-adaptive system. Given the wide use of formal approaches to provide guarantees for the decisions made by self-adaptive systems, it is important to investigate the impact of applying machine learning methods when such approaches are used. In this paper, we study one particular instance that combines linear regression to reduce the adaptation space of a self-adaptive system with statistical model checking to analyze the resulting adaptation options. We use computational learning theory to determine a theoretical bound on the impact of the machine learning method on the predictions made by the verifier. We illustrate and evaluate the theoretical result using a scenario of the DeltaIoT artifact. To conclude, we look at opportunities for future research in this area.

Recently, we have been witnessing a rapid increase in the use of machine learning techniques in self-adaptive systems. Machine learning has been used for a variety of reasons, ranging from learning a model of the environment of a system during operation to filtering large sets of possible configurations before analyzing them. While a body of work on the use of machine learning in self-adaptive systems exists, there is currently no systematic overview of this area. Such an overview is important for researchers to understand the state of the art and direct future research efforts. This article reports the results of a systematic literature review that aims at providing such an overview. We focus on self-adaptive systems that are based on a traditional Monitor-Analyze-Plan-Execute (MAPE)-based feedback loop. The research questions are centered on the problems that motivate the use of machine learning in self-adaptive systems, the key engineering aspects of learning in self-adaptation, and open challenges in this area. The search resulted in 6,709 papers, of which 109 were retained for data collection. Analysis of the collected data shows that machine learning is mostly used for updating adaptation rules and policies to improve system qualities, and managing resources to better balance qualities and resources. These problems are primarily solved using supervised and interactive learning with classification, regression, and reinforcement learning as the dominant methods. Surprisingly, unsupervised learning that naturally fits automation is only applied in a small number of studies. Key open challenges in this area include the performance of learning, managing the effects of learning, and dealing with more complex types of goals. From the insights derived from this systematic literature review, we outline an initial design process for applying machine learning in self-adaptive systems that are based on MAPE feedback loops.

Self-adaptation is a widely accepted approach to deal with uncertainties that are difficult to anticipate before deployment. We focus on architecture-based adaptation that relies on a feedback loop that reasons over architectural models of the system at runtime to make adaptation decisions. In particular, we study decentralized self-adaptive systems where self-adaptation is realized through multiple coordinating feedback loops. Such decentralization is crucial for systems where adaptation decisions cannot be made in a centralized way, such as in large scale Internet of Things (IoT). State of the art in this area is limited to either conceptual ideas or solutions dedicated to particular settings. This paper outlines a research project targeting the research question: “how to model and realize decentralized feedback loops that are capable to guarantee compliance of system goals in an efficient way despite uncertainties the system faces?” We plan to answer this question in two stage. First, we study commonalities and variability of decentralized self-adaptive systems leveraging on patterns and coordination mechanisms, and reify our insights in a framework. Second, we study language support for the design and implementation of decentralized self-adaptation, capitalizing on the outcome of the first stage. To ensure guarantees for the qualities we will found our work on formal techniques. To ensure efficiency, we will combine statistical techniques with machine learning. We plan to validate the research results in two domains: IoT and multi-cloud systems.

Recently, we have been witnessing a rapid increase in the use of machine learning techniques in self-adaptive systems. Machine learning has been used for a variety of reasons, ranging from learning a model of the environment of a system during operation to filtering large sets of possible configurations before analyzing them. While a body of work on the use of machine learning in self-adaptive systems exists, there is currently no systematic overview of this area. Such an overview is important for researchers to understand the state of the art and direct future research efforts. This article reports the results of a systematic literature review that aims at providing such an overview. We focus on self-adaptive systems that are based on a traditional Monitor-Analyze-Plan-Execute (MAPE)-based feedback loop. The research questions are centered on the problems that motivate the use of machine learning in self-adaptive systems, the key engineering aspects of learning in self-adaptation, and open challenges in this area. The search resulted in 6,709 papers, of which 109 were retained for data collection. Analysis of the collected data shows that machine learning is mostly used for updating adaptation rules and policies to improve system qualities, and managing resources to better balance qualities and resources. These problems are primarily solved using supervised and interactive learning with classification, regression, and reinforcement learning as the dominant methods. Surprisingly, unsupervised learning that naturally fits automation is only applied in a small number of studies. Key open challenges in this area include the performance of learning, managing the effects of learning, and dealing with more complex types of goals. From the insights derived from this systematic literature review, we outline an initial design process for applying machine learning in self-adaptive systems that are based on MAPE feedback loops.

When a self-adaptive system needs to adapt, it has to analyze the possible options for adaptation, i.e., the adaptation space. For systems with large adaptation spaces, this analysis process can be resource- and time-consuming. One approach to tackle this problem is using machine learning techniques to reduce the adaptation space to only the relevant adaptation options. However, existing approaches only handle threshold goals, while practical systems often need to address also optimization goals. To tackle this limitation, we propose a two-stage learning approach called Deep Learning for Adaptation Space Reduction (DLASeR). DLASeR applies a deep learner first to reduce the adaptation space for the threshold goals and then ranks these options for the optimization goal. A benefit of deep learning is that it does not require feature engineering. Results on two instances of the DeltaIoT artifact (with different sizes of adaptation space) show that DLASeR outperforms a state-of-the-art approach for settings with only threshold goals. The results for settings with both threshold goals and an optimization goal show that DLASeR is effective with a negligible effect on the realization of the adaptation goals. Finally, we observe no noteworthy effect on the effectiveness of DLASeR for larger sizes of adaptation spaces.