Engineers in other disciplines are faced with the problem of finite resources and the corresponding risk of overload in their engineered systems. Software engineering is no exception.

This edition of the “Practitioner’s Digest” features recent papers on artificial intelligence (AI) and machine learning (ML), along with papers on tech debt, energy consumption, and collaboration between industry and academia.

This edition of the “Practitioners’ Digest” covers an eclectic mix of topics from cannabis usage to world politics to large language models.

Despite the problems with software, the software services that we rely on every day tend to be up more often than they’re down. This is a mystery that requires explanation.

This special issue of IEEE Software includes ten feature articles on career development in software engineering.

This edition of the “Practitioner’s Digest” summarizes five recently published conference and journal pages on the topic of infrastructure-as-code.

Complex software systems have become increasingly entwined in a wide variety of systems, such as critical infrastructure, industrial control systems, medical devices, automobiles, airplanes, and spacecraft. Assuring the security and safety, as well as other dependability concerns, such as availability, robustness, and reliability, of software-intensive systems remains among the top priorities for governments and providers of critical systems and services. Manufacturers, owners, and operators of the components and devices that make up these software systems strive to ensure that they have adequately addressed emerging concerns related to safety hazards, security threats, and performance challenges, among others.

The theme of this issue is “Bots in Software Engineering,” and we’ve collected a number of recent papers about bots that interact with source code repositories. These papers were published at the fourth International Workshop on Bots in Software Engineering (BotSE ’22), the 37th ACM/SIGAPP Symposium on Applied Computing (SAC ’22), the International Conference on Information Systems (ICIS ’21), and the 19th International Conference on Mining Software Repositories (MSR ’22).

Following along with the theme of this issue of IEEE Software, this column reports papers about digital twins from the 2021 Empirical Assessment in Software Engineering (EASE’21) conference, the 2021 International Conference on Software Engineering (ICSE 2021), the 2021 International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS 2021), the 2021 European Conference on Software Architecture (ECSA 2021), and the 15th International Workshop on Models@Run.Time, a part of MODELS 2021. Feedback or suggestions are welcome. In addition, if you try or adopt any of the practices included in the column, please send us and the authors a note about your experiences.

DevOps connects development, delivery, and operations and, thus, facilitates a fluid collaboration of these traditionally separated silos. As an agile method, DevOps is today used across industries and is not limited anymore to IT services and specific technologies. Lorin Hochstein and I present a brief overview on DevOps best practice. A Netflix case study emphasizes DevOps in practice. I look forward to hearing from you about this column and the technologies that matter most for your work.

In honor of this issue’s theme of “Artificial Intelligence (AI) Engineering—Realizing the Potential of AI,” this edition of the “Practitioners’ Digest” brings you recent papers on AI and machine learning (ML) engineering, which we believe will be of interest to practitioners. These papers were published in the First International Conference on AI Engineering (CAIN 2022), the 26th International Conference on Evaluation and Assessment in Software Engineering 2022 (EASE 2022), and the 2022 IEEE/Association for Computing Machinery First International Workshop on Software Engineering for Responsible Artificial Intelligence (SE4RAI).

Distributed systems often face transient errors and localized component degradation and failure. Verifying that the overall system remains healthy in the face of such failures is challenging. At Netflix, we have built a platform for automatically generating and executing chaos experiments, which check how well the production system can handle component failures and slowdowns. This paper describes the platform and our experiences operating it.

While Chaos Engineering has gained currency in the Site Reliability Engineering community, service and business owners are often nervous about experimenting in production. Proving the benefits of Chaos Engineering to these stakeholders before implementing a program can be challenging. We present the business case for Chaos Engineering, through both qualitative and quantitative tactics, as well as the benefits and tools to convince stakeholders this is necessary and cost-efficient.

THIRTY YEARS AGO, Jim Gray noted that “A way to improve availability is to install proven hardware and software, and then leave it alone.”1 For companies that provide services over the Internet, “leaving it alone” isn’t an option. Such service providers must continually make changes to increase the service’s value, such as adding features and improving performance. At Netflix, engineers push new code into production and modify runtime configuration parameters hundreds of times a day. (For a look at Netflix and its system architecture, see the sidebar.) Availability is still important; a customer who can’t watch a video because of a service outage might not be a customer for long. But to achieve high availability, we need to apply a different approach than what Gray advocated. For years, Netflix has been running Chaos Monkey, an internal service that randomly selects virtualmachine instances that host our production services and terminates them.2 Chaos Monkey aims to encourage Netflix engineers to design software services that can withstand failures of individual instances. It’s active only during normal working hours so that engineers can respond quickly if a service fails owing to an instance termination. Chaos Monkey has proven successful; today all Netflix engineers design their services to handle instance failures as a matter of course. That success encouraged us to extend the approach of injecting failures into the production system to improve reliability. For example, we perform Chaos Kong exercises that simulate the failure of an entire Amazon EC2 (Elastic Compute Cloud) region. We also run Failure Injection Testing (FIT) exercises in which we cause requests between Netflix services to fail and verify that the system degrades gracefully.3 Over time, we realized that these activities share underlying themes that are subtler than simply “break things in production.” We also noticed that organizations such as Amazon,4 Google,4 Microsoft,5 and Facebook6 were applying similar techniques to test their systems’ resilience. We believe that these activities form part of a discipline that’s emerging in our industry; we call this discipline chaos engineering. Specifically, chaos engineering involves experimenting on a distributed system to build confidence in its capability to withstand turbulent conditions in production. These conditions could be anything from a hardware failure, to an unexpected surge in client requests, to a malformed value in a runtime configuration parameter. Our experience has led us to determine principles of chaos engineering (for an overview, see http://principlesofchaos .org), which we elaborate on here.

Modern software-based services are implemented as distributed systems with complex behavior and failure modes. Many large tech organizations are using experimentation to verify the reliability of such systems. We use the term "Chaos Engineering" to refer to this approach, and discuss the underlying principles and how to use it to run experiments.

Large-scale distributed systems must be built to anticipate and mitigate a variety of hardware and software failures. In order to build confidence that fault-tolerant systems are correctly implemented, Netflix (and similar enterprises) regularly run failure drills in which faults are deliberately injected in their production system. The combinatorial space of failure scenarios is too large to explore exhaustively. Existing failure testing approaches either randomly explore the space of potential failures randomly or exploit the "hunches" of domain experts to guide the search. Random strategies waste resources testing "uninteresting" faults, while programmer-guided approaches are only as good as human intuition and only scale with human effort.In this paper, we describe how we adapted and implemented a research prototype called lineage-driven fault injection (LDFI) to automate failure testing at Netflix. Along the way, we describe the challenges that arose adapting the LDFI model to the complex and dynamic realities of the Netflix architecture. We show how we implemented the adapted algorithm as a service atop the existing tracing and fault injection infrastructure, and present early results.

THIRTY YEARS AGO, Jim Gray noted that “A way to improve availability is to install proven hardware and software, and then leave it alone.”1 For companies that provide services over the Internet, “leaving it alone” isn’t an option. Such service providers must continually make changes to increase the service’s value, such as adding features and improving performance. At Netflix, engineers push new code into production and modify runtime configuration parameters hundreds of times a day. (For a look at Netflix and its system architecture, see the sidebar.) Availability is still important; a customer who can’t watch a video because of a service outage might not be a customer for long. But to achieve high availability, we need to apply a different approach than what Gray advocated. For years, Netflix has been running Chaos Monkey, an internal service that randomly selects virtualmachine instances that host our production services and terminates them.2 Chaos Monkey aims to encourage Netflix engineers to design software services that can withstand failures of individual instances. It’s active only during normal working hours so that engineers can respond quickly if a service fails owing to an instance termination. Chaos Monkey has proven successful; today all Netflix engineers design their services to handle instance failures as a matter of course. That success encouraged us to extend the approach of injecting failures into the production system to improve reliability. For example, we perform Chaos Kong exercises that simulate the failure of an entire Amazon EC2 (Elastic Compute Cloud) region. We also run Failure Injection Testing (FIT) exercises in which we cause requests between Netflix services to fail and verify that the system degrades gracefully.3 Over time, we realized that these activities share underlying themes that are subtler than simply “break things in production.” We also noticed that organizations such as Amazon,4 Google,4 Microsoft,5 and Facebook6 were applying similar techniques to test their systems’ resilience. We believe that these activities form part of a discipline that’s emerging in our industry; we call this discipline chaos engineering. Specifically, chaos engineering involves experimenting on a distributed system to build confidence in its capability to withstand turbulent conditions in production. These conditions could be anything from a hardware failure, to an unexpected surge in client requests, to a malformed value in a runtime configuration parameter. Our experience has led us to determine principles of chaos engineering (for an overview, see http://principlesofchaos .org), which we elaborate on here.

The Netflix video streaming system is composed of many interacting services. In such a large system, failures in individual services are not uncommon. This paper describes the Chaos Automation Platform, a system for running failure injection experiments on the production system to verify that failures in non-critical services do not result in system outages.

Among the many configuration management tools available, Ansible has some distinct advantages-it's minimal in nature, you don't need to install anything on your nodes, and it has an easy learning curve. This practical guide shows you how to be productive with this tool quickly, whether you're a developer deploying code to production or a system administrator looking for a better automation solution. Author Lorin Hochstein shows you how to write playbooks (Ansible's configuration management scripts), manage remote servers, and explore the tool's real power: built-in declarative modules. You'll discover that Ansible has the functionality you need and the simplicity you desire.