Widespread deployments of Network Function Virtualization (NFV) technology will replace many physical appliances in telecommunication networks with software executed on cloud platforms. Setting compute servers continuously to high-performance operating modes is a common NFV approach for achieving predictable operations. However, this has the effect that large amounts of energy are consumed even when little traffic needs to be forwarded. The Dynamic Voltage-Frequency Scaling (DVFS) technology available in Intel processors is a known option for adapting the power consumption to the workload, but it is not optimized for network traffic processing workloads. We developed a novel control method for DVFS, based observing the ongoing traffic and online predictions using machine learning. Our results show that we can save up to 27% compared to commodity DVFS, even when including the computational overhead of machine learning.

Energy efficiency and emissions awareness are core capabilities for sustainable and lower cost distributed cloud networks. In this context, metrics are fundamental for comparison and management purposes, along with the methods and tools which support such metrics’ capture and analysis. However, prior works on green metrics and tools have presented only a partial view, mainly as a result of the recent advances in green networking technologies. In this survey, we present an extensive study of metrics, methods, and tools to support sustainable operations in distributed cloud networks, with the aim of providing an end-to-end and up-to-date scenario to support current and coming research, as well as to analyze existing gaps.

The popularity of container technologies and their widespread usage for building microservices demands solutions dedicated for efficient monitoring of containers and their interactions. In this paper we present ConMon, an automated system for monitoring the network performance of container-based applications. It automatically identifies newly instantiated application containers and observes passively their traffic. Based on these observations, it configures and executes monitoring functions inside adjacent monitoring containers. The system adapts the monitoring containers to changes driven by either the application or the execution platform. The evaluation results validate the feasibility of the ConMon approach and illustrate its scalability in terms of low overhead on compute resources, moderate impact on applications, and negligible impact on the background network traffic.

SummaryNetwork function virtualization (NFV) allows to model network services as graphs interconnecting virtual network functions (VNFs), which may include nested VNFs, modeled as subgraphs of VNFs or end points. To query the performance data of network services modeled as abstract and high‐level graphs is challenging because of the recursivity of the NFV architecture and the elasticity and dynamicity provided by the NFV infrastructure. We propose to use Datalog, a declarative logic programming language, to build a framework that supports efficient data aggregation for performance metrics of recursively modeled network services. We present our recursive query language for automatic and flexible decomposition and aggregation of NFV performance metrics and describe example use cases for both compute and network metrics. We also describe the design and implementation of a proof‐of‐concept query engine using the language. Our performance evaluation shows that the total query latency is dominated by retrieval times of performance metrics from infrastructure databases, and the time for execution and automatic decomposition of high‐level queries by the query engine itself increases linearly with the size of the service graph up to 1000 nodes. Hence, our evaluations show that the query engine scales well, bounded mainly by the limited execution capabilities of our test environment. Furthermore, it can handle multiple concurrent queries up to the concurrency limits of the backend database in use. The proposed query language and engine are thus effective in recursively retrieving performance metrics of NFV environments supporting large‐scale service graphs and large numbers of query requests.

Although there is consensus that software defined networking and network functions virtualization overhaul service provisioning and deployment, the community still lacks a definite answer on how carrier-grade operations praxis needs to evolve. This article presents what lies beyond the first evolutionary steps in network management, identifies the challenges in service verification, observability, and troubleshooting, and explains how to address them using our Service Provider DevOps (SP-DevOps) framework. We compendiously cover the entire process from design goals to tool realization and employ an elastic version of an industry-standard use case to show how on-the-fly verification, software-defined monitoring, and automated troubleshooting of services reduce the cost of fault management actions. We assess SP-DevOps with respect to key attributes of software-defined telecommunication infrastructures both qualitatively and quantitatively, and demonstrate that SP-DevOps paves the way toward carrier-grade operations and management in the network virtualization era.

Determining an optimal placement for active measurement points in an arbitrary network topology is challenging. Software-defined infrastructure and the virtualization of network functions imply that re-optimized placement is needed frequently to keep up with dynamic changes in the infrastructure. We present a novel genetic algorithm that was defined for optimizing the placement of active measurement points in this environment. Initial results from simulations show that the method is effective and efficient in producing good solutions for four different topologies inspired from real networks. We also devised a strategy that enables faster reaction to incremental changes in the measured topology, reducing in half the execution time for two of the topologies.

As Software Defined Networks (SDN) deployments are reaching mainstream, network performance becomes a key concern for success. Service Providers (SPs) rely on network management capabilities, such as packet loss monitoring, to observe the network status and thereby facilitate service-level agreements. On one hand, SPs seek tools providing greater visibility into the status of their networks, but on the other hand, they are keen to limit the overhead of management capabilities in their operational networks. To meet these conflicting requirements, Efficient Passive Lightweight Estimator (EPLE) takes advantage of existing network traffic and SDN signaling, without the need of extra monitoring traffic or facilities. EPLE does not introduce any data plane overhead and the signaling overhead is reduced by locally creating microflow descriptors out of aggregated flow definitions. Our proof-of-concept prototype shows that EPLE can estimate packet loss rates accurately while keeping the processing and signaling overheads small compared to existing active measurement methods.

In Network Function Virtualization (NFV), a network service is created by combining interconnected virtual network functions (VNF), which may include nested VNFs or end points. Querying the performance of a high-level, abstract network service is challenging due to the recursivity of the NFV architecture and the elasticity and dynamicity provided by the NFV infrastructure, typically realized by Cloud virtualization technologies. In this paper, we propose to use Datalog, a declarative logic programming language, to build a query engine which can provide recursive query capabilities on performance metrics of network services. We present the language and describe some example use cases for both compute and network metrics. We describe the design of a query engine utilizing the language, based on which we implemented a proof of concept system. The resulting experimental system has shown the effectiveness of the query language to recursively retrieve monitoring results of NFV environments.

Network Function Virtualization (NFV) poses a number of monitoring challenges; one potential solution to these challenges is a recursive monitoring language. This document presents a set of requirements for such a recursive monitoring language.

Carrier-grade network management was optimized for environments built with monolithic physical nodes and involves significant deployment, integration and maintenance efforts from network service providers. The introduction of virtualization technologies, from the physical layer all the way up to the application layer, however, invalidates several well-established assumptions in this domain. This draft opens the discussion in NFVRG about challenges related to transforming the telecom network infrastructure into an agile, model-driven production environment for communication services. We take inspiration from data center DevOps regarding how to simplify and automate management processes for a telecom service provider software-defined infrastructure (SDI). Among the identified challenges, we consider scalability of observability processes and automated inference of monitoring requirements from logical forwarding graphs, as well as initial placement (and re-placement) of monitoring functionality following changes in flow paths enforced by the controllers. In another category of challenges, verifying correctness of behavior for network functions where flow rules are no longer necessary and sufficient for determining the forwarding state (for example, stateful firewalls or load balancers) is very difficult with current technology. Finally, we introduce challenges associated with operationalizing DevOps principles at scale in software-defined telecom networks in three areas related to key monitoring, verification and troubleshooting processes.

Green sustainability-oriented features have become common in network nodes and protocols. Running a network in an energy-efficient way is an important concern of network operators and datacenter networks. The implementation and coordination of the myriad of existing network features poses a challenging task. The energy efficiency capabilities must be selected according to the network conditions and can be combined to increase energy savings. However, they can conflict if not orchestrated in a proper manner. Policy-Based Network Management is a well-known approach to addressing the complexity of network management tasks. In conjunction with a refinement process to translate high-level policies down to low-level policies, it can bring business directives to the network, including sustainability goals. In this survey, we identify the major characteristics of sustainability-oriented policies, as well as the requirements a policy refinement method for such type of policies has to fulfill, including energy efficiency capabilities orchestration. We then analyze existing policy refinement techniques and discuss the challenges on how they address or need to be modified in order to be applicable to sustainability-oriented policies.

Passive monitoring of network performance parameters is experiencing a revival due to widespread adoption of virtualization and software-based implementation of network functions. Timestamping is one of the most challenging operations needed for passively monitoring network traffic performance parameters such as latency and jitter. We develop a setup whereby functions that monitor the network traffic are deployed in monitoring containers adjacently to, and interconnected through a virtual switch with the monitored Virtual Network Function instance. In this scenario, we evaluate the effects of container virtualization and virtual switch mirroring of traffic on the measurement of latency. The evaluation results indicate very low measurement errors (a few microseconds in our testbed) which are consistent over different measurement scenarios, thus validating the feasibility of this technique for passively monitoring latency.

The need to manage the energy consumption of network infrastructure has been addressed by a significant body of work in recent years. In general, energy management capabilities were developed independently and optimized for particular network layers and node features. The interaction between multiple such green capabilities when deployed simultaneously, as well as potential interactions with other existing functionality such as quality of service functions, need to be managed transparently by the operators. We developed SLIM, the SustainabiLity Information Model for Energy Efficiency Policies, as an add-on to the IETF Policy Core Information Model Extension to allow unifying the management of green capabilities throughout the network. We illustrate the flexibility of our approach by presenting a use case and describing an energy management system where SLIM was used.

Reducing the energy-related costs is a significant concern for Cloud Computing providers. Although many efficiency techniques had been evaluated, incentives for demand management according to energy sector information is not usual. The energy prices and availability vary according to daytime and geographical position, which implies in different prices for the resources in distinct periods. The higher flexibility of the Cloud Computing capacity regarding energy provider enables the adaptation of the infrastructure to take advantage of the energy prices fluctuation and the different sources availability. This work defines an infrastructure layer that allows negotiation and management of computing resources based on energy information for various time intervals. The proposed solution sets processing plans and establishes scheduling conditions to seize the most efficient usage of energy. A model that translates the customer contract information, searches for the lowest energy prices, and allocates the services based on these searches, is presented to reinforce our approach. Our results point out financial savings of almost 20% in comparison to non-energy management techniques.

The information and communications technologies (ICT) industry made significant steps towards higher efficiency through automation over the last decade. Still, carrier network operations are catching up with other ICT sectors. In this Special Issue, we use the term software-defined operations to refer to approaches that aim to further simplify and automate the daily ICT infrastructure run time and at the same time increasing reliability and enhancing visibility into the state of the system. While traditional observability methods focused on metrics such as delay, loss, or CPU utilization, enhancing the visibility includes exposing internal telemetry data such as buffer occupancy for switch chips and cache miss ratios for multi-core processors, as well as allowing to define aggregate metrics in software and computing them at data plane speeds. Looking back at the evolution of other industries, we could draw some parallels between current telecom operations and the state of vehicle manufacturing at the beginning of the 1970s when Japanese vendors introduced robots and production automation at a large scale. That step advance transformed the automobile industry as a whole and led to more reliable and safer cars. We believe that the ICT industries are headed towards a similar wave of transformation, and the development and adoption software-defined operations will play a central role in this process. Software-defined operations is part of the evolution that started with software-defined networking (SDN), a set of technologies that increased the degrees of flexibility and automation in the management and control planes of the network as discussed in RFC 7426. Control protocols such as OpenFlow and the domain-specific languages associated with configuration management tools such as Chef, Puppet, and Ansible are typical representatives of this approach. Different solutions have emerged using said tools and associated concepts to further optimize the infrastructure and increase the user friendliness towards developers and operations personnel. Software-defined operations are a stepping stone towards autonomic infrastructure, enabling operations personnel to elevate from mundane hands-on configuration tasks to an interaction driven by higher-layer management and control plane abstractions. This is a shift change compared with the myriad of variables and commands that need to be dealt with in daily routines and are currently part of various certification curricula for operations personnel. Using another automobile metaphor, think of competitive motorsports that have also changed much over the last decades. Yet higher-level abstractions (e.g., pressing buttons instead of manually shifting gears) did not eliminate drivers from Formula 1 teams—they just equipped them with more powerful knobs and levers that allowed them to focus on the situation on the track while driving at significantly higher speeds with greater safety. The road towards software-defined operations needs to overcome a series of challenges. Some of them are presented in a document discussed in the Network Function Virtualization Research Group (NFV RG) at the Internet Research Task Force (IRTF). 1 Some prominent challenges include trade-offs in terms of consistency, availability and partitioning (CAP) and observability methods that allow for transparent and programmable specification of compromises between accuracy and the overhead incurred by the infrastructure when measuring values of a certain key performance indicator (KPI). Verification methods adapted to the rapid cycles of resource scale up/down imposed by varying utilization patterns and operator policies were also identified as essential for software-defined operations based on recent research outcomes. 2 This Special Issue focuses on research results that potentially have wide applicability and relevance to simplifying operations in large-scale software-defined infrastructures. Through 2014–2016, the world experienced a significant number of natural disasters that disrupted deployed telecommunication infrastructure: earthquakes, hurricanes, and widespread forest fires. This Special Issue opens with the paper by Ying et al. entitled ‘Prediction Based Survivable Virtual Network Mapping against Disaster Failures’ that timely proposes a solution to optimize virtual network embedding to provide increased survivability in case of disasters. In contrast with the earlier literature, the authors assume that several nodes or links of the substrate network are affected by the disaster. They implement and evaluate two algorithms that attempt to minimize the loss of capacity under these constraints. In the second paper on software-defined operations, entitled ‘ReversePTP: A Clock Synchronization Scheme for Software Defined Networks’, Mizrahi et al. propose a clock synchronization scheme that takes advantage of the split architecture of OpenFlow networks to provide clock synchronization with an accuracy that matches that of the conventional clock synchronization protocol. Furthermore, as a significant part of the method is implemented in the network controller, it simplifies operations so that updates could be deployed faster and with less effort. Clock synchronization is an important enabler for automating operations, allowing the implementation of measurement capabilities and coordinated triggering of changes. The authors note that while the example implementation targets OpenFlow networks, other control protocols that implement a split control/data plane architecture could benefit from their results in environment such as industrial automation or power grid systems. Xuxia et al. in their paper entitled ‘FlowVisor-based cost-aware VN embedding in OpenFlow networks’ revisit the cost-aware network embedding problem and propose two efficient solutions. In terms of software-defined operations, these methods add clarity and transparency on the trade-offs that operators need to make when taking important network planning decisions. Their findings demonstrate the feasibility of delegating such decisions to an automated system while understanding the results and keeping control over the high-level parameters that steer run-time aspects. Finally, the Special Issue contains the paper entitled ‘A Flexible Information Service for Management of Virtualized Software-Defined Infrastructures’ by Mamatas et al. who develop and evaluate a solution for managing data generated by entities in a virtualized software-defined infrastructure. Designed as a flexible information service, the proposed solution was evaluated for scenarios that reflect two key properties: adaptability to new conditions and flexibility with respect to requirements for management data. We would like to express our appreciation to all authors who submitted their work in the journal's peer-review process, as well as the reviewers for their efforts to provide highly qualified reviews. Moreover, the editors would like to extend their gratitude to the editorial board of IJNM and, in particular, to James Won-Ki Hong and Filip De Turck for the great opportunity to serve as Guest Editors for this Special Issue on software-defined operations and for the prompt support during the submission and evaluation processes provided by Yoonseon Han. We acknowledge that part of the work of the editors has received funding from the European Union Seventh Framework Programme FP7/2007-2013 in the framework of the UNIFY project under grant agreement no. 619609.

This report presents the results of the UNIFY Service Provider DevOps activities. First, we present the final definition and assessment of the concept. SP-DevOps is realized by a combination of various functional components facilitating integrated service verification, efficient and programmable observability, and automated troubleshooting processes. Our assessment shows that SP-DevOps can help providers to reach a medium level of DevOps maturity and allows significant reduction in OPEX. Second, we focus on the evaluation of the proposed SP-DevOps components. The set of tools proposed supports ops and devs across all stages, with a focus on the deployment, operation and debugging phases, and allows to activate automated processes for operating NFV environments. Finally, we present use-cases and our demonstrators for selected process implementions, which allowed the functional validation of SP-DevOps.

The growing energy consumption has become a major concern for network service providers not only to reduce operational expenses, but also to minimize environmental problems related to the high energy expenditure. In this regard, assessing the energy consumed has become an important task to maximize energy efficiency, and thus, offer different green service levels for customers that desire to save energy. An SDN controller is presented that manages distinct green service levels based on power models to account for the consumed energy and energy saving capabilities. Specifically, we present three SLAs with distinct requirements in terms of energy consumption and models to account for power consumption for each network scenario. Our approach provides a fine-grained accounting of user power consumption in distinct network usage scenarios in order to increase the accuracy of GreenSLA management in next generation networks. The proposal is validated by emulating two use cases inspired on the Facebook topology.