The “HPI Future SOC Lab” is a cooperation of the Hasso Plattner Institute (HPI) and industry partners. Its mission is to enable and promote exchange and interaction between the research community and the industry partners. The HPI Future SOC Lab provides researchers with free of charge access to a complete infrastructure of state of the art hard and software. This infrastructure includes components, which might be too expensive for an ordinary research environment, such as servers with up to 64 cores and 2 TB main memory. The offerings address researchers particularly from but not limited to the areas of computer science and business information systems. Main areas of research include cloud computing, parallelization, and In-Memory technologies. This technical report presents results of research projects executed in 2017. Selected projects have presented their results on April 25th and November 15th 2017 at the Future SOC Lab Day events.

DevOps is a modern software engineering paradigm that is gaining widespread adoption in industry. The goal of DevOps is to bring software changes into production with a high frequency and fast feedback cycles. This conflicts with software quality assurance activities, particularly with respect to performance. For instance, performance evaluation activities --- such as load testing --- require a considerable amount of time to get statistically significant results. We conducted an industrial survey to get insights into how performance is addressed in industrial DevOps settings. In particular, we were interested in the frequency of executing performance evaluations, the tools being used, the granularity of the obtained performance data, and the use of model-based techniques. The survey responses, which come from a wide variety of participants from different industry sectors, indicate that the complexity of performance engineering approaches and tools is a barrier for wide-spread adoption of performance analysis in DevOps. The implication of our results is that performance analysis tools need to have a short learning curve, and should be easy to integrate into the DevOps pipeline in order to be adopted by practitioners.

DevOps is a modern software engineering paradigm that is gaining widespread adoption in industry. The goal of DevOps is to bring software changes into production with a high frequency and fast feedback cycles. This conflicts with software quality assurance activities, particularly with respect to performance. For instance, performance evaluation activities --- such as load testing --- require a considerable amount of time to get statistically significant results. We conducted an industrial survey to get insights into how performance is addressed in industrial DevOps settings. In particular, we were interested in the frequency of executing performance evaluations, the tools being used, the granularity of the obtained performance data, and the use of model-based techniques. The survey responses, which come from a wide variety of participants from different industry sectors, indicate that the complexity of performance engineering approaches and tools is a barrier for wide-spread adoption of performance analysis in DevOps. The implication of our results is that performance analysis tools need to have a short learning curve, and should be easy to integrate into the DevOps pipeline in order to be adopted by practitioners.

Modern applications are typically implemented as distributed systems comprising several components. Deciding where to deploy which component is a difficult task that today is usually assisted by logical topology recommendations. Choosing inefficient topologies allocates the wrong amount of resources, leads to unnecessary operation costs, or results in poor performance. Testing different topologies to find good solutions takes a lot of time and might delay productive operations. Therefore, this work introduces a software-based deployment topology optimization approach for distributed applications. We use an enhanced performance model generator that extracts models from operational monitoring data of running applications. The extracted model is used to simulate performance metrics (e.g., resource utilization, response times, throughput) and runtime costs of distributed applications. Subsequently, we introduce a deployment topology optimizer, which selects an optimized topology for a specified workload and considers on-premise, cloud, and hybrid topologies. The following three optimization goals are presented in this work: (i) minimum response time for an optimized user experience, (ii) approximate resource utilization around certain peaks, and (iii) minimum cost for running the application. To evaluate the approach, we use the SPECjEnterpriseNEXT industry benchmark as distributed application in an on-premise and in a cloud/on-premise hybrid environment. The evaluation demonstrates the accuracy of the simulation compared to the actual deployment by deploying an optimized topology and comparing measurements with simulation results.

Performance models focus on resource consumption and the effects of CPU, network, or hard-disk utilization. These resources usually have the largest effect on the response times and throughput of an application. However, deficient memory management can have severe effects on an application and its runtime, such as overlong response times or even crashes. As memory management has been disregarded in performance simulations, we address this gap with an approach based on memory measurements and derived metrics to predict the behavior of this resource and the effects on the CPU. Although numerous works exist that analyze memory management and especially garbage collections, accurate prediction models are rare. We demonstrate the automatic extraction of memory behavior using a performance model generator. Furthermore, the approach is evaluated using the SPECjEnterprise2010 and the SPECjEnterpriseNEXT industry benchmark, using different resource environments, garbage collection algorithms, and workloads. This work demonstrates that a certain set of probabilities allows one to create a memory profile for an architecture and predict the behavior of the memory management. The results of such predictions can be used for better capacity planning (on-premise), cost-prediction (cloud), architecture evaluation and optimization, or memory profiling. This approach allows for a continuous model-based evaluation of an enterprise architecture regarding its memory footprint.

Setting up an experimental evaluation for architecture-based Software Performance Engineering (SPE) approaches requires enormous efforts. This includes the selection and installation of representative applications, usage profiles, supporting tools, infrastructures, etc. Quantitative comparisons with related approaches are hardly possible due to limited repeatability of previous experiments by other researchers. This paper presents CASPA, a ready-to-use and extensible evaluation platform that already includes example applications and state-of-the-art SPE components, such as monitoring and model extraction. The platform explicitly provides interfaces to replace applications and components by custom(ized) components. The platform builds on state-of-the-art technologies such as container-based virtualization.

Algorithms processed by Graphics Processing Units (GPU) became popular recently. Bitcoin mining algorithms, image processing and all types of machine learning are famous examples for that. Infrastructure-as-a-Service provider picked up this trend and offer graphics processing power as part of their service port-folio. The performance gains when choosing a GPU implementation can be enormous. Designing and implementing a GPU-depended algorithm has some fundamental differences compared to classical algorithms, but not all algorithmic problems benefit from GPU usage regarding the overall performance and response time. Especially the interaction between Central Processing Unit (CPU) and GPU must be considered as it can become a bottleneck. Predicting and comparing the performance of GPU-depended applications in combination with their corresponding CPUs allows to assist design decisions in modern applications. In this work, we present concepts on how to predict al-gorithm performance relying on GPU processing and their relationship with the CPU using the Palladio Component Model and the Palladio Bench.

Enterprise applications are typically implemented as distributed systems composed of several components. Deciding where to deploy which component is a difficult task that today is usually assisted by logical topology recommendations. Choosing inefficient topologies allocates the wrong amount of resources, leads to unnecessary operation costs, or results in poor performance. Testing different topologies to find good solutions takes a lot of time and might delay productive operations. Therefore, this work introduces a software based deployment topology optimization approach for distributed enterprise applications. We use an enhanced performance model generator that extracts models from running applications. The extracted model is used to simulate performance metrics (e.g., resource utilization, response times, throughput) of an enterprise application. Subsequently, we introduce a deployment topology optimizer, which selects an optimized topology for a specified workload. The following two optimization goals are presented in this work: (i) minimum response time for an optimized user experience and (ii) maximize resource utilization for cost-effective topologies. To evaluate the approach we use the SPECjEnterpriseNEXT industry benchmark as distributed enterprise application. The evaluation demonstrates the accuracy of the simulation compared to the actual deployment and the pre-eminence of the selected topology compared to runner-up topologies.

Performance measurements and simulations produce large amounts of data in a short period of time. Release cycles are getting shorter due to the DevOps movement and heavily rely on live data from production or test environments. In addition, performance simulations increasingly become accurate and close to exact predictions. Results from these simulations are reliable and can be compared with live data to detect deviations from expected behavior. In this work, we present a comprehensive tool that can process and analyze measurement as well as simulation data quickly utilizing big data technologies. Live measurement data and simulation results can be analyzed for detecting performance problems, deviations from expected behavior or to simply compare a performance model with real world applications.

Determining the power demand of mobile applications (apps) is becoming a key area of interest for both end-users and developers due to the limited battery lifetime of mobile devices. Addressing this issue requires tools that measure the power demand of a mobile app. This power demand depends on the hardware components and the called operations of the mobile app. Therefore, this paper presents a prototype that records the power demand of hardware components and called operations of running mobile apps. This data can help developers in determining the cause of high power demand of their apps, which assists in power demand reduction. This paper covers an analysis of the overhead produced by the prototype and an evaluation that includes: firstly the evaluation of the accuracy of the total power demand measurements, secondly the evaluation for the allocation of the GPS power demand of the operations and thirdly the evaluation of the allocation of the CPU power demand of the operations.

Performance models assist capacity management and planning for large-scale enterprise applications by predicting their performance for different workloads and hardware environments. Manually creating these models often outweighs their benefits. Automatic performance model generators have been introduced to facilitate the model creation. These generators often use custom monitoring solutions to generate the required input data for the model creation. In contrast, standardized application performance management (APM) solutions are used in industry to control performance metrics for productive systems. This work presents the integration of industry standard APM solutions with a performance model generation framework. We apply the integration concepts using the APM solution Dynatrace and a performance model generation framework for Palladio Component Models (PCM).

DevOps is a trend towards a tighter integration between development (Dev) and operations (Ops) teams. The need for such an integration is driven by the requirement to continuously adapt enterprise applications (EAs) to changes in the business environment. As of today, DevOps concepts have been primarily introduced to ensure a constant flow of features and bug fixes into new releases from a functional perspective. In order to integrate a non-functional perspective into these DevOps concepts this report focuses on tools, activities, and processes to ensure one of the most important quality attributes of a software system, namely performance. Performance describes system properties concerning its timeliness and use of resources. Common metrics are response time, throughput, and resource utilization. Performance goals for EAs are typically defined by setting upper and/or lower bounds for these metrics and specific business transactions. In order to ensure that such performance goals can be met, several activities are required during development and operation of these systems as well as during the transition from Dev to Ops. Activities during development are typically summarized by the term Software Performance Engineering (SPE), whereas activities during operations are called Application Performance Management (APM). SPE and APM were historically tackled independently from each other, but the newly emerging DevOps concepts require and enable a tighter integration between both activity streams. This report presents existing solutions to support this integration as well as open research challenges in this area.

Resource demands are a core aspect of performance models. They describe how an operation utilizes a resource and therefore influence the systems performance metrics: response time, resource utilization and throughput. Such demands can be determined by two extraction classes: direct measurement or demand estimation. Selecting the best suited technique depends on available tools, acceptable measurement overhead and the level of granularity necessary for the performance model. This work compares two direct measurement techniques and an adaptive estimation technique based on multiple statistical approaches to evaluate strengths and weaknesses of each technique. We conduct a series of experiments using the SPECjEnterprise2010 industry benchmark and an automatic performance model generator for architecture-level performance models based on the Palladio Component Model. To compare the techniques we conduct two experiments with different levels of granularity on a standalone system, followed by one experiment using a distributed SPECjEnterprise2010 deployment combining both extraction classes for generating a full-stack performance model.

Distributed enterprise applications are typically implemented as system-ofsystems composed of components and linked via middleware. These systems often utilize corresponding resources far below available capacity. In order to increase resource utilizations the consolidation of components demands several tests on environments comparable to the production system. Performance models can be used to landscape such system architectures and to simulate changes in the component topology or resource environment without harming production systems. Therefore, this work aims at extracting performance models from distributed middleware platforms. Based on these models, an architecture optimizer is built to test different allocation topologies. Subsequently, the optimized model is simulated and the prediction accuracy of architecture changes is evaluated in this work. This allows architects to evaluate component changes and topology variations without a replica of the production system. 1 Motivation and Purpose Middleware based distributed system-of-systems architectures are state of the art in large scale enterprise applications [BVD14]. These systems are composed of components that can be moved and duplicated from one instance to another using a deployment management software [Woo09]. The placement of components is a complicated task that today is merely assisted by logical topology recommendations [Woo09, KKR11]. These recommendations can work as guidelines but cannot answer the questions on how to size the target environment for a specific deployment unit and how to optimize the topology to a certain optimization goal. Logical topologies utilize the hardware below their possible capacity as the average load of data centers today is under 25% [PVR12]. Virtualized server environments already reduce this overprovisioning thus, increasing the hardware utilization [SB10]. However, virtualized server environments limit the optimization opportunities to the granularity level of single virtual machines. Middleware systems rely on more fine-grained deployment units, allowing operation engineers and architects to utilize unused capacity more efficiently. Planning and testing such changes in productive environments comprises risks for the stability. In addition, productive alike test environments and productive systems have comparable prices. Furthermore, such environments are usually used to capacity by

This paper conducts a systematic literature review of papers published in the proceedings of the International Conference on Performance Engineering (ICPE) and its predecessors. It provides an overview of prevailing topics within the community over time. We look at research and contribution facets that have been used to address these topics. Trends are outlined in terms of evaluation methods to validate contributions. The results are complemented with a geographical and organizational dimension. The paper concludes with a look at the top ten contributing countries and organizations for this purpose.

Performance models can represent the performance relevant aspects of an enterprise application. Corresponding simulation engines use such models for simulating performance metrics (e.g., response times, resource utilization, throughput) and allow for performance evaluations without load testing the actual system. Creating such models manually often outweighs their benefits. Therefore, recent research created performance model generators, which can generate such models out of Application Performance Management software. However, a full-stack evaluation containing all relevant resources of an enterprise application (Central Processing Unit, memory, network and Hard Disk Drive) has not been conducted to the best of our knowledge. This work closes this gap using a pre-release version of the next generation industry benchmark SPECjEnterpriseNEXT of the Standard Performance Evaluation Corporation as example enterprise application, the Palladio Component Model as performance model and the performance model generator of the RETIT Capacity Manager. Furthermore, this work extends the generated model with a probabilistic garbage collection model to simulate memory allocation and releases more accurately.

Investigating the energy consumption of mobile applications (apps) is becoming a growing software engineering challenge due to the limited battery lifetime of mobile devices. Energy consumption is defined as the power demand integrated over time. Profiling the power demand of an app is a time consuming activity and the results are only valid for the target hardware used during the measurements. The energy consumption is influenced by the resource demands of an app, the hardware on which the app is running, and its workload. This work adapts resource profiles for enterprise applications to predict the energy consumption of mobile apps without the need to own a physical device. Resource profiles are models that represent all aspects influencing the energy consumption of an app. They can be used to predict the energy consumption for different hardware devices and evaluate the overall efficiency of an app. Moreover, the workload can be changed so that the impact of different usage patterns can be investigated. These capabilities lay the foundation for a platform-independent way of quantifying the energy consumption of mobile apps.

The rising energy demand in data centers and the limited battery lifetime of mobile devices introduces new challenges for the software engineering community. Addressing these challenges requires ways to measure and predict the energy consumption of software systems. Energy consumption is influenced by the resource demands of a software system, the hardware on which it is running, and its workload. Trade-off decisions between performance and energy can occur. To support these decisions, we propose an extension of the meta-model of the Palladio Component Model (PCM) that allows for energy consumption predictions. Energy consumption is defined as power demand integrated over time. The PCM meta-model is thus extended with a power consumption model element in order to predict the power demand of a software system over time. This paper covers two evaluations for this meta-model extension: one for a Java-based enterprise application (SPECjEnterprise2010) and another one for a mobile application (Runtastic). Predictions using an extended PCM meta-model for two SPECjEnterprise2010 deployments match energy consumption measurements with an error below 13