Diffusion models have become essential for high-fidelity 3D MRI synthesis, yet their deployment remains constrained by substantial GPU resource demands arising from hundreds of U-Net evaluations per sample and a highly heterogeneous kernel behavior. This paper performs a comprehensive performance analysis of the state-of-the-art medical diffusion model, Med-DDPM, across three generations of NVIDIA architectures to study kernel-level runtime breakdowns, instruction-mix characteristics, memory system utilization, warp-level activities, and profiler priority-score estimates. We show that training is overwhelmingly dominated by cuDNN convolution and implicit-GEMM kernels, with inefficiencies arising from memory-access patterns, tensor-layout conversions, and limited Tensor Core utilization. Guided by these insights, we evaluate two architecture-aware optimizations TF32 Tensor Core activation and a 3D channels-last layout and demonstrate that they reduce SM cycles by up to 100x, cut dynamic instructions by 100x, raise Tensor Core utilization from 1.45 to 9.98x, and increase IPC by 7
Large Language Models (LLMs) inference is central to modern AI applications, dominating worldwide datacenter workloads, making it critical to predict its energy footprint. Existing approaches estimate energy consumption as a simple linear function of input and output sequence. However, by analyzing the autoregressive structure of Transformers, which implies a fundamentally non-linear relationship between input and output sequence lengths and energy consumption, we demonstrate the existence of a generation energy minima. Peak efficiency occurs with short-to-moderate inputs and medium-length outputs, while efficiency drops sharply for long inputs or very short outputs. Consequently, we propose SweetSpot, an analytical model derived from the computational and memory-access complexity of the Transformer architecture, which accurately characterizes the efficiency curve as a function of input and output lengths. To assess accuracy, we measure energy consumption using TensorRT-LLM on NVIDIA H100 GPUs across a diverse set of LLMs ranging from 1B to 9B parameters, including OPT, LLaMA, Gemma, Falcon, Qwen2, and Granite. We test input and output lengths from 64 to 4096 tokens and achieve a mean MAPE of 1.79
Autoscaling is a critical component of modern cloud computing environments, improving flexibility, efficiency, and cost-effectiveness. Current approaches use centralized autoscalers that make decisions based on averaged monitoring data of managed service instances in fixed intervals. In this scheme, autoscalers are single points of failure, tightly coupled to monitoring systems, and limited in reaction times, making non-optimal scaling decisions costly. This paper presents an approach for continuous decentralized autoscaling, where decisions are made on a service instance level. By distributing scaling decisions of instances over time, autoscaling evolves into a quasi-continuous process, enabling great adaptability to different workload patterns. We analyze our approach on different abstraction levels, including a model-based, simulation-based, and real-world evaluation. Proof-of-concept experiments show that our approach is able to scale different applications deployed in containers and virtual machines in realistic environments, yielding better scaling performance compared to established baseline autoscalers, especially in scenarios with highly dynamic workloads.
The rapid growth of edge devices has revolutionized industrial AI applications, including robotics, autonomous systems, and IoT, where real-time processing is essential. These systems face the challenge of managing concurrent, high-volume workloads across resource-constrained edge devices and cloud infrastructure. A major hurdle is optimizing deep learning model deployment across edge-cloud environments in dynamic conditions, particularly where input quality and noise fluctuate under concurrent demands. This paper introduces a novel optimization framework, that addressed these challenges and dynamically selects the most suitable models from a diverse model zoo and determines optimal deployment locations (edge or cloud). The proposed framework leverages content-aware approach to minimize both communication and computation latency while considering hardware limitations and environmental factors. Using a binary linear programming (BILP) approach, our method efficiently balances model distribution of an AI pipeline, maximizing end-to-end performance. We validate this framework on a robotic AI pipeline in real-world, noise-variant environments, comparing content-aware and content-agnostic deployment strategies. Our results demonstrate significant optimization in deployment latency and system performance under high-concurrency conditions, using both content-agnostic and content-aware approaches, highlighting the framework's robustness and scalability. Additionally, we showed the effectiveness of the content-aware approach over the content-agnostic method in optimizing deployment choices and reducing latency, while maintaining the desired qualitative outcomes of the AI pipeline with different communication set up. This makes the content-aware strategy more suitable for complex, real-world environments where input quality and noise vary significantly. Overall, The proposed method presents a compelling solution for optimizing AI pipelines in edge-cloud ecosystems, offering potential for broader applications domains.
With the ever increasing importance of cloud computing and a strong focus on reliable data centers, a high amount of research has been done on failure prediction for hard disk drives. The collection of monitoring data, such as SMART statistics (Self-Monitoring, Analysis, and Reporting Technology) from operational HDDs, enables operators to obtain predictions about the expected remaining useful life. Numerous methods for HDD failure prediction have been published in recent years, and their evaluation has shown decent results. However, a naive splitting into training and test sets can lead to data leakage and, thus, over-optimistic results that cannot be achieved on the data of scientific interest. In this paper, we propose a novel data leakage measure for quantifying the amount of data leakage in training and test datasets. Further, we define four splitting techniques and evaluate our measure in terms of the performance optimism of classification models with respect to these different splitting strategies. Our results consistently show that splitting techniques prone to data leakage induce an overestimation of predictive performance. Overall, we were able to show the usefulness of the defined data leakage measure, as well as its connection with different splitting techniques and the performance optimism of prediction models.
Microservice applications are the building blocks of modern cloud applications. As such, their performance aspects have been receiving increasing attention in the software engineering community. However, many microservice performance studies use only a small set of popular microservice test applications for experiments, questioning the applicability of their approaches in practice. Researchers currently lack the opportunity to collect large and diverse datasets containing performance metrics of microservices. This is because popular test applications only represent specific technology stacks and often come with custom benchmark tooling (e.g., load generation and monitoring). In this paper, we present Creo, a framework for generating microservice applications that (1) are fully executable, (2) have configurable properties and resource usage profiles, and (3) have built-in support for standardized monitoring, load generation, and deployment. Our approach enables researchers to run experiments with diverse microservice applications with minimal effort. We demonstrate the value of our approach in the context of two use cases. First, we show that using generated applications when training machine learning models for predicting performance degradation can improve the prediction accuracy. Second, we evaluate a recent approach for performance anomaly classification on a set of generated applications highlighting strengths and weaknesses not discussed in the original work.
Cloud providers introduce features and optimizations to improve efficiency and reliability, such as Spot VMs, Harvest VMs, oversubscription, and auto-scaling. To use these effectively, it's important to understand workload characteristics. However, workload characterization can be complex and difficult to scale manually due to multiple signals involved. In this study, we conduct the first large-scale empirical study of first-party workloads at Microsoft to understand their characteristics. Through this empirical study, we aim to answer the following questions: (1) What are the critical workload characteristics that impact efficiency and reliability on cloud platforms? (2) How do these characteristics vary across different workloads? (3) How can cloud platforms leverage these insights to efficiently characterize all workloads at scale? This study provides a deeper understanding of workload characteristics and their impact on cloud performance, which can aid in optimizing cloud services and identifies potential areas for future research.
This keynote examines the transformative impact of Foundation Models (FMs), particularly Large Language Models (LLMs), on software development, emphasizing the critical role of Software Performance Engineering (SPE) in ensuring FM-powered software (FMware) achieves essential performance goals such as throughput and latency. With the LLM market projected to reach $36.1 billion by 2030 [3], addressing SPE challenges has become increasingly urgent. Drawing extensively from comprehensive literature surveys, industry-academia interactions, customer feedback, and practical experience detailed in [6]. This keynote identifies four critical SPE challenges throughout the FMware lifecycle, discusses current state-of-practice solutions, proposes future research directions, and introduces a vision for an innovative SLA-aware runtime system designed to enhance the performance and efficiency of FMware.
The resource-constrained nature of edge devices poses unique challenges in meeting strict performance requirements. However, performance benchmarks for deployed models are often run manually and infrequently, and other phases of the development workflow, such as the conversion from high-level languages to C/C++ code, might not be evaluated for performance. While these traditional approaches to performance evaluation give important insights for improvements in the final product, the integration of performance testing throughout the product development process enables early detection and mitigation of performance issues. In this work, we propose an automated workflow that streamlines the performance evaluation and optimization of deployed deep learning models on edge devices.
5G networks offer significant advancements over its predecessor, 4G Long-Term Evolution (LTE). Low latency network access, a key requirement enabling near real-time responsiveness as required by applications such as autonomous driving, factory automation and virtual reality, is one of 5G's key features. In this paper, we present the results of a long-term measurement campaign of the uplink end-to-end (e2e) latency experienced by a 5G-capable device using a commercial sub-6Ghz 5G non-standalone (NSA) network. Our results show an average uplink e2e latency of 12ms, with a 95th percentile of 21ms. This compares favorably with an average uplink e2e latency of 35ms and a 95th percentile of 53ms using 4G LTE to reach the same destination. We also characterize and define, through real-world network parameters in the uplink data transmission process, an unexpected latency pattern that impacts the performance of latency-sensitive applications, even in 5G standalone (SA) networks, such as edge computing or ultra-reliable low-latency communication (URLLC) a new class of applications targeted in 5G networks.
Microservices and containers have transformed the way applications are developed, tested, deployed, scaled, and managed. Several container orchestration platforms, like Kubernetes, have emerged, streamlining container management at scale and providing enterprise-grade support for application modernization. Driven by application, compliance, and end-user requirements, companies opt to deploy multiple Kubernetes clusters across public and private clouds. However, deploying applications in multi-cluster environments presents distinct challenges, especially managing the communication between the microservices spread across clusters. Traditionally, custom configurations, like VPNs or firewall rules, were required to connect such complex setups of clusters spanning the public cloud and on-premise infrastructure. This industry paper presents a comprehensive analysis of network performance characteristics for three popular open-source multi-cluster networking solutions (namely, Skupper, Submariner, and Istio), addressing the challenges of microservices connectivity across clusters. We evaluate key factors such as latency, throughput, and resource utilization using established tools and benchmarks, offering valuable insights for organizations aiming to optimize the network performance of their multi-cluster deployments. Our experiments revealed that each solution involves unique trade-offs in performance and resource efficiency: Submariner offers low latency and consistency, Istio excels in throughput with moderate resource consumption, and Skupper stands out for its ease of configuration while maintaining balanced performance.
Fine-tuning pre-trained models is the preferred method for adapting large language models (LLMs) for specific downstream tasks since it is significantly more efficient in terms of computational costs and energy than training the models from scratch. However, with LLMs experiencing exponential growth, fine-tuning the models becomes more challenging and expensive as they demand more computational resources. Many approaches are proposed to fine-tune state-of-the-art models efficiently, reducing the infrastructure needed, and thus, making them accessible to the public. In this paper, we investigate a technique called Low-Rank Adaptation (LoRA), one approach to efficiently fine-tuning LLMs by leveraging low intrinsic dimensions possessed by the models during fine-tuning. Specifically, we explore different data formats that can be used during LoRA fine-tuning and compare them regarding workload performance and model accuracy. The experiment compared LoRA and its quantized counterpart (QLoRA) with regular methods to fine-tune state-of-the-art LLMs. The analysis includes estimating memory usage, measuring resource utilization, and evaluating the model quality after fine-tuning. Three state-of-the-art Graphics Processing Units (GPUs) are used for experiments, including NVIDIA H100, NVIDIA A100, and NVIDIA L40. We also use the newest AMD MI300X GPU as a preliminary exploration. The experiment shows that although LoRA with a 16-bit floating-point format can significantly reduce the computational resource demand, it still requires data-center-class GPUs with ample memory to fine-tune LLMs with 70 billion parameters. Using QLoRA with 4-bit floating-point format significantly lowers the memory requirements by as much as 75% compared to LoRA, allowing a single GPU with 48 GB and 80 GB of memory to fine-tune 70 billion parameter models. In addition, QLoRA delivers model quality that is on par with or exceeds the quality of the model obtained from conventional fine-tuning.
System designs for bounded communication latencies often employ a rather basic concept at their core: First-In First-Out (FIFO) queueing. Network Calculus (NC) can compute delay bounds for the end-to-end communication of data flows crossing potentially large feedforward networks of such First-In First-Out (FIFO) systems. Analysis complexity stems from the need to keep track of the interactions between flows when they compete for resources, i.e., multiplex in shared queues. Network Calculus (NC) has an elegant solution to this: an open, so-called First-In First-Out (FIFO) parameter is introduced every time a (worst-case) First-In First-Out (FIFO) interaction occurs in the analysis. At the end of the analysis stands a (min,plus)-algebraic term with interdependent First-In First-Out (FIFO) parameters. We aim at finding an optimal setting for all open parameters. When employing standard optimization techniques, we cannot work with a parameterized (min,plus)-algebraic term directly. Thus, we show how to derive a minimum size (plus,times)-algebraic term that we can use with Non-Linear Program (NLP) solvers efficiently. Additionally, we show how to differentiate this term to open our approach to gradient-based Non-Linear Program (NLP) algorithms. In numerical evaluations, we show that our approach outperforms the complexity/accuracy tradeoff of existing heuristics to set the First-In First-Out (FIFO) parameters. With a slight increase of analysis runtime, we reduce the gap to the optimal setting by a factor of 4.4, to 0.15% on average.
The processing of large amounts of data in central high performance data analytics (HPDA) systems is playing an increasingly important role in science and business. However, many HPDA systems exhibit a low utilization of their available resources during normal operation. An important reason for this underutilization is that too many resources are reserved for individual jobs. This is often a consequence of the common practice of reserving a uniform amount of resources such as CPU or memory for the entire execution time of a job. Given that many data intensive (DI) jobs consist of different phases with different resource demands, resources are normally reserved according to the demand of the most resource-intensive phase. This results in more resources being reserved over a long period of time than are actually needed. Flexible resource allocation techniques require detailed information about the resource demands of individual jobs to be applied effectively. In this work, we present PARAGRAPH, an approach to create phase models and resulting phase-aware resource demand profiles for individual job types from training datasets of resource consumption time series. PARAGRAPH considers the individual jobs as black boxes and fully relies on recorded system-level metrics. To do this, we first extract the different phases from the resource consumption time series using a BinSeg-based algorithm. We then apply the C-DBSCAN clustering algorithm to assign labels to the individual segments. Based on this information, a phase model and a resource demand profile can be extracted. These phase-aware resource demand profiles can then be used for scheduling decisions. We evaluate PARAGRAPH in an experimental scenario that allows flexible resource reservation on a HPDA platform. Here, we show that a given set of job instances can be executed up to 29% faster for a given resource limit due to better resource utilization.
As the dependence on software systems running on cloud data centers grows on a daily basis, there is an increasingly stronger motivation to reduce their energy consumption. A necessary but not trivial step in this direction is understanding how energy is consumed in virtualized, multi-tenant environments such as the one provisioned in the cloud. Prior work focuses on isolated, non-virtualized systems and is difficult to transfer to this context. A number of industry-led approaches have appeared in the meantime in terms of tools and technological stacks building on the concept of observability as the means to achieve this goal. This paper discusses our approach in adopting one such stack and consequently assessing it for fitness to purpose through an experimental procedure. To this effect, we deploy a cloud-native application on a private cloud infrastructure instrumented for measuring energy consumption through a combination of hardware and software means. We combine the information from these instrumentation points into a mapping model to deal with the different virtualization layers and compare the model against the values reported by the observability stack. Furthermore, we use our model to attribute energy consumption across the virtualization layers and understand how energy is consumed at each one.
The end of Moore's Law and Dennard scaling has driven the proliferation of heterogeneous systems with accelerators, including CPUs, GPUs, and FPGAs, each with distinct architectures, compilers, and programming environments. GPUs excel at massively parallel processing for tasks like deep learning training and graphics rendering, while FPGAs offer hardware-level flexibility and energy efficiency for low-latency, high-throughput applications. In contrast, CPUs, while general-purpose, often fall short in high-parallelism or power-constrained applications. This architectural diversity makes it challenging to compare these accelerators effectively, leading to uncertainty in selecting optimal hardware and software tools for specific applications. To address this challenge, we introduce HeteroBench, a versatile benchmark suite for heterogeneous systems. HeteroBench allows users to evaluate multi-compute kernel applications across various accelerators, including CPUs, GPUs (fromNVIDIA, AMD, Intel), and FPGAs (AMD), supporting programming environments of Python, Numba-accelerated Python, serial C++, OpenMP (both CPUs and GPUs), OpenACC and CUDA for GPUs, and Vitis HLS for FPGAs. This setup enables users to assign kernels to suitable hardware platforms, ensuring comprehensive device comparisons. What makes HeteroBench unique is its vendor-agnostic, cross-platform approach, spanning diverse domains such as image processing, machine learning, numerical computation, and physical simulation, ensuring deeper insights for HPC optimization. Extensive testing across multiple systems provides practical reference points for HPC practitioners, simplifying hardware selection and performance tuning for both developers and end-users alike. This suite may assist to make more informed decision on AI/ML deployment and HPC development, making it an invaluable resource for advancing academic research and industrial applications.
Power consumption has become a limiting factor in all areas of computing. Hence, making the most of the available power budget is paramount. To use the available budget most efficiently, techniques like dynamic voltage and frequency scaling and idle states can be used. This work analyzes the instructions UMWAIT, TPAUSE, and MWAITX on three different systems. We analyze their instruction latencies, power consumptions, and dependencies on core frequencies. To do so, we introduce benchmarks to gather performance and power parameters, which can be used for future software optimizations. Key findings include: The expected sleep duration passed to UMWAIT and TPAUSE can influence the depth of the user idle state. The actual sleep duration of TPAUSE increases stepwise with an increasing expected sleep duration. Requesting a deeper idle state leads to an additional sleep duration, which increases with a lower core frequency. The core frequency influences the instruction latency of TPAUSE, where a low frequency can lead to an irregular performance pattern. The latency of TPAUSE, UMWAIT, and MWAITX is most often higher than requested on the evaluated systems. Core power consumption can be reduced by similar to 20 % to similar to 70 % compared to the usage of PAUSE. The latency for waking a core in user idle reflects the underlying hardware architecture with tens (desktop architecture with shallow idle states) to hundreds (server architecture with deep idle states) of nanoseconds at nominal frequencies.
Stream Aggregates are crucial in digital infrastructures for transforming continuous data streams into actionable insights. However, state-of-the-art Stream Processing Engines lack mechanisms to effectively balance performance with memory consumption - a capability that is especially crucial in environments with fluctuating computational resources and data-intensive workloads. This paper tackles this gap by introducing a novel on-demand adaptive memory compression scheme for stream Aggregates. Our approach uses Reinforcement Learning (RL) to dynamically adapt how a stream Aggregate compresses its state, balancing performance and memory utilization under a given processing latency threshold. We develop a model that incorporates the application- and data-specific nuances of stream Aggregates and create a framework to train RL Agents to adjust memory compression levels in real-time. Additionally, we shed light on a trade-off between the timeliness of an RL Agent training and its resulting behavior, defining several policies to account for this trade-off. Through extensive evaluation, we show that the proposed RL Agent supports well on-demand memory compression. We also study the effects of our policies - providing guidance on their role in RL applied to stream Aggregates - and show our framework supports lean execution of such RL jobs.
Control groups (cgroups) are a crucial isolation mechanism in containerized environments, but they don't fully prevent performance interference (noisy neighbors). This paper presents a novel, workload-agnostic approach for detecting noisy neighbors within CPU-isolated cgroups. Using in-kernel profiling with Extended Berkeley Packet Filter (eBPF), we instrument the Linux process scheduler to capture scheduling latencies and preemption frequencies. We introduce a detection method based on these metrics to identify noisy neighbors online without requiring workload profiles or offline analysis. Evaluations across various workload scenarios demonstrate the effectiveness of our approach in accurately identifying performance degradation caused by noisy neighbors.
