Power capping is an important technique for high-density servers to safely oversubscribe the power infrastructure in a data center. However, power capping is commonly accomplished by dynamically lowering the server processors’ frequency levels, which can result in degraded application performance. For servers that run important machine learning (ML) applications with Service-Level Objective (SLO) requirements, inference performance such as recognition accuracy must be optimized within a certain latency constraint, which demands high server performance. In order to achieve the best inference accuracy under the desired latency and server power constraints, this paper proposes OptimML, a multi-input-multi-output (MIMO) control framework that jointly controls both inference latency and server power consumption, by flexibly adjusting the machine learning model size (and so its required computing resources) when server frequency needs to be lowered for power capping. Our results on a hardware testbed with widely adopted ML framework (including PyTorch, TensorFlow, and MXNet) show that OptimML achieves higher inference accuracy compared with several well-designed baselines, while respecting both latency and power constraints. Furthermore, an adaptive control scheme with online model switching and estimation is designed to achieve analytic assurance of control accuracy and system stability, even in the face of significant workload/hardware variations.

Power capping, which is an essential component of power oversubscription, has been widely used in data centers to host more servers than allowed by the capacity of their power infrastructures, in order to avoid expensive power upgrade and reduce capital expenses. Traditionally, power capping is performed mainly with CPU frequency and voltage scaling, which cannot be directly applied to the GPU servers that are commonly deployed in today's data centers, because GPUs can have much higher power consumption than CPUs. Recently proposed GPU power capping solutions are designed for a single GPU and so cannot be used on GPU servers that have a host CPU and multiple GPUs to process machine learning (ML) workloads. Hence, a joint power capping solution must be designed to coordinate the host CPU and all the GPUs in a server for optimizing ML inference performance. In this paper, we propose CapGPU, a power capping framework for today's GPU servers that run ML workloads on multiple GPUs and a host CPU in each server. In sharp contrast to existing solutions that try to control the server power consumption by throttling either the CPU or one GPU, in a separate manner, CapGPU features a multi-input multi-output (MIMO) power control methodology and a novel weight assignment algorithm that dynamically adjusts the weights assigned to each CPU/GPU based on their measured throughput. Consequently, the overall ML inference performance can be optimized. Our hardware testbed results demonstrate that CapGPU outperforms several state-of-the-art power capping solutions by having more precise power control, higher inference throughput, and better inference latency guarantees.

GPUs have recently been adopted in many real-time embedded systems. However, existing GPU scheduling solutions are mostly open-loop and rely on the estimation of worst-case execution time (WCET). Although adaptive solutions, such as feedback control scheduling, have been previously proposed to handle this challenge for CPU-based real-time tasks, they cannot be directly applied to GPU, because GPUs have different and more complex architectures and so schedulable utilization bounds cannot apply to GPUs yet. In this article, we propose FC-GPU, the first Feedback Control GPU scheduling framework for real-time embedded systems. To model the GPU resource contention among tasks, we analytically derive a multi-input-multi-output (MIMO) system model that captures the impacts of task rate adaptation on the response times of different tasks. Building on this model, we design a MIMO controller that dynamically adjusts task rates based on measured response times. Our extensive hardware testbed results on an Nvidia RTX 3090 GPU and an AMD MI-100 GPU demonstrate that FC-GPU can provide better real-time performance even when the task execution times significantly increase at runtime.

The deployment of deep neural networks (DNNs) on edge devices is becoming increasingly common in latency-sensitive applications such as autonomous driving, real-time video analytics, and augmented reality. However, modern DNNs are rapidly growing in complexity, and edge GPUs often lack the computational resources available in cloud counterparts. This leads to increased inference latency and challenges in meeting strict Service Level Agreements (SLAs). In this paper, we introduce SEEB-GPU, a portable and efficient inference framework that addresses such challenges through a combination of application-level and system-level optimizations. At the system level, SEEB-GPU assigns dedicated Thread Processing Cluster (TPC) masks to individual inference tasks, preventing resource contention by spatially isolating GPU Streaming Multiprocessors (SMs). At the application level, it employs a deadline-aware heuristic that dynamically adjusts batch sizes and confidence thresholds, utilizing early-exit mechanisms in DNNs to significantly reduce inference latency while minimally reducing accuracy. Experimental results demonstrate that SEEB-GPU reduces inference latency by up to 15x, while maintaining competitive accuracy compared to existing edge inference methods. To our knowledge, SEEB-GPU is the first framework to jointly integrate batching, early-exits, and fine-grained GPU partitioning, thereby meeting SLAs and improving throughput on edge GPUs.

Today's data centers often need to run various machine learning (ML) applications with stringent SLO (Service-Level Objective) requirements, such as inference latency. To that end, data centers prefer to 1) over-provision the number of servers used for inference processing and 2) isolate them from other servers that run ML training, despite both use GPUs extensively, to minimize possible competition of computing resources. Those practices result in a low GPU utilization and thus a high capital expense. Hence, if training and inference jobs can be safely co-located on the same GPUs with explicit SLO guarantees, data centers could flexibly run fewer training jobs when an inference burst arrives and run more afterwards to increase GPU utilization, reducing their capital expenses. In this paper, we propose GPUColo, a two-tier co-location solution that provides explicit ML inference SLO guarantees for co-located GPUs. In the outer tier, we exploit GPU spatial sharing to dynamically adjust the percentage of active GPU threads allocated to spatially co-located inference and training processes, so that the inference latency can be guaranteed. Because spatial sharing can introduce considerable overheads and thus cannot be conducted at a fine time granularity, we design an inner tier that puts training jobs into periodic sleep, so that the inference jobs can quickly get more GPU resources for more prompt latency control. Our hardware testbed results show that GPUColo can precisely control the inference latency to the desired SLO, while maximizing the throughput of the training jobs co-located on the same GPUs. Our large-scale simulation with a 57-day real-world data center trace (6500 GPUs) also demonstrates that GPU Colo enables latency-guaranteed inference and training co-location. Consequently, it allows 74.9 % of GPUs to be saved for a much lower capital expense.

Tensor decomposition is a key operation in many data analytics and machine learning applications for extracting previously unknown patterns from sparse and/or multivariate data. The Canonical Polyadic Decomposition/Parallel Factors Analysis (CPD/PARAFAC) method, which is based on a combination of sparse and dense linear algebra operations, has received a lot of attention recently and is used in a variety of applications. The Alternate Least Squares Algorithm (ALS) is a numerical optimization algorithm that is used in the CPD/PARAFAC decomposition. It consists of several operations, including MTTKRP (Matrix times Tensor Khatri Rao Product), tall and wide matrix multiplication, and the computation of the pseudo-inverse of the resultant dense matrix. In recent days, GPUs are being used for high-performance computing applications instead of multi-core CPUs. A predominant challenge faced by the GPUs with respect to multicore CPUs is effective resource utilization for the CPD/PARAFAC-ALS algorithm with different levels of complexity and parallelism. To address this challenge we characterize the performance of CPD/PARAFAC-ALS on AMD MI100 GPU and propose OptiCPD as an optimal scheduling mechanism for the CPD/PARAFAC algorithm. Using OptiCPD we were able to achieve an average speedup of 7.5x for certain tensor benchmarks with respect to the existing baseline.

Data centers have been growing rapidly in recent years to meet the surging demand of cloud services. However, the expanding scale and powerful servers generate a great amount of heat, resulting in significant cooling costs. A trend in modern data centers is to raise the temperature and maintain all servers in a relatively hot environment. While this can save on cooling costs given benign workloads running in servers, the hot environment increases the risk of a cooling failure. In this article, we unveil a new vulnerability of existing data centers with aggressive cooling energy saving policies. Such a vulnerability might be exploited to launch thermal attacks that could severely worsen the thermal conditions in a data center. Specifically, we conduct thermal measurements and uncover effective thermal attack vectors at the server, rack, and data center levels. We also present damage assessments of thermal attacks. Our results demonstrate that thermal attacks can (1) largely increase the temperature of victim servers degrading their performance and reliability, (2) negatively impact on thermal conditions of neighboring servers causing local hotspots, (3) raise the cooling cost, and (4) even lead to cooling failures. Finally, we propose and evaluate effective server and data center level defenses to enhance thermal stabilities.

Unit type errors, where values with physical unit types (e.g., meters, hours) are used incorrectly in a computation, are common in today's unmanned aerial system (UAS) firmware. Recent studies show that unit type errors represent over 10% of bugs in UAS firmware. Moreover, the consequences of unit type errors are severe. Over 30% of unit type errors cause UAS crashes. This paper proposes SA4U: a practical system for detecting unit type errors in real-world UAS firmware. SA4U requires no modifications to firmware or developer annotations. It deduces the unit types of program variables by analyzing simulation traces and protocol definitions. SA4U uses the deduced unit types to identify when unit type errors occur. SA4U is effective: it identified 14 previously undetected bugs in two popular open-source firmware (ArduPilot & PX4.)

In a Vehicular Cyber Physical System (VCPS), ensuring the real-time delivery of safety messages is an important research problem for Vehicle to Vehicle (V2V) communication. Unfortunately, existing work relies only on one or two pre-selected control channels for safety message communication, which can result in poor packet delivery and potential accident when the vehicle density is high. If all the available channels can be dynamically utilized when the control channel is having severe contention, then safety messages can have a much better chance to meet their real-time deadlines. In this article, we propose MC-Safe, a multi-channel V2V communication framework that monitors all the available channels and dynamically selects the best one for safety message transmission. During normal driving, MC-Safe monitors periodic beacons sent by other vehicles and estimates the communication delay on all the channels. Upon the detection of a potential accident, MC-Safe leverages a novel channel negotiation scheme that allows all the involved vehicles to work collaboratively, in a distributed manner, for identifying a communication channel that meets the delay requirement. MC-safe also features a novel coordinator selection algorithm that minimizes the delay of channel negotiation. Once a channel is selected, all the involved vehicles switch to the same selected channel for real-time communication with the least amount of interference. Our evaluation results both in simulation and on a hardware testbed with scaled cars show that MC-Safe outperforms existing single-channel solutions and other well-designed multi-channel baselines by having a 23.4% lower packet delay on average compared with other well-designed channel selection baselines.

The rapid growth of autonomous driving in recent years has posed some new research challenges to the traditional vehicle control system. For example, in order to flexibly change the yawing rate and moving speed of a vehicle based on the detected road conditions, autonomous driving control often needs to dynamically tune its control parameters for better trajectory tracking and vehicle stability. Consequently, the execution time of driving control can increase significantly, resulting in missing the end-to-end (E2E) deadline from detection to computation and actuation, and thus possible accidents. In this paper, we propose AutoE2E, a two-tier real-time middleware system that helps the automotive OS meet the E2E deadlines of all the tasks despite execution time variations, while achieving the maximum possible computation precision (and thus minimum tracking errors) for driving control. The inner loop of AutoE2E dynamically controls the CPU utilizations of all the on-board processors to stay below their respective schedulable utilization bounds, by adjusting the invocation rates of the vehicle tasks running on those processors. The outer loop is designed to adapt the computation time and precision of driving control, when the inner loop loses its control capability due to rate saturation caused by vehicle speed changes. Our evaluations, both on a hardware testbed with scaled cars and in larger-scale simulation, show that AutoE2E can effectively reduce the deadline miss ratio by 35.4% on average, compared to well-designed baselines, while having smaller precision loss and tracking errors.

Recent studies have proposed to use the Channel State Information (CSI) of WiFi wireless channel for human gesture recognition. As an important application, CSI-based driver activity recognition in passenger vehicles has received increasing research attention. However, a serious limitation of almost all the existing WiFi-based recognition solutions is that they can only recognize the activity of a single person at a time, because the activities of other people (if performed at the same time) can interfere with the WiFi signals. In a sharp contrast, there can often be one or more passengers in any vehicles. In this paper, we propose CARIN, CSI-based driver Activity Recognition under the INterference of passengers. CARIN features a combination-based solution that profiles all the possible activity combinations of driver and (one or more) passengers in offline training and then performs recognition online. To attack possible combination explosion, we first leverage in-car pressure sensors to significantly reduce combinations, because there are only limited seating options in a passenger vehicle. We then formulate a distance minimization problem for fast runtime recognition. In addition, a period analysis methodology is designed based on the kNN classifier to recognize activities that have a sequence of body movements, like continuous head nodding due to driver fatigue. Our results in a real car with 3,000 real-world traces show that CARIN can achieve an overall F1 score of 90.9%, and outperforms the three state-of-the-art solutions by 32.2%.

Abnormal battery drain (ABD) can negatively impact the user experience of smartphone apps, by consuming an unnecessarily high amount of energy and causing short battery lifetime. Unfortunately, user reports on ABD are usually too vague for app developers to precisely know how and when the ABD manifests. Therefore, it is important to have a diagnostic tool that can help app developers identify the ABD manifestation point for root cause analysis.In this paper, we propose EnergyDx, an automated diagnosis framework that assists developers in pinpointing the functions that either directly lead to or commonly coincide with the manifestation of ABD. EnergyDx features a novel 5-step analysis algorithm to distinguish the real ABD manifestation point from the power transition points caused by normal phone usage. We have prototyped EnergyDx in Android and evaluated it with 40 different real-world apps for diagnosing ABD cases caused by various types of issues. Our results show that EnergyDx reduces, on average, 93% of the amount of code that the developers would need to search for the root causes of ABD.

The massive energy consumption of data centers worldwide has resulted in a large carbon footprint, raising serious concerns to sustainable IT initiatives and attracting a great amount of research attention. Nonetheless, the current efforts to date, despite encouraging, have been primarily centered around owner-operated data centers (e.g., Google data center), leaving out another major segment of data center industry-colocation data centers-much less explored. As a major hindrance to carbon efficiency desired by the operator, colocation suffers from "split incentive": tenants may not be willing to manage their servers for carbon efficiency. In this paper, we aim at minimizing the carbon footprint of geo-distributed colocation data centers, while ensuring that the operator's cost meets a long-term budget constraint. We overcome the "split incentive" hurdle by devising a novel online carbon-aware incentive mechanism, called GreenColo, in which tenants voluntarily bid for energy reduction at self-determined prices and will receive financial rewards if their bids are accepted at runtime. Using trace based simulation we show that GreenColo results in a carbon footprint fairly close (23 versus 18 percent) to the optimal offline solution with future information, while being able to satisfy the colocation operator's long-term budget constraint. We demonstrate the effectiveness of GreenColo in practical scenarios via both simulation studies and scaled-down prototype experiments. Our results show that GreenColo can reduce the carbon footprint by up to 24 percent without incurring any additional cost for the colocation operator (compared to the no-incentive baseline case), while tenants receive financial rewards for "free" without violating service level agreement.

Computational sprinting is an effective mechanism to temporarily boost the performance of data center servers. However, given the great effect on performance improvement, how to make the sprinting process controllable and how to maximize the sprinting efficiency have not been well discussed yet. Those can be significant problems for a data center when computational sprinting is needed for more than a few minutes, since it requires the support of energy storage, whose capacity is limited. The control and efficiency of sprinting not only involve how fast to run servers and how to allocate resources to co-running workloads, but also the impact on power overload, and how to handle the overload with circuit breakers and energy storage to ensure power safety. Different workloads can impact sprinting in different ways, and hence efficient sprinting requires workload-specific strategies. In this paper, we propose SprintCon to realize controllable and efficient computational sprinting for data center servers. SprintCon mainly consists of a power load allocator and two different power controllers. The allocator analyzes how to divide the power load to different power sources. The server power controller adapts the CPU cores that process batch workloads, to improve the efficiency in terms of computing, energy and cost. The UPS power controller dynamically adjusts the discharge rate of UPS energy storage to satisfy the time-varying power demand of interactive workloads, and ensure power safety. The experiment results show that compared to state-of-the-art solutions, SprintCon can achieve 6-56% better computing performance and up to 87% less demand of energy storage.

A major concern for today’s smartphones is their much faster battery drain than traditional feature phones, despite their greater battery capacities. The difference is mainly contributed by those more powerful but also much more power-consuming smartphone components, such as the multi-core application processor and the high-definition (HD) display. While the application processor must be active when any smart apps are being used, it is also unnecessarily waken up, even during idle periods, to perform operations related to basic phone functions (i.e., incoming calls and text messages). In addition, the power-hungry HD display is also used unnecessarily for such basic functions. In this article, we investigate how to increase the battery life of smartphones by minimizing the use of application processor and HD display for operations related to basic functions. We find that the application processor is often waken up by a process running on it, called the Radio Interface Layer Daemon (RILD), which interfaces the user and apps to the GSM/LTE cellular network. In particular, we demonstrate that a great amount of energy could be saved if RILD is stopped, such that the application processor can sleep more often. Based on this key finding, we design a Smart On Demand (SOD) configuration that reduces the smartphone energy consumption by running RILD operations on a secondary low-power microcontroller and by using a secondary low-power display to interface the user with basic functions. As a result, basic phone functions can be handled at much lower energy costs and the power-consuming components, i.e., application processor and HD display, are waken up only when one needs to use any smart apps, in an on-demand manner. We have built a hardware prototype of SOD and evaluated it with real user traces. Our results show that SOD can increase its battery life by up to 2.5 more days.

Two critical quality factors for mobile devices (e.g., smartphones, tablets) are battery life and apps' userperceived performance, e.g., responsiveness of user actions and frame rate of video playback. Sadly, state-of-the-art solutions have at least one of the following two limitations: 1) they cannot efficiently handle concurrent foreground apps and so may lead to performance imbalance and high energy consumption, 2) they try to regulate app performance periodically and thus may not efficiently handle the aperiodicity of user actions, which can result in poor responsiveness or high overheads. In this paper, we present SURF, Supervisory control of User-perceived peRFormance, which is designed to overcome the two limitations. First, it dynamically allocates resources to concurrent apps for balanced performance. Second, SURF uses supervisory control theory to handle the aperiodicity of user actions. SURF features a two-level architecture design that performs the two tasks at different time scales, according to their different overheads and timing requirements. We test SURF on several mobile device models with real-world opensource apps and show that it can reduce the CPU energy consumption by 30-90% compared to state-of-the-art solutions while causing no perceivable performance degradation.

A critical quality factor for smartphone apps is responsiveness, which indicates how fast an app reacts to user actions. A soft hang occurs when the app's response time of handling a certain user action is longer than a user-perceivable delay. Soft hangs can be caused by normal User Interface (UI) rendering or some blocking operations that should not be conducted on the app's main thread (i.e., soft hang bugs ). Existing solutions on soft hang bug detection focus mainly on offline app code examination to find previously known blocking operations and then move them off the main thread. Unfortunately, such offline solutions can fail to identify blocking operations that are previously unknown or hidden in libraries. In this paper, we present Hang Doctor, a runtime methodology that supplements the existing offline algorithms by detecting and diagnosing soft hangs caused by previously unknown blocking operations. Hang Doctor features a two-phase algorithm that first checks response time and performance event counters for detecting possible soft hang bugs with small overheads, and then performs stack trace analysis when diagnosis is necessary. A novel soft hang filter based on correlation analysis is designed to minimize false positives and negatives for high detection performance and low overhead. We have implemented a prototype of Hang Doctor and tested it with the latest releases of 114 real-world apps. Hang Doctor has identified 34 new soft hang bugs that are previously unknown to their developers, among which 62%, so far, have been confirmed by the developers, and 68% are missed by offline algorithms.

Due to the increasing popularity of smartphones, many people are now equipped with both a smartphone and at least one desktop (or laptop) computer. Although the two computing devices are used for similar purposes (e.g., email, web browsing), they are often both kept on for the user's convenience, despite only one device is actively used at a time. Therefore, if one of the two computing devices can be put into an energy-saving mode when the other one is in use, a significant amount of energy can be saved for both the phone and the desktop.In this paper, we propose CoSmart, a light-weight solution that coordinates the smartphone with the desktop computer for joint energy savings. CoSmart dynamically degrades the smartphone to a feature phone with only basic GSM functions when the user is detected to be with the desktop, in order to save both computation and idle energy. The desktop is then put into sleep for energy savings when the user leaves it, while the phone can be turned back to a smartphone, such that the user can continue the operation with seamless task migration. There are several research challenges in the design of CoSmart, which include 1) predicting whether the user would stay long enough with the desktop to offset the migration overheads, and 2) determining the best time point for task migration that can result in the most energy savings. To this end, we propose a novel algorithm for dynamic idle time length prediction, and model joint energy savings as an optimization problem for the most energy savings. A prototype of CoSmart is implemented in Android and evaluated using different real user traces and popular apps. Results show CoSmart can achieve, on average, 61.3% energy savings for the smartphone and 46.7% energy savings for the desktop, which outperforms other baselines by as much as 17.2% to 19.0%.

Smartphone users spend more than 80% of their phone time accessing web information, which could cause undesirably large energy drain. To provide web information, a web activity may invoke asynchronous execution in different hardware devices. Thus, traditional energy estimation methods based on system statistics are usually insufficient to capture the secluded energy cost. In this paper, we propose REEWA, a runtime energy estimation framework for web activities on smartphones. In sharp contrast to the traditional modeling methods, REEWA features a design to provide highly accurate and low-overhead energy estimation based on hardware performance counters that can accurately record hardware-level events. Specifically, REEWA features (1) a set of energy models for smartphone hardware components involved in web activities, which are built based on their respective performance counters; (2) a correlation study on the counter selection process that provides the best tradeoff between the estimation accuracy and overhead; (3) a performance counter management mechanism for activity deployment. We prototyped and evaluated REEWA in two real android smart-phones. The results show that, compared to traditional estimation methods, REEWA achieves an average 33% higher estimation accuracy with a negligible overhead (less than 1%, worst-case). We applied REEWA to support heterogeneous core scheduling for web activities, which can help reduce 40% energy consumption.

A major concern for today’s smartphones is their much faster battery drain than traditional feature phones, despite their greater battery capacities. The difference is mainly contributed by those more powerful but also much more powerconsuming smartphone components, such as the multi-core application processor. While the application processor must be active when any smart apps are being used, it is also unnecessarily waken up, even during idle periods, to perform operations related to basic phone functions (i.e., incoming calls and text messages).In this paper, we investigate how to increase the battery life of smartphones by minimizing the use of the application processor during idle periods.We find that the application processor is often waken up by a process running on it, called the Radio Interface Layer Daemon (RILD), which interfaces the user and apps to the GSM/LTE cellular network. In particular, we demonstrate that a great amount of energy could be saved if RILD is stopped, such that the application processor can sleep more often. Based on this key finding, we design a Smart On Demand (SOD) configuration that reduces smartphone idle energy consumption by running RILD operations on a secondary low-power microcontroller. As a result, RILD operations can be handled at much lower energy costs and the application processor is waken up only when one needs to use any smart apps, in an on-demand manner. We have built a hardware prototype of SOD and evaluated it with real user traces. Our results show that SOD can increase its battery life by up to 2.5 more days.