Power Capping of GPU Servers for Machine Learning Inference Optimization.

Power capping, which is an essential component of power oversubscription, has been widely used in data centers to host more servers than allowed by the capacity of their power infrastructures, in order to avoid expensive power upgrade and reduce capital expenses. Traditionally, power capping is performed mainly with CPU frequency and voltage scaling, which cannot be directly applied to the GPU servers that are commonly deployed in today's data centers, because GPUs can have much higher power consumption than CPUs. Recently proposed GPU power capping solutions are designed for a single GPU and so cannot be used on GPU servers that have a host CPU and multiple GPUs to process machine learning (ML) workloads. Hence, a joint power capping solution must be designed to coordinate the host CPU and all the GPUs in a server for optimizing ML inference performance. In this paper, we propose CapGPU, a power capping framework for today's GPU servers that run ML workloads on multiple GPUs and a host CPU in each server. In sharp contrast to existing solutions that try to control the server power consumption by throttling either the CPU or one GPU, in a separate manner, CapGPU features a multi-input multi-output (MIMO) power control methodology and a novel weight assignment algorithm that dynamically adjusts the weights assigned to each CPU/GPU based on their measured throughput. Consequently, the overall ML inference performance can be optimized. Our hardware testbed results demonstrate that CapGPU outperforms several state-of-the-art power capping solutions by having more precise power control, higher inference throughput, and better inference latency guarantees.