Enabling Client-Autonomous Training Optimizations for Efficient Federated Learning

Federated Learning (FL) enables collaborate model training without privacy violation, where clients periodically report their updates to the server in communication rounds. Due to heterogeneous resource and limited bandwidth, FL processes often suffer from low efficiency. Existing works in that regard are oblivious to the intra-round execution status on clients, failing to tackle runtime stragglers or hide the communication overheads for some early-converged layers. In this paper, we propose FedCA, a novel mechanism that allows clients to autonomously exploit intra-round training status for higher efficiency while preserving accuracy performance. We first devise a metric to help quantify the statistical contribution of different iterations in a round, which can be efficiently profiled at runtime with the periodical sampling strategy. With the instantaneous system and statistical status, to improve computation efficiency, clients under FedCA can adaptively determine the intra-round workloads based on a utility function depicting the marginal computation benefit. Besides, to mitigate the communication bottleneck, for some parameters attaining fast local convergence, clients under FedCA can eagerly transmit their updates to the FL server prior to round completion. We also extend FedCA to FedCA+, integrating speculative sparsification to futher reduce the cumulative communication amount. We implemented FedCA and FedCA+ atop PyTorch, and large-scale experiments show that they can improve the FL efficiency by up to 45.3%.