When Foresight Pruning Meets Zeroth-Order Optimization: Efficient Federated Learning for Low-Memory Devices
arxiv(2024)
摘要
Although Federated Learning (FL) enables collaborative learning in Artificial
Intelligence of Things (AIoT) design, it fails to work on low-memory AIoT
devices due to its heavy memory usage. To address this problem, various
federated pruning methods are proposed to reduce memory usage during inference.
However, few of them can substantially mitigate the memory burdens during
pruning and training. As an alternative, zeroth-order or backpropagation-free
(BP-Free) methods can partially alleviate the memory consumption, but they
suffer from scaling up and large computation overheads, since the gradient
estimation error and floating point operations (FLOPs) increase as the
dimensionality of the model parameters grows. In this paper, we propose a
federated foresight pruning method based on Neural Tangent Kernel (NTK), which
can seamlessly integrate with federated BP-Free training frameworks. We present
an approximation to the computation of federated NTK by using the local NTK
matrices. Moreover, we demonstrate that the data-free property of our method
can substantially reduce the approximation error in extreme data heterogeneity
scenarios. Since our approach improves the performance of the vanilla BP-Free
method with fewer FLOPs and truly alleviates memory pressure during training
and inference, it makes FL more friendly to low-memory devices. Comprehensive
experimental results obtained from simulation- and real test-bed-based
platforms show that our federated foresight-pruning method not only preserves
the ability of the dense model with a memory reduction up to 9x but also boosts
the performance of the vanilla BP-Free method with dramatically fewer FLOPs.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要