It is known that federated learning (FL) incurs heavy communication overhead for model training by exchanging model updates between clients and the parameter server (PS) over the Internet for multiple rounds. Compressing model updates is an effective approach to alleviating communication overhead in FL. Yet the tradeoff between compression and model accuracy in the networked environment remains unclear and, for simplicity, most implementations adopt a fixed compression rate only during the entire learning process. In this paper, we for the first time systematically examine this tradeoff, explicitly quantifying the relation between the compression error, the final model accuracy and the learning rate. Specifically, we factor the compression error of each global iteration into the convergence rate analysis under non-convex loss for both unbiased and biased compression algorithms. We then present an adaptation framework to maximize the final model accuracy by strategically adjusting the compression rate in each iteration. We further discuss key implementation issues of our framework in practical networks with classical compression algorithms. Experiments over the most representative MNIST, CIFAR-10 and CIFAR-100 datasets demonstrate that our solutions effectively shrink network traffic volume while maintain high model accuracy in FL.

In Federated Learning (FL) paradigm, a parameter server (PS) concurrently communicates with distributed participating clients for model collection, update aggregation, and model distribution over multiple rounds, without touching private data owned by individual clients. FL is appealing in preserving data privacy; yet the communication between the PS and scattered clients can be a severe bottleneck. Model compression algorithms, such as quantization and sparsification, have been suggested but they generally assume a fixed code length, which does not reflect the heterogeneity and variability of model updates. In this paper, through both analysis and experiments, we show strong evidences that variable-length is beneficial for compression in FL. We accordingly present Fed-CVLC (Federated Learning Compression with Variable-Length Codes), which fine-tunes the code length in response of the dynamics of model updates. We develop optimal tuning strategy that minimizes the loss function (equivalent to maximizing the model utility) subject to the budget for communication. We further demonstrate that Fed-CVLC is indeed a general compression design that bridges quantization and sparsification, with greater flexibility. Extensive experiments have been conducted with public datasets to demonstrate that Fed-CVLC remarkably outperforms state-of-the-art baselines, improving model utility by 1.50%-5.44%, or shrinking communication traffic by 16.67%-41.61%.

Recently, federated learning (FL) has gained momentum because of its capability in preserving data privacy. To conduct model training by FL, multiple clients exchange model updates with a parameter server via Internet. To accelerate the communication speed, it has been explored to deploy a programmable switch (PS) in lieu of the parameter server to coordinate clients. The challenge to deploy the PS in FL lies in its scarce memory space, prohibiting running memory consuming aggregation algorithms on the PS. To overcome this challenge, we propose Federated Learning in-network Aggregation with Compression (FediAC) algorithm, consisting of two phases: client voting and model aggregating. In the former phase, clients report their significant model update indices to the PS to estimate global significant model updates. In the latter phase, clients upload global significant model updates to the PS for aggregation. FediAC consumes much less memory space and communication traffic than existing works because the first phase can guarantee consensus compression across clients. The PS easily aligns model update indices to swiftly complete aggregation in the second phase. Finally, we conduct extensive experiments by using public datasets to demonstrate that FediAC remarkably surpasses the state-of-the-art baselines in terms of model accuracy and communication traffic.

Federated learning (FL) has been extensively exploited in the training of machine learning models to preserve data privacy. In particular, wireless FL enables multiple clients to collaboratively train models by sharing model updates via wireless communication without exposing raw data. The state-of-the-art wireless FL advocates efficient aggregation of model updates from multiple clients by over-the-air computing. However, a significant deficiency of over-the-air aggregation lies in the infeasibility of TopK model compression given that top model updates cannot be aggregated directly before they are aligned according to their indices. In view of the fact that TopK can greatly accelerate FL, we design a novel wireless FL with voting based TopK algorithm, namely WFL-VTopK, so that top model updates can be aggregated by over-the-air computing directly. Specifically, there are two phases in WFL-VTopK. In Phase 1, clients vote their top model updates, based on which global top model updates can be efficiently identified. In Phase 2, clients formally upload global top model updates so that they can be directly aggregated by over-the-air computing. Furthermore, the convergence of WFL-VTopK is theoretically guaranteed under non-convex loss. Based on the convergence of WFL-VTopK, we optimize model utility subjecting to training time and energy constraints. To validate the superiority of WFL-VTopK, we extensively conduct experiments with real datasets under wireless communication. The experimental results demonstrate that WFL-VTopK can effectively aggregate models by only communicating 1%-2% top models updates, and hence significantly outperforms the state-of-the-art baselines. By significantly reducing the wireless communication traffic, our work paves the road to train large models in wireless FL.

Recently, federated learning (FL) has received tremendous attention in both academia and industry, in which decentralized clients collaboratively complete model training by exchanging model updates with a parameter server through the Internet. Its distributed nature well utilizes the localized data and preserves clients’ privacy, but also incurs heavy communication overhead. Existing studies on model update have mostly focused on the bandwidth constraint of the communication channels. Today's Internet however is highly unreliable. Simply using Transmission Control Protocol (TCP) would lead to low network utilization under frequent losses. In this paper, we closely examine the optimal transmission strategies in FL over the realistic lossy Internet. We systematically integrate model compression, forward error correction (FEC) and retransmission towards Federated Learning with Lossy Communications (FedLC). We derive the convergence rate of FedLC under non-convex loss with the optimal transmission. We then decompose this non-convex problem and present effective practical solutions. Public datasets are exploited for performance evaluation by varying the packet loss rate from 10% to 50%. In a fixed training time budget, FedLC can improve model accuracy by 3.91% on average or reduce the communication traffic by 34.27%-47.57% in comparison with state-of-the-art baselines.

Edge computing architectures can help us quickly process the data collected by Internet of Things (IoT) and caching files to edge nodes can speed up the response speed of IoT devices requesting files. Blockchain architectures can help us ensure the security of data transmitted by IoT. Therefore, we have proposed a system that combines IoT devices, edge nodes, remote cloud, and blockchain. In the system, we designed a new algorithm in which blockchain-assisted compressed algorithm of federated learning is applied for content caching, called CREAT to predict cached files. In the CREAT algorithm, each edge node uses local data to train a model and then uses the model to learn the features of users and files, so as to predict popular files to improve the cache hit rate. In order to ensure the security of edge nodes’ data, we use federated learning (FL) to enable multiple edge nodes to cooperate in training without sharing data. In addition, for the purpose of reducing communication load in FL, we will compress gradients uploaded by edge nodes to reduce the time required for communication. What is more, in order to ensure the security of the data transmitted in the CREAT algorithm, we have incorporated blockchain technology in the algorithm. We design four smart contracts for decentralized entities to record and verify the transactions to ensure the security of data. We used MovieLens data sets for experiments and we can see that CREAT greatly improves the cache hit rate and reduces the time required to upload data.

Recently, blockchain-based federated learning (BFL) has attracted intensive research attention due to that the training process is auditable and the architecture is serverless avoiding the single point failure of the parameter server in vanilla federated learning (VFL). Nevertheless, BFL tremendously escalates the communication traffic volume because all local model updates (i.e., changes of model parameters) obtained by BFL clients will be transmitted to all miners for verification and to all clients for aggregation. In contrast, the parameter server and clients in VFL only retain aggregated model updates. Consequently, the huge communication traffic in BFL will inevitably impair the training efficiency and hinder the deployment of BFL in reality. To improve the practicality of BFL, we are among the first to propose a fast blockchain-based communication-efficient federated learning framework by compressing communications in BFL, called BCFL. Meanwhile, we derive the convergence rate of BCFL with non-convex loss. To maximize the final model accuracy, we further formulate the problem to minimize the training loss of the convergence rate subject to a limited training time with respect to the compression rate and the block generation rate, which is a bi-convex optimization problem and can be efficiently solved. To the end, to demonstrate the efficiency of BCFL, we carry out extensive experiments with standard CIFAR-10 and FEMNIST datasets. Our experimental results not only verify the correctness of our analysis, but also manifest that BCFL can remarkably reduce the communication traffic by 95–98% or shorten the training time by 90–95% compared with BFL.

Federated Learning (FL) incurs high communication overhead, which can be greatly alleviated by compression for model updates. Yet the tradeoff between compression and model accuracy in the networked environment remains unclear and, for simplicity, most implementations adopt a fixed compression rate only. In this paper, we for the first time systematically examine this tradeoff, identifying the influence of the compression error on the final model accuracy with respect to the learning rate. Specifically, we factor the compression error of each global iteration into the convergence rate analysis under both strongly convex and non-convex loss functions. We then present an adaptation framework to maximize the final model accuracy by strategically adjusting the compression rate in each iteration. We have discussed the key implementation issues of our framework in practical networks with representative compression algorithms. Experiments over the popular MNIST and CIFAR-10 datasets confirm that our solution effectively reduces network traffic yet maintains high model accuracy in FL.

Federated Learning (FL) is an emerging decentralized learning framework through which multiple clients can collaboratively train a learning model. However, a major obstacle that impedes the wide deployment of FL lies in massive communication traffic. To train high dimensional machine learning models (such as CNN models), heavy communication traffic can be incurred by exchanging model updates via the Internet between clients and the parameter server (PS), implying that the network resource can be easily exhausted. Compressing model updates is an effective way to reduce the traffic amount. However, a flexible unbiased compression algorithm applicable for both uplink and downlink compression in FL is still absent from existing works. In this work, we devise the Model Update Compression by Soft Clustering (MUCSC) algorithm to compress model updates transmitted between clients and the PS. In MUCSC, it is only necessary to transmit cluster centroids and the cluster ID of each model update. Moreover, we prove that: 1) The compressed model updates are unbiased estimation of their original values so that the convergence rate by transmitting compressed model updates is unchanged; 2) MUCSC can guarantee that the influence of the compression error on the model accuracy is minimized. Then, we further propose the boosted MUCSC (B-MUCSC) algorithm, a biased compression algorithm that can achieve an extremely high compression rate by grouping insignificant model updates into a super cluster. B-MUCSC is suitable for scenarios with very scarce network resource. Ultimately, we conduct extensive experiments with the CIFAR-10 and FEMNIST datasets to demonstrate that our algorithms can not only substantially reduce the volume of communication traffic in FL, but also improve the training efficiency in practical networks.

Recently, Federated Learning (FL) has drawn tremendous attentions due to its ability to protect client's privacy. In FL, clients collaboratively train machine learning models by merely sharing intermediate computations, i.e., gradients of model parameters. However, training a complicated model involves multiple rounds of interactions between clients and the server via the Internet. Consequently, communication is a primary bottleneck of FL attributed to the poor network conditions and the large amount of interchanged computations. To overcome the communication bottleneck, we propose the ClusterGrad algorithm to compress gradients which can considerably reduce the volume of communicated computations. Our design is based on the fact that there is only a small fraction of gradients whose values are far away from the origin in each round of interaction in FL. We first identify these essential gradients that are far away from 0 using the K-means algorithm. These gradient values are approximated by a novel clustering based quantization algorithm. Then, the rest gradients lying close to 0 are approximated with a single value. We can prove that ClusterGrad outperforms the latest FL gradient compression algorithms: Probability Quantization (PQ) and Deep Gradient Compression (DGC). We conduct extensive experiments with the CIFAR-10 datasets which further demonstrate that ClusterGrad can achieve compression ratio (used interchangeably with compression rate) 123 on average in comparison with PQ and DGC with compression ratios 16 and 60 respectively.