Disentangling Data Distribution for Optimal and Communication-Efficient Federated Learning

Federated Learning (FL) facilitates collaborative training of a global model whose performance is boosted by private data owned by distributed clients, without compromising data privacy. Yet the wide applicability of FL is hindered by the entanglement of data distributions across different clients. This paper demonstrates for the first time that by disentangling data distributions, FL can in principle achieve efficiencies comparable to those of distributed systems, requiring only one round of communication. To this end, we propose a novel FedDistr algorithm, which employs diffusion models to decouple and recover data distributions. Empirical results on the CIFAR100, DomainNet, OfficeHome, and ISIC2020 datasets show that FedDistr significantly enhances model utility and efficiency in both disentangled and near-disentangled scenarios while ensuring privacy, outperforming traditional federated learning methods.