Pseudo-label Based Clustered Federated Learning with Non-IID Data

For non-independent and identically distributed (Non-IID) client data, clustered federated learning (CFL) addresses data heterogeneity by grouping clients with similar data distributions and training specialized models for each cluster. However, the existing CFL methods have the risk of privacy leakage due to the sharing of model parameters. Worse, they rely on rigid clustering schemes that struggle with boundary clients. To overcome these limitations, this paper proposes a Pseudo-Label based Clustered Federated Learning (PL-CFL) framework. PL-CFL computes client similarity from pseudo-label datasets generated by applying client models to a public unlabeled dataset. This eliminates any exchange of sensitive data or model parameters. The soft hierarchical clustering algorithm with virtual connections is designed in PL-CFL. It allows each client to belong to multiple clusters and effectively handles boundary cases. Additionally, cluster-wise consistency regularization aligns predictions within clusters, thereby reducing model divergence and improving generalization. Through theoretical analysis, the statistical stability of the pseudo-label similarity measure is established. And it shows that consistency regularization reduces intra-cluster generalization error. Finally, experiments are conducted on the MNIST and CIFAR-10 datasets with three Non-IID scenarios: Dirichlet-based partitioning, structured clustered partitioning, and label preference shifting. It demonstrates that PL-CFL consistently outperforms existing federated learning baselines in personalized accuracy and convergence speed.