Chrome Extension
WeChat Mini Program
Use on ChatGLM

Accelerated Training Via Device Similarity in Federated Learning

European Conference on Computer Systems (EuroSys)(2021)CCF B

Univ Minnesota

Cited 11|Views36
Abstract
Federated Learning is a privacy-preserving, machine learning technique that generates a globally shared model with in-situ model training on distributed devices. These systems are often comprised of millions of user devices and only a subset of available devices can be used for training in each epoch. Designing a device selection strategy is challenging, given that devices are highly heterogeneous in both their system resources and training data. This heterogeneity makes device selection very crucial for timely model convergence and sufficient model accuracy. Existing approaches have addressed system heterogeneity for device selection but have largely ignored the data heterogeneity. In this work, we analyze the impact of data heterogeneity on device selection, model convergence, model accuracy, and fault tolerance in a federated learning setting. Based on our analysis, we propose that clustering devices with similar data distributions followed by selecting the devices with the best processing capacity from each cluster can significantly improve the model convergence without compromising model accuracy. This clustering also guides us in designing policies for fault tolerance in the system. We propose three methods for identifying groups of devices with similar data distributions. We also identify and discuss rich trade-offs between privacy, bandwidth consumption, and computation overhead for each of these proposed methods. Our preliminary experiments show that the proposed methods can provide a 46% - 58% reduction in training time compared to existing approaches in reaching the same accuracy.
More
Translated text
PDF
Bibtex
AI Read Science
AI Summary
AI Summary is the key point extracted automatically understanding the full text of the paper, including the background, methods, results, conclusions, icons and other key content, so that you can get the outline of the paper at a glance.
Example
Background
Key content
Introduction
Methods
Results
Related work
Fund
Key content
  • Pretraining has recently greatly promoted the development of natural language processing (NLP)
  • We show that M6 outperforms the baselines in multimodal downstream tasks, and the large M6 with 10 parameters can reach a better performance
  • We propose a method called M6 that is able to process information of multiple modalities and perform both single-modal and cross-modal understanding and generation
  • The model is scaled to large model with 10 billion parameters with sophisticated deployment, and the 10 -parameter M6-large is the largest pretrained model in Chinese
  • Experimental results show that our proposed M6 outperforms the baseline in a number of downstream tasks concerning both single modality and multiple modalities We will continue the pretraining of extremely large models by increasing data to explore the limit of its performance
Try using models to generate summary,it takes about 60s
Must-Reading Tree
Example
Generate MRT to find the research sequence of this paper
Related Papers

Community-Structured Decentralized Learning for Resilient EI

Proceedings of the First Workshop on Systems Challenges in Reliable and Secure Federated Learning 2021

被引用0

Community-Aware Federated Video Summarization.

2023 INTERNATIONAL JOINT CONFERENCE ON NEURAL NETWORKS, IJCNN 2023

被引用0

Can Hierarchical Client Clustering Mitigate the Data Heterogeneity Effect in Federated Learning?

2023 IEEE INTERNATIONAL PARALLEL AND DISTRIBUTED PROCESSING SYMPOSIUM WORKSHOPS, IPDPSW 2023

被引用0

Communication Topologies for Decentralized Federated Learning

Michael Doetzer, Yixin Mao,Klaus Diepold
2023 EIGHTH INTERNATIONAL CONFERENCE ON FOG AND MOBILE EDGE COMPUTING, FMEC 2023

被引用0

Clustered Federated Learning Framework with Acceleration Based on Data Similarity.

ZhiPeng Gao, ZiJian Xiong,Chen Zhao, FuTeng Feng
ALGORITHMS AND ARCHITECTURES FOR PARALLEL PROCESSING, ICA3PP 2023, PT VII 2024

被引用0

Data Disclaimer
The page data are from open Internet sources, cooperative publishers and automatic analysis results through AI technology. We do not make any commitments and guarantees for the validity, accuracy, correctness, reliability, completeness and timeliness of the page data. If you have any questions, please contact us by email: report@aminer.cn
Chat Paper

要点】:该论文提出了一种基于设备相似性的联邦学习训练加速方法,通过分析数据异质性对设备选择、模型收敛性、准确性和容错能力的影响,提出了一种新的设备选择策略,以提高模型训练的效率而不牺牲准确性。

方法】:提出三种识别具有相似数据分布的设备组的方法,并对每种方法的隐私保护、带宽消耗和计算开销进行了详细的权衡分析。

实验】:初步实验结果表明,与现有方法相比,所提出的方法可以减少46%至58%的训练时间,同时达到相同的准确性。实验使用的数据集未在摘要中明确提及。