Automated Federated Pipeline for Parameter-Efficient Fine-Tuning of Large Language Models
arxiv(2024)
摘要
Recently, there has been a surge in the development of advanced intelligent
generative content (AIGC), especially large language models (LLMs). However,
for many downstream tasks, it is necessary to fine-tune LLMs using private
data. While federated learning offers a promising privacy-preserving solution
to LLM fine-tuning, the substantial size of an LLM, combined with high
computational and communication demands, makes it hard to apply to downstream
tasks. More importantly, private edge servers often possess varying computing
and network resources in real-world scenarios, introducing additional
complexities to LLM fine-tuning. To tackle these problems, we design and
implement an automated federated pipeline, named FedPipe, to fine-tune LLMs
with minimal training cost but without adding any inference latency. FedPipe
firstly identifies the weights to be fine-tuned based on their contributions to
the LLM training. It then configures a low-rank adapter for each selected
weight to train local low-rank adapters on an edge server, and aggregate local
adapters of all edge servers to fine-tune the whole LLM. Finally, it
appropriately quantizes the parameters of LLM to reduce memory space according
to the requirements of edge servers. Extensive experiments demonstrate that
FedPipe expedites the model training and achieves higher accuracy than
state-of-the-art benchmarks.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要