ColA: Collaborative Adaptation with Gradient Learning
CoRR(2024)
摘要
A primary function of back-propagation is to compute both the gradient of
hidden representations and parameters for optimization with gradient descent.
Training large models requires high computational costs due to their vast
parameter sizes. While Parameter-Efficient Fine-Tuning (PEFT) methods aim to
train smaller auxiliary models to save computational space, they still present
computational overheads, especially in Fine-Tuning as a Service (FTaaS) for
numerous users. We introduce Collaborative Adaptation (ColA) with Gradient
Learning (GL), a parameter-free, model-agnostic fine-tuning approach that
decouples the computation of the gradient of hidden representations and
parameters. In comparison to PEFT methods, ColA facilitates more cost-effective
FTaaS by offloading the computation of the gradient to low-cost devices. We
also provide a theoretical analysis of ColA and experimentally demonstrate that
ColA can perform on par or better than existing PEFT methods on various
benchmarks.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要