LoRA-drop: Efficient LoRA Parameter Pruning based on Output Evaluation
CoRR(2024)
摘要
Low-Rank Adaptation (LoRA) introduces auxiliary parameters for each layer to
fine-tune the pre-trained model under limited computing resources. But it still
faces challenges of resource consumption when scaling up to larger models.
Previous studies employ pruning techniques by evaluating the importance of LoRA
parameters for different layers to address the problem. However, these efforts
only analyzed parameter features to evaluate their importance. Indeed, the
output of LoRA related to the parameters and data is the factor that directly
impacts the frozen model. To this end, we propose LoRA-drop which evaluates the
importance of the parameters by analyzing the LoRA output. We retain LoRA for
important layers and the LoRA of the other layers share the same parameters.
Abundant experiments on NLU and NLG tasks demonstrate the effectiveness of
LoRA-drop.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要