LightFormer: Light-weight Transformer Using SVD-based Weight Transfer and Parameter Sharing.
ACL (Findings)(2023)
摘要
Transformer has become an important technique for natural language processing tasks with great success.However, it usually requires huge storage space and computational cost, making it difficult to be deployed on resourceconstrained edge devices.To compress and accelerate Transformer, we propose LightFormer, which adopts a low-rank factorization initialized by SVD-based weight transfer and parameter sharing.The SVD-based weight transfer can effectively utilize the well-trained Transformer parameter knowledge to speed up the model convergence, and effectively alleviate the lowrank bottleneck problem combined with parameter sharing.We validate this method on machine translation, text summarization, and text classification tasks.Experiments show that on IWSLT'14 De-En and WMT'14 En-De, LightFormer achieves similar performance to the baseline Transformer with 3.8× and 1.8× fewer parameters, and achieves 2.3× speedup and 1.5× speedup respectively, generally outperforming recent light-weight Transformers.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要