Merging Text Transformer Models from Different Initializations
arxiv(2024)
摘要
Recent work on one-shot permutation-based model merging has shown impressive
low- or zero-barrier mode connectivity between models from completely different
initializations. However, this line of work has not yet extended to the
Transformer architecture, despite its dominant popularity in the language
domain. Therefore, in this work, we investigate the extent to which separate
Transformer minima learn similar features, and propose a model merging
technique to investigate the relationship between these minima in the loss
landscape. The specifics of the architecture, like its residual connections,
multi-headed attention, and discrete, sequential input, require specific
interventions in order to compute model permutations that remain within the
same functional equivalence class. In merging these models with our method, we
consistently find lower loss barriers between minima compared to model
averaging for several models trained on a masked-language modeling task or
fine-tuned on a language understanding benchmark. Our results show that the
minima of these models are less sharp and isolated than previously understood,
and provide a basis for future work on merging separately trained Transformer
models.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要