A Morphology-Based Investigation of Positional Encodings
arxiv(2024)
摘要
How does the importance of positional encoding in pre-trained language models
(PLMs) vary across languages with different morphological complexity? In this
paper, we offer the first study addressing this question, encompassing 23
morphologically diverse languages and 5 different downstream tasks. We choose
two categories of tasks: syntactic tasks (part-of-speech tagging, named entity
recognition, dependency parsing) and semantic tasks (natural language
inference, paraphrasing). We consider language-specific BERT models trained on
monolingual corpus for our investigation. The main experiment consists of
nullifying the effect of positional encoding during fine-tuning and
investigating its impact across various tasks and languages. Our findings
demonstrate that the significance of positional encoding diminishes as the
morphological complexity of a language increases. Across all experiments, we
observe clustering of languages according to their morphological typology -
with analytic languages at one end and synthetic languages at the opposite end.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要