Cross-lingual language models enable the transfer of linguistic knowledge across languages, however, they often perform worse for low-resource or typologically distant languages. Prior work has explored alignment and adapter methods, but the use of code-switching remains limited and typically confined to fine-tuning with static word substitutions. In this work, we propose an approach that integrates code-switching directly into masked language model pretraining. Instead of applying word substitutions after pretraining, we introduce a multiview probabilistic translation strategy that samples candidate translations based on alignment likelihoods, applying substitutions only to unmasked tokens. This exposes the model to cross-lingual ambiguity and encourages more robust cross-lingual representations. Our results on a diverse set of eight language pairs show that this approach improves zero-shot cross-lingual natural language understanding performance across all languages relative to bilingual baselines. We further observe gains on downstream named entity recognition tasks in most languages when incorporating our code-switched pretraining approach.
原文链接
分享
引用
