Effectiveness of Pre-Trained Language Models for the Japanese Winograd Schema Challenge.

This paper compares Japanese and multilingual lan-guage models (LMs) in a Japanese pronoun reference resolution task to determine the factors of LMs that contribute to Japanese pronoun resolution. Specifi-cally, we tackle the Japanese Winograd schema chal-lenge task (WSC task), which is a well-known pronoun reference resolution task. The Japanese WSC task requires inter-sentential analysis, which is more chal-lenging to solve than intra-sentential analysis. A pre-vious study evaluated pre-trained multilingual LMs in terms of training language on the target WSC task, including Japanese. However, the study did not per-form pre-trained LM-wise evaluations, focusing on the training language-wise evaluations with a multilingual WSC task. Furthermore, it did not investigate the effectiveness of factors (e.g., model size, learning set-tings in the pre-training phase, or multilingualism) to improve the performance. In our study, we com-pare the performance of inter-sentential analysis on the Japanese WSC task for several pre-trained LMs, including multilingual ones. Our results confirm that XLM, a pre-trained LM on multiple languages, per-forms the best among all considered LMs, which we at-tribute to the amount of data in the pre-training phase.