Multilingual KERMIT: It’s Not Easy Being Generative

We present multilingual KERMIT, a generative model over multiple languages. Multilingual KERMIT models the joint distribution over multiple languages, and all its decompositions using a single neural network. KERMIT can be trained by feeding it N way parallel-data, bilingual data, or monolingual data. At inference, KERMIT can generate translations for a particular target language, or up to N− 1 languages in parallel. It can also unconditionally generate sentences in multiple languages. Our experiments on the Multi30K dataset containing English, French, Czech, and German languages suggest that the multitask training with the joint objective leads to improvements in bilingual translations. We provide a quantitative analysis of the quality-diversity trade-offs for different variants of KERMIT for conditional generation, and a measurement of self-consistency during unconditional generation. We provide qualitative examples for parallel greedy decoding across languages and sampling from the joint distribution of the 4 languages.