Byte Pair Encoding Is All You Need For Automatic Bengali Speech Recognition
CoRR(2024)
摘要
Byte pair encoding (BPE) emerges as an effective tokenization method for
tackling the out-of-vocabulary (OOV) challenge in various natural language and
speech processing tasks. Recent research highlights the dependency of BPE
subword tokenization's efficacy on the morphological nature of the language,
particularly in languages rich in inflectional morphology, where fewer BPE
merges suffice for generating highly productive tokens. Motivated by this, our
study empirically identifies the optimal number of BPE tokens for Bengali, a
language known for its morphological complexity, thus enhancing
out-of-distribution automatic speech recognition (ASR) performance.
Experimental evaluation reveals that an excessively high number of BPE tokens
can lead to overfitting, while approximately 500-1000 tokens result in superior
OOV performance. Furthermore, we conduct a comparative analysis of BPE with
character-based and unigram-based tokenization methods. By introducing BPE
tokenization to Bengali ASR, we achieve a substantial reduction in the word
error rate (WER) from 66.44
on the LB-ASRTD eval set and from 46.34
of which include out-of-distribution data.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要