New Entropy-Based Vocabulary Optimization Approach For Chinese Language Modeling

This paper proposed a new entropy-based vocabulary optimization approach for Chinese language modeling. This approach alms to directly optimize the language model by extendm*g the vocabulary, that is, to minimize the character perplexity of the lanRuaRe model. A new criterion for new words selection was developed based on the character perplexity metric. A fast computing method and a simple divideand-conquer method were proposed to deal - I iments,v th very large corpus. Experi showed about 3% character perplexitv reduction and 3% character error rate reduction in a speech recognition task. Comparison experiments were also conducted to compare with other approaches.