Learning FOFE based FNN-LMs with noise contrastive estimation and part-of-speech features

A simple but powerful language model called fixed-size ordinally-forgetting encoding (FOFE) based feedforward neural network language models (FNN-LMs) has been proposed recently. Experimental results have shown that FOFE based FNN-LMs can outperform not only the standard FNN-LMs but also the popular recurrent neural network language models (RNN-LMs). In this paper, we extend FOFE based FNN-LMs from several aspects. Firstly, we have proposed a new method to further improve the performance of FOFE based FNN-LMs by adding transitions of part-of-speech (POS) tags as additional features. Secondly, we have investigated how to speedup the learning of FOFE based FNN-LMs by using noise contrastive estimation (NCE). As a result, we can dramatically speedup the learning of FOFE based FNN-LMs while we still achieve very competitive experimental results on Large Text Compression Benchmark (LTCB).