Using a Neural Network Language Model

In practice, training language models for individual authors is often expensive because of limited data resources. In such cases, Neural Network Language Models (NNLMs), generally outperform the traditional non-parametric N-gram models. Here we investigate the performance of a feed-forward NNLM on an authorship attribution problem, with moderate author set size and relatively limited data. We also consider how the text topics impact performance. Compared with a well-constructed N-gram baseline method with Kneser-Ney smoothing, the proposed method achieves nearly 2.5% reduction in perplexity and increases author classification accuracy by 3.43% on average, given as few as 5 test sentences. The performance is very competitive with the state of the art in terms of accuracy and demand on test data. The source code, preprocessed datasets, a detailed description of the methodology and results are available at https: //github.com/zge/authorship-attribution. Introduction Authorship attribution refers to identifying authors from given texts by their unique textual features. It is challenging since the author’s style may vary from time to time by topics, mood and environment. Many methods have been explored to address this problem, such as Latent Dirichlet Allocation for topic modeling (Seroussi, Zukerman, and Bohnert 2011) and Naive Bayes for text classification (Coyotl-Morales et al. 2006). Regarding language modeling methods, there is mixed advocacy for the conventional N-gram methods (Kešelj et al. 2003) and methods using more compact and distributed representations, like Neural Network Language Models (NNLMs), which was claimed to capture semantics better with limited training data (Bengio et al. 2003). Most NNLM toolkits available (Mikolov et al. 2010) are designed for recurrent NNLMs which are better for capturing complex and longer text patterns and require more training data. In contrast, the feed-forward NNLM framework we proposed is less computationally expensive and more suitable for language modeling with limited data. It is developed in MATLAB with full network tuning functionalities. The database we use is composed of transcripts of 16 video courses taken from Coursera, collected one sentence per line into a text file for each course. To reduce the influence of “topic” on author/instructor classification, courses were all selected from science and engineering fields, such Copyright c © 2015, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. as Algorithm, DSP, Data Mining, IT, Machine Learning, NLP, etc. There are 8000+ sentences/course and about 20 words/sentence on average. The vocabulary size of each author varies from 3000 to 9000. After stemming with Porter’s algorithm and pruning words with frequency less than 1/100, 000, author vocabulary size is reduced to a range from 1800 to 2700, with average size around 2000. Fig. 1 shows the vocabulary size for each course, under various conditions and the database coverage with the most frequent k words (k = 500, 1000, 2000) after stemming and pruning. 0 2 4 6 8 10 12 14 16 0 500