Efficient Contextualized Representation: Language Model Pruning for Sequence LabelingEI
Many efforts have been made to facilitate natural language processing tasks with pre-trained language models (PTLM), and brought significant improvements to various applications. To fully leverage the nearly unlimited corpora and capture linguistic information of multifarious levels, large-size LMs are required; but for a specific task, only parts of these information are useful. Such large models, even in the inference stage, lead to overwhelming computation workloads, thus making them too time-consuming for real-world applications. ...更多
- 6Song Han, Jeff Pool, John Tran, William J Dally. Learning both Weights and Connections for Efficient Neural Networks.neural information processing systems, pp. 1135-1143, 2015.
- 7Lev-Arie Ratinov, Dan Roth. Design challenges and misconceptions in named entity recognition.CoNLL, pp. 147-155, 2009.
- 8Erik F. Tjong Kim Sang, Sabine Buchholz. Introduction to the CoNLL-2000 Shared Task Chunking.CoNLL/LLL, pp. 127-132, 2000.
- 9Walter Murray, Kien-Ming Ng. An algorithm for nonlinear optimization problems with binary variables.Computational Optimization and Applications, pp. 257-288, 2010.
- 11Jason P C Chiu, Eric Nichols. Named Entity Recognition with Bidirectional LSTM-CNNs.TACL, 2015.
- 14Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, Chris Dyer. Neural Architectures for Named Entity Recognition.HLT-NAACL, 2016.
- 18Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, Hai Li. Learning Structured Sparsity in Deep Neural Networks.NIPS, pp. 2074-2082, 2016.
- 19Kazuyuki Hara, Daisuke Saitoh, Hayaru Shouno. Analysis of Dropout Learning Regarded as Ensemble Learning.CoRR, 2017.
- 22Alec Radford, Rafal Józefowicz, Ilya Sutskever. Learning to Generate Reviews and Discovering Sentiment.CoRR, 2017.
- 23Marek Rei, Semi-supervised Multitask Learning for Sequence Labeling.ACL, 2017.
- 24Matthew E. Peters, Waleed Ammar, Chandra Bhagavatula, Russell Power. Semi-supervised sequence tagging with bidirectional language models.ACL, 2017.
- 25Fréderic Godin, Joni Dambre, Wesley De Neve. Improving Language Modeling using Densely Connected Recurrent Neural Networks.Rep4NLP@ACL, pp. 186-190, 2017.
- 26Gábor Melis, Chris Dyer, Phil Blunsom. On the State of the Art of Evaluation in Neural Language Models.CoRR, 2017.
- 28Zhongliang Li, Raymond Kulhanek, Shaojun Wang, Yunxin Zhao, Shuang Wu. Slim Embedding Layers for Recurrent Neural Language Models.AAAI, 2017.