Transfer learning in NLP迁移学习(Transfer Learning)无疑是目前深度学习中的新热点。在NLP中,迁移学习主要限于使用预训练的单词嵌入(这大大改善了基线)。最近,研究人员正在努力将整个模型从一项任务转移到另一项任务。Sebastian Ruder和Jeremy Howard是第一个通过其提出的ULMFiT方法,在NLP中的应用了迁移学习方法,该方法超越了所有最新的文本分类技术。紧接着,OpenAI 在几个NLP任务上扩大了他们的想法,并超越了SOTA。在2018年NAACL上,获得最佳论文奖的是介绍ELMo的论文,该论文是一种新的词嵌入技术,与ULMFiT背后的思想非常相似,该技术来自位于UWash的AllenAI和 Luke Zettlemoyer小组的研究人员。
Raffel Colin,Shazeer Noam, Roberts Adam,Lee Katherine, Narang Sharan, Matena Michael, Zhou Yanqi,Li Wei,Liu Peter J.
JOURNAL OF MACHINE LEARNING RESEARCH, no. 140 (2020): 1-67
While many modern approaches to transfer learning for natural language processing use a Transformer architecture consisting of only a single “stack”, we found that using a standard encoder-decoder structure achieved good results on both generative and classification tasks
Cited by866BibtexViews378
0
0
arxiv, (2020)
We find that acceptability judgment probing task performance is generally uncorrelated with the target task performance, except for AJ-Corpus of Linguistic Acceptability
Cited by27BibtexViews106
0
0
Alham Fikri Aji, Nikolay Bogoychev,Kenneth Heafield,Rico Sennrich
ACL, pp.7701-7710, (2020)
Transfer learning is a common method for lowresource neural machine translation
Cited by8BibtexViews61
0
0
Hill Felix, Mokra Sona, Wong Nathaniel, Harley Tim
Recent work has described neural-network-based agents that are trained with reinforcement learning (RL) to execute language-like commands in simulated worlds, as a step towards an intelligent agent or robot that can be instructed by human users. However, the optimisation of mul...
Cited by0BibtexViews1
0
0
EMNLP, pp.441-459, (2020)
We have presented a Versatile Language Model which learns five diverse natural language generation tasks in a single model
Cited by0BibtexViews27DOI
0
0
pp.15-18 (2019)
Introduction: This section will introduce the theme of the tutorial: how transfer learning is used in current Natural Language Processing
Cited by98BibtexViews99
0
0
ACL (1), pp.3098-3112, (2019)
We propose multinomial adversarial network-MoE, a multilingual model transfer approach that exploits both language-invariant features and language-specific features, which departs from most previous models that can only make use of shared features
Cited by29BibtexViews72DOI
0
0
Yifan Peng, Shankai Yan,Zhiyong Lu
SIGBIOMED WORKSHOP ON BIOMEDICAL NATURAL LANGUAGE PROCESSING (BIONLP 2019), (2019): 58-65
We introduce Biomedical Language Understanding Evaluation, a collection of resources for evaluating and analyzing biomedical natural language representation models
Cited by29BibtexViews32DOI
0
0
arXiv: Computation and Language, (2019)
SiATL avoids catastrophic forgetting of the language distribution learned by the pretrained Language Models
Cited by29BibtexViews1
0
0
Hirofumi Inaguma, Jaejin Cho, Murali Karthick Baskar,Tatsuya Kawahara,Shinji Watanabe
international conference on acoustics, speech, and signal processing, (2019)
We explored the usage of linguistic context from the external language model during adaptation of the language-independent S2S model to target low-resource languages
Cited by18BibtexViews8DOI
0
0
THIRTY-THIRD AAAI CONFERENCE ON ARTIFICIAL INTELLIGENCE / THIRTY-FIRST INNOVATIVE APPLICATIONS OF AR..., (2019): 4959-4966
We showed that ELMoL performance is comparable to Embeddings from Language Model, and it is faster at runtime and better suited for practical Spoken Language Understanding systems
Cited by13BibtexViews29
0
0
INTERSPEECH, pp.1198-1202, (2019)
We investigate the capacity of domain portability brought by our approach, that consists on starting from an existing spoken language understanding model dedicated to a task, MEDIA, in order to build a new SLU model dedicated to another task, PORT-MEDIA
Cited by10BibtexViews31DOI
0
0
Sergey Golovanov, Rauf Kurbanov,Sergey Nikolenko, Kyryl Truskovskyi, Alexander Tselousov,Thomas Wolf
ACL (1), pp.6053-6058, (2019)
We have presented various ways in which large-scale pretrained language models can be adapted to natural language generation tasks, comparing single-input and multi-input solutions
Cited by9BibtexViews44DOI
0
0
Thomas Wolf,Victor Sanh, Julien Chaumond, Clement Delangue
arXiv: Computation and Language, (2019)
Transfer learning from language models have been recently shown to bring strong empirical improvements in discriminative language understanding tasks
Cited by1BibtexViews7
0
0
Mozafari Marzieh,Farahbakhsh Reza, Crespi Noel
COMPLEX NETWORKS (1), pp.928-940, (2019)
We introduce new fine-tuning strategies to examine the effect of different layers of BERT in hate speech detection task
Cited by0BibtexViews8DOI
0
0
Malte Aditya, Ratadiya Pratik
We have provided a lucid summary of recent advances in the domain of transfer learning in the domain of natural language processing
Cited by0BibtexViews28
0
0
MLHC, pp.383-402, (2018)
We show that such pretraining of the named entity recognition model weights is a good initialization strategy for the optimizer as it leads to substantial improvements in the F1 scores for four benchmark datasets
Cited by33BibtexViews63
0
0
Raj Dabre, Tetsuji Nakagawa,Hideto Kazawa
PACLIC, pp.282-286, (2017)
We presented our work on an empirical study of language relatedness for transfer learning in Neural Machine Translation
Cited by16BibtexViews10
0
0
international joint conference on natural language processing, (2017)
The source word embeddings are copied with the rest of the model, with the ith parent-language word embedding being assigned to the ith childlanguage word
Cited by5BibtexViews26
0
0
EMNLP, (2016): 1568-1575
We show that Neural machine translation is an exceptional re-scorer of ‘traditional’ MT output; even Neural machine translation that on its own is worse than syntax based machine translation is consistently able to improve upon syntax based machine translation system output when ...
Cited by314BibtexViews91DOI
0
0