Self-supervised learning in NLP虽然计算机视觉在最近几年才在自监督学习方面取得了令人惊叹的进展,但我监督学习在很长一段时间内一直是NLP研究的一等公民。语言模型早在90年代就已经存在了,甚至在“自监督学习”这个词被定义之前就已经存在了。2013年的word2vec论文普及了这一范式,该领域在许多问题上应用这些自我监督的方法取得了快速进展。
Yaochen Xie, Zhao Xu, Zhengyang Wang, Shuiwang Ji
We summarize current self-supervised learning methods and provide unified reviews for the two approaches
Cited by0BibtexViews24
0
0
ICLR, (2020)
We have convincing evidence that sentence order prediction is a more consistently-useful learning task that leads to better language representations, we hypothesize that there could be more dimensions not yet captured by the current self-supervised training losses that could crea...
Cited by1071BibtexViews1000
0
0
NIPS 2020, (2020)
Similar approaches are common in NLP, we demonstrate that this approach can be a surprisingly strong baseline for semi-supervised learning in computer vision, outperforming the state-of-the-art by a large margin
Cited by130BibtexViews1721
0
0
ICLR, (2020)
We evaluate models on two benchmarks: TIMIT is a 5h dataset with phoneme labels and Wall Street Journal is a 81h dataset for speech recognition
Cited by85BibtexViews373
0
0
NIPS 2020, (2020)
We presented wav2vec 2.0, a framework for self-supervised learning of speech representations which masks latent representations of the raw waveform and solves a contrastive task over quantized speech representations
Cited by67BibtexViews840
0
0
Ravanelli Mirco, Zhong Jianyuan, Pascual Santiago, Swietojanski Pawel, Monteiro Joao, Trmal Jan,Bengio Yoshua
ICASSP, pp.6989-6993, (2020)
The proposed problemagnostic speech encoder+ architecture is based on an online speech distortion module, a convolutional encoder coupled with a quasi-recurrent neural network layer, and a set of workers solving self-supervised problems
Cited by45BibtexViews316DOI
0
0
There exist several comprehensive reviews related to Pre-trained Language Models, Generative Adversarial Networks, Autoencoder and contrastive learning for visual representation
Cited by32BibtexViews5254
0
0
Shaolei Wang,Wanxiang Che,Qi Liu, Pengda Qin,Ting Liu,William Yang Wang
national conference on artificial intelligence, (2020)
Experimental results on the commonly used English Switchboard test set show that our approach can achieve competitive performance compared to the previous systems by using less than 1% of the training data
Cited by10BibtexViews348
0
0
William Falcon,Kyunghyun Cho
We evaluated each key design choice of augmented multi-scale deep information maximization and contrastive predictive coding, which are two representative Contrastive self-supervised learning algorithms, and used our framework to construct a new approach to which we refer as Yet ...
Cited by9BibtexViews12
0
0
We studied self-supervised learning via an information-theoretical perspective
Cited by7BibtexViews752
0
0
NIPS 2020, (2020)
Our results demonstrate the emergence of powerful, high-level visual representations from developmentally realistic natural videos using generic self-supervised learning objectives
Cited by1BibtexViews69
0
0
Klein Tassilo,Nabi Moin
ACL, pp.7517-7523, (2020)
It is less susceptible to gender and number biases as the performance on KnowRef suggests. All this taken together confirms that selfsupervision is possible for commonsense reasoning tasks
Cited by0BibtexViews43
0
0
Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee,Fillia Makedon
Technologies, no. 1 (2020): 2
The works based on contrastive learning have shown promising results on several downstream tasks such as image/video classification, object detection, and other natural language processing tasks
Cited by0BibtexViews20DOI
0
0
Kun Zhou, Hui Wang,Wayne Xin Zhao, Yutao Zhu, Sirui Wang, Fuzheng Zhang, Zhongyuan Wang,Ji-Rong Wen
CIKM '20: The 29th ACM International Conference on Information and Knowledge Management Virt..., pp.1893-1902, (2020)
Non-sequential recommendation methods perform worse than sequential recommendation methods, since the sequential pattern is important to consider in our task
Cited by0BibtexViews42DOI
0
0
Jason D. Lee, Qi Lei, Nikunj Saunshi, Jiacheng Zhuo
We leave it for future work to investigate other mechanisms by which pretext tasks help with downstream tasks
Cited by0BibtexViews10
0
0
Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rihawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Debsindhu Bhowmik,Burkhard Rost
Up-scaling Language Models to the enormous sizes of protein databases on Summit threw up six main challenges that we addressed as follows
Cited by0BibtexViews32DOI
0
0
Andy T. Liu, Shang-Wen Li, Hung-yi Lee
We propose a novel multi-target self-supervised training scheme called TERA, where we use multiple auxiliary objectives instead of one during pre-training
Cited by0BibtexViews11
0
0
IJCAI 2020, pp.1083-1089, (2020)
We believe that our work can be a meaningful step in realistic Visual Question Answering and solving the language bias issue, and this self-supervision can be generalized to other tasks that are subject to the inherent data biases
Cited by0BibtexViews355DOI
0
0
Fang Hongchao, Xie Pengtao
On three natural lanuage understanding tasks, CERT outperforms BERT significantly, demonstrating the effectiveness of contrastive self-supervised learning for language representation
Cited by0BibtexViews9
0
0
Chi Po-Han, Chung Pei-Hung, Wu Tsung-Han, Hsieh Chun-Cheng, Li Shang-Wen, Lee Hung-yi
We present a novel model, Audio ALBERT
Cited by0BibtexViews12
0
0