SKEP: Sentiment Knowledge Enhanced Pre-training for Sentiment Analysis

ACL, pp. 4067-4076, 2020.

Cited by: 0|Bibtex|Views98
EI
Other Links: arxiv.org|dblp.uni-trier.de|academic.microsoft.com
Weibo:
We propose Sentiment Knowledge Enhanced Pre-training for sentiment analysis

Abstract:

Recently, sentiment analysis has seen remarkable advance with the help of pre-training approaches. However, sentiment knowledge, such as sentiment words and aspect-sentiment pairs, is ignored in the process of pre-training, despite the fact that they are widely used in traditional sentiment analysis approaches. In this paper, we introdu...More
0
Introduction
  • Sentiment analysis refers to the identification of sentiment and opinion contained in the input texts that are often user-generated comments.
  • Pre-training methods (Peters et al, 2018; Radford et al, 2018; Devlin et al, 2019; Yang et al, 2019) have shown their powerfulness in learning general semantic representations, and have remarkably improved most natural language processing (NLP) tasks like sentiment analysis
  • These methods build unsupervised objectives at word-level, such as masking strategy (Devlin et al, 2019), next-word prediction (Radford et al, 2018) or permutation (Yang et al, 2019).
  • In fine-tuning step, the final hidden state of [CLS] is often used as the overall semantic representation of the input sequence
Highlights
  • Sentiment analysis refers to the identification of sentiment and opinion contained in the input texts that are often user-generated comments
  • In order to learn a unified sentiment representation for multiple sentiment analysis tasks, we propose Sentiment Knowledge Enhanced Pre-training (SKEP), where sentiment knowledge about words, polarity, and aspect-sentiment pairs are included to guide the process of pre-training
  • We propose sentiment knowledge enhanced pre-training for sentiment analysis, which provides a unified sentiment representation for multiple sentiment analysis tasks
  • We propose Sentiment Knowledge Enhanced Pre-training, Sentiment Knowledge Enhanced Pre-training, which incorporates sentiment knowledge by self-supervised training
  • We propose Sentiment Knowledge Enhanced Pre-training for sentiment analysis
Results
  • The authors compare the SKEP method with the strong pretraining baseline RoBERTa and previous SOTA.
  • Comparing with RoBERTa, SKEP significantly and consistently improves the performance on both From SST-2 Sem-L Model RoBERTa. Sentence Samples altogether , this is :su:c:c:e:s:s:f:u:l as a film , while at the same time being a most touching reconsideration of the familiar :m:a:s:t:e:rp:i:e:c:e .
  • Altogether , this is :su:c:c:e:s:s:f:u:l as a film , while at the same time being a most touching reconsideration of the familiar :m:a:s:t:e:rp:i:e:c:e.
  • The author got this at an :am::a:z:i:n:g price from Amazon and it arrived just in time .
Conclusion
  • The authors propose Sentiment Knowledge Enhanced Pre-training for sentiment analysis.
  • Sentiment masking and three sentiment pre-training objectives are designed to incorporate various types of knowledge for pre-training model.
  • The authors' work verifies the necessity of utilizing sentiment knowledge for pre-training models, and provides a unified sentiment representation for a wide range of sentiment analysis tasks.
  • The authors hope to apply SKEP on more sentiment analysis tasks, to further see the generalization of SKEP, and the authors are interested in exploiting more types of sentiment knowledge and more fine-grained sentiment mining methods
Summary
  • Introduction:

    Sentiment analysis refers to the identification of sentiment and opinion contained in the input texts that are often user-generated comments.
  • Pre-training methods (Peters et al, 2018; Radford et al, 2018; Devlin et al, 2019; Yang et al, 2019) have shown their powerfulness in learning general semantic representations, and have remarkably improved most natural language processing (NLP) tasks like sentiment analysis
  • These methods build unsupervised objectives at word-level, such as masking strategy (Devlin et al, 2019), next-word prediction (Radford et al, 2018) or permutation (Yang et al, 2019).
  • In fine-tuning step, the final hidden state of [CLS] is often used as the overall semantic representation of the input sequence
  • Results:

    The authors compare the SKEP method with the strong pretraining baseline RoBERTa and previous SOTA.
  • Comparing with RoBERTa, SKEP significantly and consistently improves the performance on both From SST-2 Sem-L Model RoBERTa. Sentence Samples altogether , this is :su:c:c:e:s:s:f:u:l as a film , while at the same time being a most touching reconsideration of the familiar :m:a:s:t:e:rp:i:e:c:e .
  • Altogether , this is :su:c:c:e:s:s:f:u:l as a film , while at the same time being a most touching reconsideration of the familiar :m:a:s:t:e:rp:i:e:c:e.
  • The author got this at an :am::a:z:i:n:g price from Amazon and it arrived just in time .
  • Conclusion:

    The authors propose Sentiment Knowledge Enhanced Pre-training for sentiment analysis.
  • Sentiment masking and three sentiment pre-training objectives are designed to incorporate various types of knowledge for pre-training model.
  • The authors' work verifies the necessity of utilizing sentiment knowledge for pre-training models, and provides a unified sentiment representation for a wide range of sentiment analysis tasks.
  • The authors hope to apply SKEP on more sentiment analysis tasks, to further see the generalization of SKEP, and the authors are interested in exploiting more types of sentiment knowledge and more fine-grained sentiment mining methods
Tables
  • Table1: Numbers of samples for each dataset. Sem-R and Sem-L refer to restaurant and laptop parts of SemEval 2014 Task 4
  • Table2: Hyper-parameters for fine-tuning on each dataset. Batch and Epoch indicate batch size and maximum epoch respectively
  • Table3: Comparison with RoBERTa and previous SOTA. For MPQA, here reports both binary-F1 and prop-F1 as (Marasovicand Frank, 2018), which are split by a slash. The scores of previous SOTA come from: 1(<a class="ref-link" id="cRaffel_et+al_2019_a" href="#rRaffel_et+al_2019_a">Raffel et al, 2019</a>; <a class="ref-link" id="cLan_et+al_2019_a" href="#rLan_et+al_2019_a">Lan et al, 2019</a>); 2(<a class="ref-link" id="cXie_et+al_2019_a" href="#rXie_et+al_2019_a">Xie et al, 2019</a>); 3(<a class="ref-link" id="cZhao_et+al_2019_a" href="#rZhao_et+al_2019_a">Zhao et al, 2019</a>); 4(<a class="ref-link" id="cRietzler_et+al_2019_a" href="#rRietzler_et+al_2019_a">Rietzler et al, 2019</a>); 5(Marasovicand Frank,
  • Table4: Effectiveness of objectives. SW, WP, AP refers to pre-training objectives: Sentiment Word prediction, Word Polarity prediction and Aspect-sentiment Pair prediction. “Random Token” denotes random token masking used in RoBERTa. AS-I denotes predicting words in an aspect-sentiment pair independently
  • Table5: Visualization of chosen samples. Words above wavy underline are mean sentiment words, and words above double underlines mean aspects. Color depth denotes importance for classification. The deeper color means more importance. The color depth is calculated by the attention weights with the classification token [CLS]
  • Table6: Comparison of vector used for aspectsentiment pair prediction. Sent-Vector uses sentence representation (output vector of [CLS]) for prediction, while pair-vector uses the concatenation of output vectors of the two words in a pair
  • Table7: Sentiment seed words used in our experiment
Download tables as Excel
Related work
  • Sentiment Analysis with Knowledge Various types of sentiment knowledge, including sentiment words, word polarity, aspect-sentiment pairs, have been proved to be useful for a wide range of sentiment analysis tasks.

    Sentiment words with their polarity are widely used for sentiment analysis, including sentencelevel sentiment classification (Taboada et al, 2011; Shin et al, 2017; Lei et al, 2018; Barnes et al, 2019), aspect-level sentiment classification (Vo and Zhang, 2015), opinion extraction (Li and Lam, 2017), emotion analysis (Gui et al, 2017; Fan et al, 2019) and so on. Lexicon-based method (Turney, 2002; Taboada et al, 2011) directly utilizes polarity of sentiment words for classification. Traditional feature-based approaches encode sentiment word information in manually-designed features to improve the supervised models (Pang et al, 2008; Agarwal et al, 2011). In contrast, deep learning approaches enhance the embedding representation with the help of sentiment words (Shin et al, 2017), or absorb the sentiment knowledge through linguistic regularization (Qian et al, 2017; Fan et al, 2019).

    Aspect-sentiment pair knowledge is also useful for aspect-level classification and opinion extraction. Previous works often provide weak supervision by this type of knowledge, either for aspectlevel classification (Zeng et al, 2019) or for opinion extraction (Yang et al, 2017; Ding et al, 2017).
Funding
  • This work was supported by the National Key Research and Development Project of China (No 2018AAA0101900)
Reference
  • Apoorv Agarwal, Boyi Xie, Ilia Vovsha, Owen Rambow, and Rebecca Passonneau. 201Sentiment analysis of twitter data. In Proceedings of the Workshop on Language in Social Media (LSM 2011).
    Google ScholarLocate open access versionFindings
  • Jeremy Barnes, Samia Touileb, Lilja Øvrelid, and Erik Velldal. 2019. Lexicon information in neural sentiment analysis: a multi-task learning approach. In Proceedings of the 22nd Nordic Conference on Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL 2019.
    Google ScholarLocate open access versionFindings
  • Ying Ding, Jianfei Yu, and Jing Jiang. 2017. Recurrent neural networks with auxiliary labels for crossdomain opinion target extraction. In AAAI 2017.
    Google ScholarLocate open access versionFindings
  • Chuang Fan, Hongyu Yan, Jiachen Du, Lin Gui, Lidong Bing, Min Yang, Ruifeng Xu, and Ruibin Mao. 2019. A knowledge regularized hierarchical approach for emotion cause analysis. In EMNLP 2019.
    Google ScholarLocate open access versionFindings
  • Lin Gui, Jiannan Hu, Yulan He, Ruifeng Xu, Qin Lu, and Jiachen Du. 2017. A question answering approach for emotion cause extraction. In EMNLP 2017.
    Google ScholarLocate open access versionFindings
  • Luyao Huang, Chi Sun, Xipeng Qiu, and Xuanjing Huang. 2019. GlossBERT: BERT for word sense disambiguation with gloss knowledge. In EMNLP 2019.
    Google ScholarLocate open access versionFindings
  • Ganesh Jawahar, Benoıt Sagot, and Djame Seddah. 2019. What does BERT learn about the structure of language? In ACL 2019.
    Google ScholarLocate open access versionFindings
  • Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 201SpanBERT: Improving pre-training by representing and predicting spans. arXiv preprint arXiv:1907.10529.
    Findings
  • Zhen-Zhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for selfsupervised learning of language representations. ArXiv, abs/1909.11942.
    Findings
  • Zeyang Lei, Yujiu Yang, Min Yang, and Yi Liu. 2018. A multi-sentiment-resource enhanced attention network for sentiment classification. In ACL 2018.
    Google ScholarLocate open access versionFindings
  • Yoav Levine, Barak Lenz, Or Dagan, Dan Padnos, Or Sharir, Shai Shalev-Shwartz, Amnon Shashua, and Yoav Shoham. 2019. Sensebert: Driving some sense into bert.
    Google ScholarFindings
  • Xin Li and Wai Lam. 2017. Deep multi-task learning for aspect term extraction with memory interaction. In EMNLP 2017.
    Google ScholarLocate open access versionFindings
  • Bing Liu. 2012. Sentiment analysis and opinion mining. In Synthesis Lectures on Human Language Technologies 5.1 (2012): 1-167.
    Google ScholarFindings
  • Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
    Findings
  • Ana Marasovicand Anette Frank. 2018. SRL4ORL: Improving opinion role labeling using multi-task learning with semantic role labeling. In NAACL 2018.
    Google ScholarLocate open access versionFindings
  • Bo Pang, Lillian Lee, et al. 2008. Opinion mining and sentiment analysis. Foundations and Trends R in Information Retrieval, 2(1–2):1–135.
    Google ScholarLocate open access versionFindings
  • Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 20Deep contextualized word representations. arXiv preprint arXiv:1802.05365.
    Findings
  • Maria Pontiki, Dimitris Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014. SemEval-2014 task 4: Aspect based sentiment analysis. In SemEval 2014.
    Google ScholarLocate open access versionFindings
  • Qiao Qian, Minlie Huang, Jinhao Lei, and Xiaoyan Zhu. 2017. Linguistically regularized LSTM for sentiment classification. In ACL 2017.
    Google ScholarLocate open access versionFindings
  • Alec Radford, Karthik Narasimhan, Time Salimans, and Ilya Sutskever. 2018. Improving language understanding with unsupervised learning. Technical report, Technical report, OpenAI.
    Google ScholarFindings
  • Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer.
    Google ScholarFindings
  • Alexander Rietzler, Sebastian Stabinger, Paul Opitz, and Stefan Engl. 2019. Adapt or get left behind: Domain adaptation through bert language model finetuning for aspect-target sentiment classification. ArXiv, abs/1908.11860.
    Findings
  • Bonggun Shin, Timothy Lee, and Jinho D. Choi. 2017. Lexicon integrated CNN models with attention for sentiment analysis. In Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis.
    Google ScholarLocate open access versionFindings
  • Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP 2013.
    Google ScholarLocate open access versionFindings
  • Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019. Ernie: Enhanced representation through knowledge integration. arXiv preprint arXiv:1904.09223.
    Findings
  • Maite Taboada, Julian Brooke, Milan Tofiloski, Kimberly Voll, and Manfred Stede. 2011. Lexicon-based methods for sentiment analysis. Computational linguistics, 37(2):267–307.
    Google ScholarLocate open access versionFindings
  • Duyu Tang, Furu Wei, Nan Yang, Ming Zhou, Ting Liu, and Bing Qin. 2014. Learning sentiment-specific word embedding for twitter sentiment classification. In ACL 2014.
    Google ScholarLocate open access versionFindings
  • Peter D Turney. 2002. Thumbs up or thumbs down?: semantic orientation applied to unsupervised classification of reviews. In ACL 2002.
    Google ScholarLocate open access versionFindings
  • Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS 2017.
    Google ScholarFindings
  • Duy-Tin Vo and Yue Zhang. 2015. Target-dependent twitter sentiment classification with rich automatic features. In IJCAI 2015.
    Google ScholarLocate open access versionFindings
  • Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP.
    Google ScholarLocate open access versionFindings
  • Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. Language Resources and Evaluation.
    Google ScholarFindings
  • Theresa Ann Wilson. 2008. Fine-grained subjectivity and sentiment analysis: Recognizing the intensity, polarity, and attitudes of private states.
    Google ScholarFindings
  • Qizhe Xie, Zihang Dai, Eduard H. Hovy, Minh-Thang Luong, and Quoc V. Le. 2019. Unsupervised data augmentation. CoRR, abs/1904.12848.
    Findings
  • Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding.
    Google ScholarFindings
  • Zhilin Yang, Ruslan Salakhutdinov, and William W. Cohen. 2017. Transfer learning for sequence tagging with hierarchical recurrent networks. ArXiv, abs/1703.06345.
    Findings
  • Ziqian Zeng, Wenxuan Zhou, Xin Liu, and Yangqiu Song. 2019. A variational approach to weakly supervised document-level multi-aspect sentiment classification. In NAACL 2019.
    Google ScholarLocate open access versionFindings
  • Lei Zhang, Shuai Wang, and Bing Liu. 2018. Deep learning for sentiment analysis: A survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery.
    Google ScholarFindings
  • Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In NIPS 2015.
    Google ScholarLocate open access versionFindings
  • Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: Enhanced language representation with informative entities. In ACL 2019.
    Google ScholarLocate open access versionFindings
  • Pinlong Zhao, Linlin Hou, and Ou Wu. 2019. Modeling sentiment dependencies with graph convolutional networks for aspect-level sentiment classification. CoRR, abs/1906.04501.
    Findings
  • For sentiment knowledge mining, we construct 46 sentiment seed words as follows. We first count the 9,750 items of Qian et al., (2017) on training data of Amazon-2, and get 50 most frequent sentiment words. Then, we manually filter out inappropriate words from these 50 words in a few minutes and finally get 46 sentiment words with polarities (Table 7). The filtered words are need, fun, plot and fine respectively, which are all negative words.
    Google ScholarLocate open access versionFindings
Full Text
Your rating :
0

 

Tags
Comments