Coupled Hierarchical Transformer for Stance Aware Rumor Verification in Social Media Conversations

EMNLP 2020, 2020.

Cited by: 0|Bibtex|Views34
Weibo:
We further propose a Coupled Transformer Module to capture the inter-task interactions and a Post-Level Attention layer to use the predicted stance labels for rumor verification, respectively

Abstract:

The prevalent use of social media enables rapid spread of rumors on a massive scale, which leads to the emerging need of automatic rumor verification (RV). A number of previous studies focus on leveraging stance classification to enhance RV with multi-task learning (MTL) methods. However, most of these methods failed to employ pre-trained...More

Code:

Data:

0
Introduction
  • Recent years have witnessed a profound revolution in social media, as many individuals gradually turn to different social platforms to share the latest news and voice personal opinions.
  • The flourish of social media enables rapid dissemination of unverified information on a massive scale, which may cause serious harm to the society (e.g., impacting presidential election decisions (Allcott and Gentzkow, 2017)).
  • Veracity Label: False Rumor Source Post.
  • Stance Label Support R1: Reply Post.
  • He died several days ago.
Highlights
  • Recent years have witnessed a profound revolution in social media, as many individuals gradually turn to different social platforms to share the latest news and voice personal opinions
  • To address the above two shortcomings, we explore the potential of BERT for stance-aware rumor verification, and propose a new multi-task learning model based on Transformer (Vaswani et al, 2017), named Coupled Hierarchical Transformer
  • Stance Classification (SC): We first consider the following competitive approaches that focus on stance classification (SC) only: (1) SVM is a baseline method that feeds conversation-based and affective-based features to linear SVM (Pamungkas et al, 2018); (2) BranchLSTM is an LSTM-based architecture designed by Kochkina et al (2018), which focuses on modeling the sequential branches in each thread; (3) Temporal ATT is an attention-based model proposed by Veyseh et al (2017), which treats each post’s adjacent posts in a conversation timeline as its local context, followed by employing attention mechanism over the local context to learn the importance of each adjacent post; (4) Conversational GCN is the state-of-the-art approach recently proposed by Wei et al (2019), which leverages graph convolutional network to model the relations between posts in each thread
  • It is clear to observe that our Hierarchical Transformer model performs much better than all the compared systems on Macro-F1
  • This is crucial for veracity prediction, because the support and deny stances usually provide important clues to identify the true and false rumors respectively. All these observations demonstrate the general effectiveness of our Hierarchical Transformer model
  • We evaluate the effectiveness of our Coupled Hierarchical Transformer model, and consider several multi-task learning frameworks for stance-aware rumor verification: (1) BranchLSTM+NileTMRG is a pipeline approach, which first trains a BranchLSTM model for SC, followed by a SVM classifier for RV (Kochkina et al, 2018); (2) MTL2 is the MTL framework proposed in (Kochkina et al, 2018), which shares a single LSTM channel but uses two separate output layers for SC and RV, respectively; (3) Hierarchical PSV is a hierarchical model proposed by (Wei et al, 2019), which first learns content and stance features via Conversational-GCN, followed by exploiting temporal evolution for RV via Stance-Aware RNN; (4) MTL2-Hierarchical Transformer is our adapted MTL2 model which is introduced in Section 3.3
Methods
Results
  • 4.2.1 Evaluation on Single-Task Models

    In this subsection, the authors compare the proposed Hierarchical Transformer with existing single-task models for SC and RV, respectively.
  • Compared with previous approaches, the model shows its strong capability of detecting posts belonging to the support and deny stances
  • This is crucial for veracity prediction, because the support and deny stances usually provide important clues to identify the true and false rumors respectively.
  • All these observations demonstrate the general effectiveness of the Hierarchical Transformer model
Conclusion
  • The authors first examined the limitations of existing approaches to stance classification (SC) and rumor verification (RV).
  • To tackle these limitations, the authors first proposed a single-task model (i.e., Hierarchical Transformer) for SC and RV, followed by designing a multi-task learning framework with a Coupled Transformer module to capture intertask interactions and a Post-Level Attention Layer to use stance distributions for the RV task.
Summary
  • Introduction:

    Recent years have witnessed a profound revolution in social media, as many individuals gradually turn to different social platforms to share the latest news and voice personal opinions.
  • The flourish of social media enables rapid dissemination of unverified information on a massive scale, which may cause serious harm to the society (e.g., impacting presidential election decisions (Allcott and Gentzkow, 2017)).
  • Veracity Label: False Rumor Source Post.
  • Stance Label Support R1: Reply Post.
  • He died several days ago.
  • Methods:

    SVM (Pamungkas et al, 2018) BranchLSTM (Kochkina et al, 2018) Temporal ATT (Veyseh et al, 2017) Conversational-GCN (Wei et al, 2019)

    Hierarchical Transformer (Ours)

    Single Stance Type Evaluation

    Support-F1 Deny-F1 Query-F1 Comment-F1

    Overall Evaluation

    Macro-F1 Accuracy

    Setting Single-Task

    BranchLSTM (Kochkina et al, 2018) TD-RvNN (Ma et al, 2018b) Hierarchical GCN-RNN (Wei et al, 2019) HiTPLAN (Khoo et al, 2020)

    SemEval-2017 Dataset PHEME Dataset Multi-Task.
  • Results:

    4.2.1 Evaluation on Single-Task Models

    In this subsection, the authors compare the proposed Hierarchical Transformer with existing single-task models for SC and RV, respectively.
  • Compared with previous approaches, the model shows its strong capability of detecting posts belonging to the support and deny stances
  • This is crucial for veracity prediction, because the support and deny stances usually provide important clues to identify the true and false rumors respectively.
  • All these observations demonstrate the general effectiveness of the Hierarchical Transformer model
  • Conclusion:

    The authors first examined the limitations of existing approaches to stance classification (SC) and rumor verification (RV).
  • To tackle these limitations, the authors first proposed a single-task model (i.e., Hierarchical Transformer) for SC and RV, followed by designing a multi-task learning framework with a Coupled Transformer module to capture intertask interactions and a Post-Level Attention Layer to use stance distributions for the RV task.
Tables
  • Table1: Basic statistics of the SemEval-2017 dataset and the PHEME dataset
  • Table2: Results of stance classification on the SemEval-2017 dataset
  • Table3: Results of rumor veracity prediction. Single-Task indicates that stance labels are not used during the training stage. † indicates that our Coupled Hieararchical Transformer model is significantly better than the best compared system with p-value < 0.05 based on McNemar’s significance test
  • Table4: Ablation study on the PHEME dataset
Download tables as Excel
Related work
  • Stance Classification: Although stance classification has been well studied in different contexts such as online forums (Hasan and Ng, 2013; Lukasik et al, 2016; Ferreira and Vlachos, 2016; Mohammad et al, 2016), a recent trend is to study stance classification towards rumors in different social media platforms (Mendoza et al, 2010; Qazvinian et al, 2011). These studies can be roughly categorized into two groups. One line of work aims to design different features to capture the sequential property of conversation threads (Zubiaga et al, 2016; Aker et al, 2017; Pamungkas et al, 2018; Zubiaga et al, 2018b; Giasemidis et al, 2018). Another line of work attempts to apply recent deep learning models to automatically capture effective stance features (Kochkina et al, 2017; Veyseh et al, 2017). Our work extends the latter line of work by proposing a hierarchical Transformer based on the recent pre-trained BERT for this task. Moreover, we notice that our BERT-based hierarchical Transformer is similar to the model proposed in (Pappagari et al, 2019), but we want to point out that our model design in the input and output layers is specific to stance classification, which is different from their work. Rumor Verification: Due to the negative impact of various rumors spreading on social media, rumor verification has attracted increasing attention in recent years. Existing approaches to single-task rumor verification generally belong to two groups. The first line of work focuses on either employing a myriad of hand-crafted features (Qazvinian et al, 2011; Yang et al, 2012; Kwon et al, 2013; Ma et al, 2015) including post contents, user profiles, information credibility features (Castillo et al, 2011), and propagation patterns, or resorting to various kinds of kernels to model the event propagation structure (Wu et al, 2015; Ma et al, 2017). The second line of work applies variants of several neural network models to automatically capture important features among all the propagated posts (Ma et al, 2016; Ruchansky et al, 2017; Chen et al, 2018). Different from these studies, the goal in this paper is to leverage stance classification to improve rumor verification with a multi-task learning architecture. Stance-Aware Rumor Verification: The recent advance in rumor verification is to exploit stance information to enhance rumor verification with different multi-task learning approaches. Specifically, Ma et al (2018a) and Kochkina et al (2018) respectively proposed two multi-task learning architectures to jointly optimize stance classification and rumor verification based on two different variants of RNN, i.e., GRU and LSTM. More recently, Kumar and Carley (2019) proposed another multi-task LSTM model based on tree structures for stanceaware rumor verification. Our work bears the same intuition to these previous studies, and aims to explore the potential of the pre-trained BERT to this multi-task learning task.
Funding
  • This research is supported by DSO grant DSOCL18009, the Natural Science Foundation of China (No 61672288, 62076133, and 62006117), and the Natural Science Foundation of Jiangsu Province for Young Scholars (SBK2020040749) and Distinguished Young Scholars (SBK2020010154)
Study subjects and analysis
posts with the top-5 attention weights in the thread: 5
Case Study: To provide deeper insights into our Coupled Hierarchical Transformer, we carefully choose one representative sample from our test set, and show the stance and veracity prediction results as well as the attention weights of each post learnt in the post-level attention layer. Due to space limitation, we only show five posts with the top-5 attention weights in the thread. Predicted Veracity Label: False Rumor Source Post

Reference
  • Ahmet Aker, Leon Derczynski, and Kalina Bontcheva. 2017. Simple open stance classification for rumour analysis. In Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017, pages 31–39.
    Google ScholarLocate open access versionFindings
  • Elena Kochkina and Maria Liakata. 2020. Estimating predictive uncertainty for rumour verification models. In Proceedings of ACL.
    Google ScholarLocate open access versionFindings
  • Elena Kochkina, Maria Liakata, and Isabelle Augenstein. 2017. Turing at semeval-2017 task 8: Sequential approach to rumour stance classification with branch-lstm. In Proceedings of SemEval.
    Google ScholarLocate open access versionFindings
  • Elena Kochkina, Maria Liakata, and Arkaitz Zubiaga. 2018. All-in-one: Multi-task learning for rumour verification. In Proceedings of COLING.
    Google ScholarLocate open access versionFindings
  • Sumeet Kumar and Kathleen Carley. 2019. Tree LSTMs with convolution units to predict stance and rumor veracity in social media conversations. In Proceedings of ACL.
    Google ScholarLocate open access versionFindings
  • Sejeong Kwon, Meeyoung Cha, Kyomin Jung, Wei Chen, and Yajun Wang. 2013. Prominent features of rumor propagation in online social media. In Proceedings of ICDM.
    Google ScholarLocate open access versionFindings
  • Quanzhi Li, Qiong Zhang, and Luo Si. 2019. Rumor detection by exploiting user credibility information, attention and multi-task learning. In Proceedings of ACL.
    Google ScholarLocate open access versionFindings
  • Xiaomo Liu, Armineh Nourbakhsh, Quanzhi Li, Rui Fang, and Sameena Shah. 2015. Real-time rumor debunking on twitter. In Proceedings of CIKM.
    Google ScholarLocate open access versionFindings
  • Natali Ruchansky, Sungyong Seo, and Yan Liu. 2017. Csi: A hybrid deep model for fake news detection. In Proceedings of CIKM.
    Google ScholarLocate open access versionFindings
  • Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of NIPS.
    Google ScholarLocate open access versionFindings
  • Michal Lukasik, PK Srijith, Duy Vu, Kalina Bontcheva, Arkaitz Zubiaga, and Trevor Cohn. 2016. Hawkes processes for continuous time sequence classification: an application to rumour stance classification in twitter. In Proceedings of ACL.
    Google ScholarLocate open access versionFindings
  • Jing Ma, Wei Gao, Prasenjit Mitra, Sejeong Kwon, Bernard J Jansen, Kam-Fai Wong, and Meeyoung Cha. 2016. Detecting rumors from microblogs with recurrent neural networks. In Proceedings of IJCAI.
    Google ScholarLocate open access versionFindings
  • Jing Ma, Wei Gao, Zhongyu Wei, Yueming Lu, and Kam-Fai Wong. 2015. Detect rumors using time series of social context information on microblogging websites. In Proceedings of CIKM.
    Google ScholarLocate open access versionFindings
  • Jing Ma, Wei Gao, and Kam-Fai Wong. 2017. Detect rumors in microblog posts using propagation structure via kernel learning. In Proceedings of ACL.
    Google ScholarLocate open access versionFindings
  • Jing Ma, Wei Gao, and Kam-Fai Wong. 2018a. Detect rumor and stance jointly by neural multi-task learning. In Companion Proceedings of the The Web Conference 2018.
    Google ScholarLocate open access versionFindings
  • Jing Ma, Wei Gao, and Kam-Fai Wong. 2018b. Rumor detection on twitter with tree-structured recursive neural networks. In Proceedings of ACL.
    Google ScholarLocate open access versionFindings
  • Marcelo Mendoza, Barbara Poblete, and Carlos Castillo. 2010. Twitter under crisis: can we trust what we rt? In Proceedings of the 3rd Workshop on Social Network Mining and Analysis, SNAKDD 2009, Paris, France, June 28, 2009.
    Google ScholarLocate open access versionFindings
  • Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016. Semeval-2016 task 6: Detecting stance in tweets. In Proceedings of SemEval.
    Google ScholarLocate open access versionFindings
  • EW Pamungkas, V Basile, and V Patti. 2018. Stance classification for rumour analysis in twitter: Exploiting affective information and conversation structure. In 2nd International Workshop on Rumours and Deception in Social Media (RDSM 2018), volume 2482, pages 1–7.
    Google ScholarLocate open access versionFindings
  • Amir Pouran Ben Veyseh, Javid Ebrahimi, Dejing Dou, and Daniel Lowd. 2017. A temporal attentional model for rumor stance classification. In Proceedings of CIKM.
    Google ScholarLocate open access versionFindings
  • Penghui Wei, Nan Xu, and Wenji Mao. 2019. Modeling conversation structure and temporal dynamics for jointly predicting rumor stance and veracity. In Proceedings of EMNLP.
    Google ScholarLocate open access versionFindings
  • Ke Wu, Song Yang, and Kenny Q. Zhu. 2015. False rumors detection on sina weibo by propagation structures. In Proceedings of ICDE.
    Google ScholarLocate open access versionFindings
  • Fan Yang, Yang Liu, Xiaohui Yu, and Min Yang. 2012. Automatic detection of rumor on sina weibo. In Proceedings of the ACM SIGKDD Workshop on Mining Data Semantics, MDS ’12, pages 13:1–13:7.
    Google ScholarLocate open access versionFindings
  • Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of NAACL.
    Google ScholarLocate open access versionFindings
  • Arkaitz Zubiaga, Ahmet Aker, Kalina Bontcheva, Maria Liakata, and Rob Procter. 2018a. Detection and resolution of rumours in social media: A survey. ACM Computing Surveys (CSUR), 51(2):32.
    Google ScholarLocate open access versionFindings
  • Arkaitz Zubiaga, Elena Kochkina, Maria Liakata, Rob Procter, Michal Lukasik, Kalina Bontcheva, Trevor Cohn, and Isabelle Augenstein. 2018b. Discourseaware rumour stance classification in social media using sequential classifiers. Information Processing & Management, 54(2):273–290.
    Google ScholarLocate open access versionFindings
  • Arkaitz Zubiaga, Maria Liakata, Rob Procter, Geraldine Wong Sak Hoi, and Peter Tolmie. 2016. Analysing how people orient to and spread rumours in social media by looking at conversational threads. PloS one, 11(3):e0150989.
    Google ScholarLocate open access versionFindings
  • Raghavendra Pappagari, Piotr Zelasko, Jesus Villalba, Yishay Carmiel, and Najim Dehak. 2019. Hierarchical transformers for long document classification. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU).
    Google ScholarLocate open access versionFindings
  • Vahed Qazvinian, Emily Rosengren, Dragomir R Radev, and Qiaozhu Mei. 2011. Rumor has it: Identifying misinformation in microblogs. In Proceedings of EMNLP.
    Google ScholarLocate open access versionFindings
Full Text
Your rating :
0

 

Tags
Comments