Multi-hop Reading Comprehension across Multiple Documents by Reasoning over Heterogeneous Graphs

ACL (1), pp. 2704-2713, 2019.

Cited by: 18|Bibtex|Views106|DOI:https://doi.org/10.18653/v1/p19-1260
EI
Other Links: dblp.uni-trier.de|arxiv.org
Weibo:
We propose a new Graph Neural Networks-based method for multihop reading comprehension across multiple documents

Abstract:

Multi-hop reading comprehension (RC) across documents poses new challenge over single-document RC because it requires reasoning over multiple documents to reach the final answer. In this paper, we propose a new model to tackle the multi-hop RC problem. We introduce a heterogeneous graph with different types of nodes and edges, which is ...More

Code:

Data:

0
Introduction
  • Being able to comprehend a document and output correct answer given a query/question about content in the document, often referred as machine reading comprehension (RC) or question answering (QA), is an important and challenging task in natural language processing (NLP).
  • Query: record label get ready Support doc 1: Mason Durell Betha, better known by stage name Mase, is an American hip hop recording artist and minister.
  • He is best known for being signed to Sean “Diddy” Combs’s label Bad Boy Records.
  • Support doc 3: Bad Boy Entertainment is an American record label founded in 1993 by Sean Combs.
Highlights
  • Being able to comprehend a document and output correct answer given a query/question about content in the document, often referred as machine reading comprehension (RC) or question answering (QA), is an important and challenging task in natural language processing (NLP)
  • Inspired by the success of Graph Neural Networks based methods (Song et al, 2018; De Cao et al, 2018) for multi-hop reading comprehension, we introduce a new type of graph, called Heterogeneous Document-Entity (HDE) graph
  • We show the effectiveness of our proposed Heterogeneous Document-Entity graph for multihop multi-document reading comprehension task
  • We propose a new Graph Neural Networks-based method for multihop reading comprehension across multiple documents
  • We introduce the Heterogeneous Document-Entity graph, a heterogeneous graph for multiple-hop reasoning over nodes representing different granularity levels of information
  • Evaluated on WIKIHOP, our end-to-end trained single neural model delivers competitive results while our ensemble model achieves the state-of-the-art performance
Methods
  • The authors describe different modules of the proposed Heterogeneous Document-Entity (HDE) graph-based multi-hop RC model.
  • Given a query q with the form of (s, r, ?) which represents subject, relation and unknown object respectively, a set of support documents Sq and a set of candidates Cq, the task is to predict the correct answer a∗ to the query.
  • To encode information including in the text of query, candidates and support documents, the authors use a pretrained embedding matrix (Pennington et al, 2014) to convert word sequences to sequences of vectors.
Results
  • In Table 1, the authors show the results of the the proposed HDE graph based model on both development and test set and compare it with previously published results.
  • The authors show that the proposed HDE graph based model improves the published state-of-the-art accuracy on development set from 67.1% (Kundu et al, 2018) to 68.1%, on the blind test set from 70.6% (Zhong et al, 2019) to 70.9%.
  • Even though the single model is a little worse than the “DynSAN”, the ensemble model is better than both the ensembled “DynSAN” and the ensembled “Entity-GCN”
Conclusion
  • The authors propose a new GNN-based method for multihop RC across multiple documents.
  • The authors introduce the HDE graph, a heterogeneous graph for multiple-hop reasoning over nodes representing different granularity levels of information.
  • The authors use co-attention and self-attention to encode candidates, documents, entities of mentions of candidates and query subjects into query-aware representations, which are employed to initialize graph node representations.
  • The authors would like to investigate explainable GNN for this task, such as explicit reasoning path in (Kundu et al, 2018), and work on other data sets such as HotpotQA
Summary
  • Introduction:

    Being able to comprehend a document and output correct answer given a query/question about content in the document, often referred as machine reading comprehension (RC) or question answering (QA), is an important and challenging task in natural language processing (NLP).
  • Query: record label get ready Support doc 1: Mason Durell Betha, better known by stage name Mase, is an American hip hop recording artist and minister.
  • He is best known for being signed to Sean “Diddy” Combs’s label Bad Boy Records.
  • Support doc 3: Bad Boy Entertainment is an American record label founded in 1993 by Sean Combs.
  • Methods:

    The authors describe different modules of the proposed Heterogeneous Document-Entity (HDE) graph-based multi-hop RC model.
  • Given a query q with the form of (s, r, ?) which represents subject, relation and unknown object respectively, a set of support documents Sq and a set of candidates Cq, the task is to predict the correct answer a∗ to the query.
  • To encode information including in the text of query, candidates and support documents, the authors use a pretrained embedding matrix (Pennington et al, 2014) to convert word sequences to sequences of vectors.
  • Results:

    In Table 1, the authors show the results of the the proposed HDE graph based model on both development and test set and compare it with previously published results.
  • The authors show that the proposed HDE graph based model improves the published state-of-the-art accuracy on development set from 67.1% (Kundu et al, 2018) to 68.1%, on the blind test set from 70.6% (Zhong et al, 2019) to 70.9%.
  • Even though the single model is a little worse than the “DynSAN”, the ensemble model is better than both the ensembled “DynSAN” and the ensembled “Entity-GCN”
  • Conclusion:

    The authors propose a new GNN-based method for multihop RC across multiple documents.
  • The authors introduce the HDE graph, a heterogeneous graph for multiple-hop reasoning over nodes representing different granularity levels of information.
  • The authors use co-attention and self-attention to encode candidates, documents, entities of mentions of candidates and query subjects into query-aware representations, which are employed to initialize graph node representations.
  • The authors would like to investigate explainable GNN for this task, such as explicit reasoning path in (Kundu et al, 2018), and work on other data sets such as HotpotQA
Tables
  • Table1: Performance comparison among different models on WIKIHOP development and test set. The results of “BiDAF” are presented in the paper by <a class="ref-link" id="cWelbl_et+al_2018_a" href="#rWelbl_et+al_2018_a">Welbl et al (2018</a>). Models annotated with “*” are unpublished but available on WIKIHOP leaderboard. “-” indicates unavailable numbers
  • Table2: Ablation results on the WIKIHOP dev set
  • Table3: Accuracy(%) comparison under different types of samples
Download tables as Excel
Related work
  • The study presented in this paper is directly related to existing research on multi-hop reading comprehension across multiple documents (Dhingra et al, 2018; Song et al, 2018; De Cao et al, 2018; Zhong et al, 2019; Kundu et al, 2018). The method presented in this paper is similar to previous studies using GNN for multi-hop reasoning (Song et al, 2018; De Cao et al, 2018). Our novelty is that we propose to use a heterogeneous graph instead of a graph with single type of nodes to incorporate different granularity levels of information. The co-attention and self-attention based encoding of multi-level information presented in each input is also inspired by the CFC model (Zhong et al, 2019) because they show the effectiveness of attention mechanisms. Our model is very different from the other two studies (Dhingra et al, 2018; Kundu et al, 2018): these two studies both explicitly score the possible reasoning paths with extra NER or coreference resolution systems while our method does not require these modules and we do multi-hop reasoning over graphs. Besides these studies, our work is also related to the following research directions.
Reference
  • Joost Bastings, Ivan Titov, Wilker Aziz, Diego Marcheggiani, and Khalil Simaan. 2017. Graph convolutional encoders for syntax-aware neural machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1957–1967.
    Google ScholarLocate open access versionFindings
  • Kyunghyun Cho, B van Merrienboer, Caglar Gulcehre, F Bougares, H Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoderdecoder for statistical machine translation. In Conference on Empirical Methods in Natural Language Processing (EMNLP 2014).
    Google ScholarLocate open access versionFindings
  • Nicola De Cao, Wilker Aziz, and Ivan Titov. 2018. Question answering by reasoning across documents with graph convolutional networks. arXiv preprint arXiv:1808.09920.
    Findings
  • Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
    Findings
  • Bhuwan Dhingra, Qiao Jin, Zhilin Yang, William Cohen, and Ruslan Salakhutdinov. 2018. Neural models for reasoning over multiple mentions using coreference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), volume 2, pages 42–48.
    Google ScholarLocate open access versionFindings
  • Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. 2017. Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1263–1272. JMLR. org.
    Google ScholarLocate open access versionFindings
  • Will Hamilton, Zhitao Ying, and Jure Leskovec. 201Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems, pages 1024–1034.
    Google ScholarLocate open access versionFindings
  • Kazuma Hashimoto, Yoshimasa Tsuruoka, Richard Socher, et al. 2017. A joint many-task model: Growing a neural network for multiple nlp tasks. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1923– 1933.
    Google ScholarLocate open access versionFindings
  • Daniel Hewlett, Alexandre Lacoste, Llion Jones, Illia Polosukhin, Andrew Fandrianto, Jay Han, Matthew Kelcey, and David Berthelot. 2016. Wikireading: A novel large-scale language understanding task over wikipedia. arXiv preprint arXiv:1608.03542.
    Findings
  • Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking beyond the surface:a challenge set for reading comprehension over multiple sentences. In Proceedings of North American Chapter of the Association for Computational Linguistics (NAACL).
    Google ScholarLocate open access versionFindings
  • Thomas N Kipf and Max Welling. 2016. Semisupervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907.
    Findings
  • Tomas Kocisky, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gaabor Melis, and Edward Grefenstette. 2018. The narrativeqa reading comprehension challenge. Transactions of the Association of Computational Linguistics, 6:317–328.
    Google ScholarLocate open access versionFindings
  • Souvik Kundu, Tushar Khot, and Ashish Sabharwal. 2018. Exploiting explicit paths for multihop reading comprehension. arXiv preprint arXiv:1811.01127.
    Findings
  • Pengfei Liu, Shuaichen Chang, Xuanjing Huang, Jian Tang, and Jackie Chi Kit Cheung. 2018. Contextualized non-local neural networks for sequence learning. arXiv preprint arXiv:1811.08600.
    Findings
  • Edward Loper and Steven Bird. 2002. Nltk: the natural language toolkit. arXiv preprint cs/0205028.
    Google ScholarFindings
  • Diego Marcheggiani, Joost Bastings, and Ivan Titov. 2018. Exploiting semantics in neural machine translation with graph convolutional networks. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational
    Google ScholarLocate open access versionFindings
  • Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789.
    Findings
  • Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch.
    Google ScholarFindings
  • Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543.
    Google ScholarLocate open access versionFindings
  • Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), volume 1, pages 2227–2237.
    Google ScholarLocate open access versionFindings
  • Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you dont know: Unanswerable questions for squad. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 784–789.
    Google ScholarLocate open access versionFindings
  • Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392.
    Google ScholarLocate open access versionFindings
  • Siva Reddy, Danqi Chen, and Christopher D Manning. 2018. Coqa: A conversational question answering challenge. arXiv preprint arXiv:1808.07042.
    Findings
  • Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016. Bidirectional attention flow for machine comprehension. arXiv preprint arXiv:1611.01603.
    Findings
  • Linfeng Song, Zhiguo Wang, Mo Yu, Yue Zhang, Radu Florian, and Daniel Gildea. 2018. Exploring graph-structured passage representation for multihop reading comprehension with graph neural networks. arXiv preprint arXiv:1809.02040.
    Findings
  • Kai Sun, Dian Yu, Dong Yu, and Claire Cardie. 2018. Improving machine reading comprehension with general reading strategies. arXiv preprint arXiv:1810.13441.
    Findings
  • Yi Tay, Anh Tuan Luu, Siu Cheung Hui, and Jian Su. 2018. Densely connected attention propagation for reading comprehension. In Advances in Neural Information Processing Systems, pages 4911–4922.
    Google ScholarLocate open access versionFindings
  • Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017. Graph attention networks. arXiv preprint arXiv:1710.10903.
    Findings
  • Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. Constructing datasets for multi-hop reading comprehension across documents. Transactions of the Association of Computational Linguistics, 6:287–302.
    Google ScholarLocate open access versionFindings
  • Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart van Merrienboer, Armand Joulin, and Tomas Mikolov. 2015. Towards ai-complete question answering: A set of prerequisite toy tasks. arXiv preprint arXiv:1502.05698.
    Findings
  • Chien-Sheng Wu, Richard Socher, and Caiming Xiong. 2019. Global-to-local memory pointer networks for task-oriented dialogue. arXiv preprint arXiv:1901.04713.
    Findings
  • Caiming Xiong, Victor Zhong, and Richard Socher. 2016. Dynamic coattention networks for question answering. arXiv preprint arXiv:1611.01604.
    Findings
  • Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. 2018. How powerful are graph neural networks? arXiv preprint arXiv:1810.00826.
    Findings
  • Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369–2380.
    Google ScholarLocate open access versionFindings
  • Liang Yao, Chengsheng Mao, and Yuan Luo. 2018. Graph convolutional networks for text classification. arXiv preprint arXiv:1809.05679.
    Findings
  • Yuhao Zhang, Peng Qi, and Christopher D Manning. 2018. Graph convolution over pruned dependency trees improves relation extraction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2205–2215.
    Google ScholarLocate open access versionFindings
  • Victor Zhong, Caiming Xiong, Nitish Shirish Keskar, and Richard Socher. 2019. Coarse-grain fine-grain coattention network for multi-evidence question answering. arXiv preprint arXiv:1901.00603.
    Findings
Full Text
Your rating :
0

 

Tags
Comments