IMoJIE: Iterative Memory-Based Joint Open Information Extraction

Kolluru Keshav
Kolluru Keshav
Aggarwal Samarth
Aggarwal Samarth
Rathore Vipul
Rathore Vipul

ACL, pp. 5871-5886, 2020.

Cited by: 0|Bibtex|Views40|Links
EI
Keywords:
Mean Number of OccurrencesMultilayer PerceptronIterative MemOry Joint Open Information Extractionopen domain information extractionf1 ptMore(8+)
Weibo:
We present IMoJIE, an extension to CopyAttention, which produces the extraction conditioned on all previously extracted tuples

Abstract:

While traditional systems for Open Information Extraction were statistical and rule-based, recently neural models have been introduced for the task. Our work builds upon CopyAttention, a sequence generation OpenIE model (Cui et. al., 2018). Our analysis reveals that CopyAttention produces a constant number of extractions per sentence, a...More
0
Introduction
  • Extracting structured information from unstructured text has been a key research area within NLP.
  • The paradigm of Open Information Extraction (OpenIE) (Banko et al, 2007) uses an open vocabulary to convert natural text to semi-structured representations, by extracting a set of tuples.
  • Traditional OpenIE systems are statistical or rule-based.
  • They are largely unsupervised in nature, or bootstrapped from extractions made by earlier systems.
  • They often consist of several components like POS tagging, and syntactic parsing.
  • To bypass error accumulation in such pipelines, end-to-end neural systems have been proposed recently
Highlights
  • Extracting structured information from unstructured text has been a key research area within NLP
  • Iterative MemOry Joint Open Information Extraction produce a variable number of diverse extractions for a sentence, We present an unsupervised aggregation scheme to bootstrap training data by combining extractions from multiple Open Information Extraction systems. Iterative MemOry Joint Open Information Extraction trained on this data establishes a new
  • Using CaRB evaluation, we find that, contrary to previous papers, neural Open Information Extraction systems are not necessarily better than prior non-neural systems (Table 3)
  • We propose Iterative MemOry Joint Open Information Extraction for the task of Open Information Extraction
  • Iterative MemOry Joint Open Information Extraction significantly improves upon the existing Open Information Extraction systems in all three metrics, Optimal F1, AUC, and Last F1, establishing a new State Of the Art system
  • Unlike existing neural Open Information Extraction systems, Iterative MemOry Joint Open Information Extraction produces non-redundant as well as a variable number of Open Information Extraction tuples depending on the sentence, by iteratively generating them conditioned on the previous tuples
Methods
  • Methods in Natural Language

    Processing (EMNLP), Austin, Texas. Association for Computational Linguistics.

    Gabriel Stanovsky, Jessica Ficler, Ido Dagan, and Yoav Goldberg. 2016.
  • Processing (EMNLP), Austin, Texas.
  • Association for Computational Linguistics.
  • Gabriel Stanovsky, Jessica Ficler, Ido Dagan, and Yoav Goldberg.
  • Getting more out of syntax with PropS.
  • CoRR, abs/1603.01648.
  • Gabriel Stanovsky, Mausam, and Ido Dagan.
  • OpenIE as an intermediate structure for semantic tasks.
  • In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 303–308
Results
  • Results and Analysis

    6.1 Performance of Existing Systems

    How well do the neural systems perform as compared to the rule-based systems?

    Using CaRB evaluation, the authors find that, contrary to previous papers, neural OpenIE systems are not necessarily better than prior non-neural systems (Table 3).
  • RnnOIE performs much better, suffers due to its lack of generating auxilliary verbs and implied prepositions.
  • Example, it can only generate (Trump; President; US) instead of (Trump; is President of; US) from the sentence Filtering.
  • None Extraction-based Sentence-based Score-And-Filter “US President Trump...”.
  • It is trained only on limited number of pseudo-gold extractions, generated by Michael et al (2018), which does not take advantage of boostrapping techniques
Conclusion
  • The authors propose IMOJIE for the task of OpenIE. IMOJIE significantly improves upon the existing OpenIE systems in all three metrics, Optimal F1, AUC, and Last F1, establishing a new State Of the Art system.
  • See et al (2017) showed that using a coverage loss to track the attention over the decoded words improves the quality of the generated output
  • The authors add to this narrative by showing that deep inter-attention between the input and the partially-decoded words creates a better representation for iterative generation of triples.
  • This general observation may be of independent interest beyond OpenIE, such as in text summarization
Summary
  • Introduction:

    Extracting structured information from unstructured text has been a key research area within NLP.
  • The paradigm of Open Information Extraction (OpenIE) (Banko et al, 2007) uses an open vocabulary to convert natural text to semi-structured representations, by extracting a set of tuples.
  • Traditional OpenIE systems are statistical or rule-based.
  • They are largely unsupervised in nature, or bootstrapped from extractions made by earlier systems.
  • They often consist of several components like POS tagging, and syntactic parsing.
  • To bypass error accumulation in such pipelines, end-to-end neural systems have been proposed recently
  • Methods:

    Methods in Natural Language

    Processing (EMNLP), Austin, Texas. Association for Computational Linguistics.

    Gabriel Stanovsky, Jessica Ficler, Ido Dagan, and Yoav Goldberg. 2016.
  • Processing (EMNLP), Austin, Texas.
  • Association for Computational Linguistics.
  • Gabriel Stanovsky, Jessica Ficler, Ido Dagan, and Yoav Goldberg.
  • Getting more out of syntax with PropS.
  • CoRR, abs/1603.01648.
  • Gabriel Stanovsky, Mausam, and Ido Dagan.
  • OpenIE as an intermediate structure for semantic tasks.
  • In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 303–308
  • Results:

    Results and Analysis

    6.1 Performance of Existing Systems

    How well do the neural systems perform as compared to the rule-based systems?

    Using CaRB evaluation, the authors find that, contrary to previous papers, neural OpenIE systems are not necessarily better than prior non-neural systems (Table 3).
  • RnnOIE performs much better, suffers due to its lack of generating auxilliary verbs and implied prepositions.
  • Example, it can only generate (Trump; President; US) instead of (Trump; is President of; US) from the sentence Filtering.
  • None Extraction-based Sentence-based Score-And-Filter “US President Trump...”.
  • It is trained only on limited number of pseudo-gold extractions, generated by Michael et al (2018), which does not take advantage of boostrapping techniques
  • Conclusion:

    The authors propose IMOJIE for the task of OpenIE. IMOJIE significantly improves upon the existing OpenIE systems in all three metrics, Optimal F1, AUC, and Last F1, establishing a new State Of the Art system.
  • See et al (2017) showed that using a coverage loss to track the attention over the decoded words improves the quality of the generated output
  • The authors add to this narrative by showing that deep inter-attention between the input and the partially-decoded words creates a better representation for iterative generation of triples.
  • This general observation may be of independent interest beyond OpenIE, such as in text summarization
Tables
  • Table1: IMOJIE vs. CopyAttention. CopyAttention suffers from stuttering, which IMOJIE does not
  • Table2: IMOJIE vs. OpenIE-4. Pipeline nature of OpenIE-4 can get confused by long convoluted sentences, but IMOJIE responds gracefully
  • Table3: Comparison of various OpenIE systems - nonneural, neural and proposed models. (*) Cannot compute AUC as Sense-OIE, MinIE do not emit confidence values for extractions and released code for Span-OIE does not provision calculation of confidence values. In these cases, we report the Last F1 as the Opt. F1
  • Table4: Models to solve the redundancy issue prevalent in Generative Neural OpenIE systems. All systems are bootstrapped on OpenIE-4
  • Table5: IMOJIE trained with different combinations of bootstrapping data from 3 systems - OpenIE-4, ClausIE, RNNOIE. Graph filtering is not used over single datasets
  • Table6: Performance of IMOJIE on aggregated dataset OpenIE-4+ClausIE+RnnOIE, with different filtering techniques. For comparison, SenseOIE trained on multiple system extractions gives an F1 of 17.2 on CaRB
  • Table7: Measuring redundancy of extractions. MNO stands for Mean Number of Occurrences. IOU stands for Intersection over Union
  • Table8: Evaluating models trained with different bootstrapping systems
  • Table9: Evaluation on other datasets with the CaRB evaluation strategy
Download tables as Excel
Related work
  • Open Information Extraction (OpenIE) involves extracting (arg1 phrase, relation phrase, arg2 phrase) assertions from a sentence. Traditional open extractors are rule-based or statistical, e.g., Textrunner (Banko et al, 2007), ReVerb (Fader et al, 2011; Etzioni et al, 2011), OLLIE (Mausam et al, 2012), Stanford-IE (Angeli et al, 2015), ClausIE (Del Corro and Gemulla, 2013), OpenIE4 (Christensen et al, 2011; Pal and Mausam, 2016), OpenIE-5 (Saha et al, 2017, 2018), PropS (Stanovsky et al, 2016), and MinIE (Gashteovski et al, 2017). These use syntactic or semantic parsers combined with rules to extract tuples from sentences.

    Recently, to reduce error accumulation in these pipeline systems, neural OpenIE models have been proposed. They belong to one of two paradigms: sequence labeling or sequence generation.

    Sequence Labeling involves tagging each word in the input sentence as belonging to the subject, predicate, object or other. The final extraction is obtained by collecting labeled spans into different fields and constructing a tuple. RnnOIE (Stanovsky et al, 2018) is a labeling system that first identifies the relation words and then uses sequence labelling to get their arguments. It is trained on OIE2016 dataset, which postprocesses SRL data for OpenIE (Stanovsky and Dagan, 2016).
Funding
  • IIT Delhi authors are supported by IBM AI Horizons Network grant, an IBM SUR award, grants by Google, Bloomberg and 1MG, and a Visvesvaraya faculty award by Govt. of India
  • Soumen is supported by grants from IBM and Amazon
Reference
  • Gabor Angeli, Melvin Jose Johnson Premkumar, and Christopher D. Manning. 2015. Leveraging Linguistic Structure for Open Domain Information Extraction. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing (ACL), 2015, pages 344– 354.
    Google ScholarLocate open access versionFindings
  • Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In International Conference on Learning Representations (ICLR), 2015.
    Google ScholarLocate open access versionFindings
  • Niranjan Balasubramanian, Stephen Soderland, Mausam, and Oren Etzioni. 201Generating Coherent Event Schemas at Scale. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2013, pages 1721–1731.
    Google ScholarLocate open access versionFindings
  • Michele Banko, Michael J Cafarella, Stephen Soderland, Matthew Broadhead, and Oren Etzioni. 2007. Open information extraction from the web. In International Joint Conference on Artificial Intelligence (IJCAI), 2007, volume 7, pages 2670–2676.
    Google ScholarLocate open access versionFindings
  • Sangnie Bhardwaj, Samarth Aggarwal, and Mausam. 2019. CaRB: A Crowdsourced Benchmark for OpenIE. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019, pages 6263–6268.
    Google ScholarLocate open access versionFindings
  • Janara Christensen, Mausam, Stephen Soderland, and Oren Etzioni. 2011. An analysis of open information extraction based on semantic role labeling. In Proceedings of the sixth international conference on Knowledge capture, pages 113–120. ACM.
    Google ScholarLocate open access versionFindings
  • Janara Christensen, Stephen Soderland, Gagan Bansal, et al. 2014. Hierarchical summarization: Scaling up multi-document summarization. In Proceedings of the 52nd annual meeting of the association for computational linguistics (volume 1: Long papers), pages 902–912.
    Google ScholarLocate open access versionFindings
  • Lei Cui, Furu Wei, and Ming Zhou. 201Neural open information extraction. In Proceedings of Association for Computational Linguistics (ACL), 2018, pages 407–413.
    Google ScholarLocate open access versionFindings
  • Luciano Del Corro and Rainer Gemulla. 2013. ClausIE: clause-based open information extraction. In Proceedings of the 22nd international conference on World Wide Web (WWW), 2013, pages 355–366. ACM.
    Google ScholarLocate open access versionFindings
  • Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of
    Google ScholarFindings
  • Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Oren Etzioni, Anthony Fader, Janara Christensen, Stephen Soderland, and Mausam. 2011. Open Information Extraction: The Second Generation. In IJCAI 2011, Proceedings of the 22nd International Joint Conference on Artificial Intelligence, Barcelona, Catalonia, Spain, July 16-22, 2011, pages 3–10. IJCAI/AAAI.
    Google ScholarLocate open access versionFindings
  • Anthony Fader, Stephen Soderland, and Oren Etzioni. 2011. Identifying Relations for Open Information Extraction. In Proceedings of the Conference of Empirical Methods in Natural Language Processing (EMNLP ’11), Edinburgh, Scotland, UK.
    Google ScholarLocate open access versionFindings
  • Angela Fan, Claire Gardent, Chloe Braud, and Antoine Bordes. 2019. Using Local Knowledge Graph Construction to Scale Seq2Seq Models to MultiDocument Inputs. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLPIJCNLP), 2019.
    Google ScholarLocate open access versionFindings
  • Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2018. AllenNLP: A Deep Semantic Natural Language Processing Platform. In Proceedings of Workshop for NLP Open Source Software (NLP-OSS), pages 1–6.
    Google ScholarLocate open access versionFindings
  • Kiril Gashteovski, Rainer Gemulla, and Luciano del Corro. 2017. MinIE: minimizing facts in open information extraction. In Association for Computational Linguistics (ACL), 2017.
    Google ScholarLocate open access versionFindings
  • Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O. K. Li. 2016. Incorporating Copying Mechanism in Sequence-to-Sequence Learning. In Proceedings of Association for Computational Linguistics (ACL), 2016. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Zhengbao Jiang, Pengcheng Yin, and Graham Neubig. 2019. Improving Open Information Extraction via Iterative Rank-Aware Learning. In Proceedings of the Association for Computational Linguistics (ACL), 2019.
    Google ScholarLocate open access versionFindings
  • William Lechelle, Fabrizio Gotti, and Philippe Langlais. 2018. Wire57: A fine-grained benchmark for open information extraction. In LAW@ACL.
    Google ScholarFindings
  • Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In Advances in Neural Information Processing Systems (NIPS), 2019, pages 13–23.
    Google ScholarLocate open access versionFindings
  • Mausam. 2016. Open information extraction systems and downstream applications. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence (IJCAI), 2016, pages 4074–4077. AAAI Press.
    Google ScholarLocate open access versionFindings
  • Mausam, Michael Schmitz, Robert Bart, Stephen Soderland, and Oren Etzioni. 2012. Open language learning for information extraction. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 523–534. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Julian Michael, Gabriel Stanovsky, Luheng He, Ido Dagan, and Luke Zettlemoyer. 2018. Crowdsourcing Question-Answer Meaning Representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), 2018, Volume 2 (Short Papers), pages 560–568.
    Google ScholarLocate open access versionFindings
  • Harinder Pal and Mausam. 2016. Demonyms and compound relational nouns in nominal OpenIE. In Proceedings of the 5th Workshop on Automated Knowledge Base Construction, pages 35–39.
    Google ScholarLocate open access versionFindings
  • Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training (2018).
    Google ScholarFindings
  • Carsten Rother, Vladimir Kolmogorov, Victor S. Lempitsky, and Martin Szummer. 2007. Optimizing Binary MRFs via Extended Roof Duality. 2007 IEEE Conference on Computer Vision and Pattern Recognition, pages 1–8.
    Google ScholarLocate open access versionFindings
  • Arpita Roy, Youngja Park, Taesung Lee, and Shimei Pan. 2019. Supervising Unsupervised Open Information Extraction Models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 728–737.
    Google ScholarLocate open access versionFindings
  • Swarnadeep Saha, Harinder Pal, and Mausam. 2017. Bootstrapping for numerical OpenIE. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 317–323. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Swarnadeep Saha et al. 2018. Open information extraction from conjunctive sentences. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2288–2299.
    Google ScholarLocate open access versionFindings
  • Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointergenerator networks. In Association for Computational Linguistics (ACL), 2017.
    Google ScholarLocate open access versionFindings
  • Gabriel Stanovsky and Ido Dagan. 2016. Creating a large benchmark for open information extraction. In Proceedings of the 2016 Conference on Empirical
    Google ScholarLocate open access versionFindings
  • Methods in Natural Language Processing (EMNLP), Austin, Texas. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Gabriel Stanovsky, Jessica Ficler, Ido Dagan, and Yoav Goldberg. 2016. Getting more out of syntax with PropS. CoRR, abs/1603.01648.
    Findings
  • Gabriel Stanovsky, Mausam, and Ido Dagan. 2015. OpenIE as an intermediate structure for semantic tasks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 303–308.
    Google ScholarLocate open access versionFindings
  • Gabriel Stanovsky, Julian Michael, Luke Zettlemoyer, and Ido Dagan. 2018. Supervised Open Information Extraction. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), Volume 1 (Long Papers), pages 885–895.
    Google ScholarLocate open access versionFindings
  • Mingming Sun, Xu Li, Xin Wang, Miao Fan, Yue Feng, and Ping Li. 2018. Logician: A unified end-toend neural approach for open-domain information extraction. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, pages 556–564.
    Google ScholarLocate open access versionFindings
  • Jesse Vig. 2019. A multiscale visualization of attention in the transformer model. In Proceedings of Association for Computational Linguistics (ACL), 2019.
    Google ScholarLocate open access versionFindings
  • Ashwin K. Vijayakumar, Michael Cogswell, Ramprasaath R. Selvaraju, Qing Sun, Stefan Lee, David J. Crandall, and Dhruv Batra. 2018. Diverse Beam Search for Improved Description of Complex Scenes. In AAAI Conference on Artificial Intelligence, 2018, pages 7371–7379.
    Google ScholarLocate open access versionFindings
  • Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015.
    Google ScholarFindings
  • Show, attend and tell: Neural image caption generation with visual attention. In International Conference on Machine Learning (ICML), 2015, pages 2048–2057.
    Google ScholarLocate open access versionFindings
  • Junlang Zhan and Hai Zhao. 2020. Span Model for Open Information Extraction on Accurate Corpus. In AAAI Conference on Artificial Intelligence, 2020, pages 5388–5399.
    Google ScholarLocate open access versionFindings
Your rating :
0

 

Tags
Comments