AI helps you reading Science

AI generates interpretation videos

AI extracts and analyses the key points of the paper to generate videos automatically


pub
Go Generating

AI Traceability

AI parses the academic lineage of this thesis


Master Reading Tree
Generate MRT

AI Insight

AI extracts a summary of this paper


Weibo:
Entity global representations model the semantic information of an entire document with R-GCN, and entity local representations aggregate the contextual information of mentions selectively using multi-head attention

Global to Local Neural Networks for Document Level Relation Extraction

EMNLP 2020, pp.3711-3721, (2020)

Cited by: 0|Views430
Full Text
Bibtex
Weibo

Abstract

Relation extraction (RE) aims to identify the semantic relations between named entities in text. Recent years have witnessed it raised to the document level, which requires complex reasoning with entities and mentions throughout an entire document. In this paper, we propose a novel model to document-level RE, by encoding the document info...More

Code:

Data:

0
Introduction
  • Relation extraction (RE) aims to identify the semantic relations between named entities in text.
  • While previous work (Zeng et al, 2014; Zhang et al, 2015, 2018) focuses on extracting relations within a sentence, a.k.a. sentence-level RE, recent studies (Verga et al, 2018; Christopoulou et al, 2019; Sahu et al, 2019; Yao et al, 2019) have escalated it to the document level, since a large amount of relations between entities usually span across multiple sentences in the real world.
  • To identify the relations between entities appearing in different sentences, document-level RE models must be capable of modeling the complex interactions between multiple entities and synthesizing the context information of multiple mentions
Highlights
  • Relation extraction (RE) aims to identify the semantic relations between named entities in text
  • To identify the relations between entities appearing in different sentences, document-level RE models must be capable of modeling the complex interactions between multiple entities and synthesizing the context information of multiple mentions
  • We find that: (1) BERT had a greater influence on DocRED than Chemical-Disease Relations (CDR)
  • Entity global representations model the semantic information of an entire document with R-GCN, and entity local representations aggregate the contextual information of mentions selectively using multi-head attention
  • Context relation representations encode the topic information of other relations using self-attention
  • Our experiments demonstrated the superiority of GLRE over many comparative models, especially the big leads in extracting relations between entities of long distance and with multiple mentions
Results
  • The authors implemented the GLRE with PyTorch 1.5. The source code and datasets are available online.1 the authors report the experimental results. 4.1 Datasets

    The authors evaluated GLRE on two public document-level RE datasets.
  • 2018; Zheng et al, 2018) achieved comparable results, while the best graph-based model (Christopoulou et al, 2019) outperformed the best non-graph (Nguyen and Verspoor, 2018)
  • The authors attribute it to the document graph on the entity level, which can better model the semantic information in a document.
  • (3) From the results of Wang et al (2019); Tang et al (2020), the BERT-based models showed stronger prediction power for document-level RE
  • They outperformed the other comparative models on both CDR and DocRED.
Conclusion
  • The authors proposed GLRE, a global-to-local neural network for document-level RE.
  • Entity global representations model the semantic information of an entire document with R-GCN, and entity local representations aggregate the contextual information of mentions selectively using multi-head attention.
  • The authors' experiments demonstrated the superiority of GLRE over many comparative models, especially the big leads in extracting relations between entities of long distance and with multiple mentions.
  • The authors plan to integrate knowledge graphs and explore other document graph modeling ways to improve the performance
Tables
  • Table1: Dataset statistics (Inst.: relation instances excluding N/A relation; N/A Inst.: negative examples)
  • Table2: Result comparison on CDR
  • Table3: Result comparison on DocRED
  • Table4: Results of ablation study
  • Table5: Results w.r.t. different pre-training models
  • Table6: Case study on the CDR test set. CID is short for the “chemical-induced disease” relation. Target entities and related entities are colored accordingly
  • Table7: Notations in the paper
  • Table8: Hyperparameters in the experiments
  • Table9: Case study on the DocRED development set. Target entities and related entities are colored
Download tables as Excel
Related work
  • RE has been intensively studied in a long history. In this section, we review closely-related work.

    Sentence-level RE. Conventional work addressed sentence-level RE by using carefully-designed patterns (Soderland et al, 1995), features (Kambhatla, 2004) and kernels (Culotta and Sorensen, 2004). Recently, deep learning-based work has advanced the state-of-the-art without heavy feature engineering. Various neural networks have been exploited, e.g., CNN (Zeng et al, 2014), RNN (Zhang et al, 2015; Cai et al, 2016) and GNN (Zhang et al, 2018). Furthermore, to cope with the wrong labeling problem caused by distant supervision, Zeng et al (2015) adopted Piecewise CNN (PCNN), Lin et al (2016); Zhang et al (2017) employed attention mechanisms, and Zhang et al (2019); Qu et al (2019) leveraged knowledge graphs as external resources. All these models are limited to extracting intra-sentential relations. They also ignore the interactions of entities outside a target entity pair.
Funding
  • This work is supported partially by the National Key R&D Program of China (No 2018YFB1004300), the National Natural Science Foundation of China (No 61872172), and the Water Resource Science & Technology Project of Jiangsu Province (No 2019046)
Study subjects and analysis
patients: 13
Case 2 Label: CID GLRE: CID Wang et al.: N/A. [S1] Clinical evaluation of adverse effects during bepridil administration for atrial fibrillation and flutter. ... [S8] There was marked QT prolongation greater than 0.55 s in 13 patients ... and general fatigue in 1 patient each. Case 3 Label: CID GLRE: N/A Wang et al.: N/A

Reference
  • Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. arXiv:1607.06450.
    Findings
  • Rui Cai, Xiaodong Zhang, and Houfeng Wang. 2016. Bidirectional recurrent convolutional neural network for relation classification. In ACL, pages 756–765, Berlin, Germany. ACL.
    Google ScholarLocate open access versionFindings
  • Billy Chiu, Gamal Crichton, Anna Korhonen, and Sampo Pyysalo. 2016. How to train good word embeddings for biomedical NLP. In BioNLP, pages 166–174, Berlin, Germany. ACL.
    Google ScholarLocate open access versionFindings
  • Fenia Christopoulou, Makoto Miwa, and Sophia Ananiadou. 2019. Connecting the dots: Document-level neural relation extraction with edge-oriented graphs. In EMNLP-IJCNLP, pages 4925–4936, Hong Kong. ACL.
    Google ScholarLocate open access versionFindings
  • Aron Culotta and Jeffrey Sorensen. 2004. Dependency tree kernels for relation extraction. In ACL, pages 423–429, Barcelona, Spain. ACL.
    Google ScholarLocate open access versionFindings
  • Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, pages 4171–4186, Minneapolis, MN, USA. ACL.
    Google ScholarLocate open access versionFindings
  • Jinghang Gu, Fuqing Sun, Longhua Qian, and Guodong Zhou. 201Chemical-induced disease relation extraction via convolutional neural network. Database, page bax024.
    Google ScholarLocate open access versionFindings
  • Pankaj Gupta, Subburam Rajaram, Hinrich Schutze, and Thomas Runkler. 2019. Neural relation extraction within and across sentence boundaries. In AAAI, pages 6513–6520, Honolulu, HI, USA. AAAI Press.
    Google ScholarLocate open access versionFindings
  • Nanda Kambhatla. 2004. Combining lexical, syntactic, and semantic features with maximum entropy models for extracting relations. In ACL 2004, pages 178– 181, Barcelona, Spain. ACL.
    Google ScholarLocate open access versionFindings
  • Diederik P. Kingma and Jimmy Lei Ba. 2015. Adam: A method for stochastic optimization. In ICLR, San Diego, CA, USA. OpenReview.net.
    Google ScholarLocate open access versionFindings
  • Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In ICLR, Addis Ababa, Ethiopia. OpenReview.net.
    Google ScholarLocate open access versionFindings
  • Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. BioBERT: A pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, page btz682.
    Google ScholarLocate open access versionFindings
  • Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017. Zero-shot relation extraction via reading comprehension. In CoNLL, pages 333–342, Vancouver, Canada. ACL.
    Google ScholarLocate open access versionFindings
  • Jiao Li, Yueping Sun, Robin J Johnson, Daniela Sciaky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J Mattingly, Thomas C Wiegers, and Zhiyong Lu. 2016. BioCreative V CDR task corpus: A resource for chemical disease relation extraction. Database, page baw068.
    Google ScholarLocate open access versionFindings
  • Yankai Lin, Shiqi Shen, Zhiyuan Liu, Huanbo Luan, and Maosong Sun. 2016. Neural relation extraction with selective attention over instances. In ACL, pages 2124–2133, Berlin, Germany. ACL.
    Google ScholarLocate open access versionFindings
  • Shanshan Liu, Xin Zhang, Sheng Zhang, Hui Wang, and Weiming Zhang. 2019. Neural machine reading comprehension: Methods and trends. Applied Sciences, 9(18):3698.
    Google ScholarLocate open access versionFindings
  • Guoshun Nan, Zhijiang Guo, Ivan Sekulic, and Wei Lu. 2020. Reasoning with latent structure refinement for document-level relation extraction. In ACL, pages 1546–1557, Online. ACL.
    Google ScholarLocate open access versionFindings
  • Dat Quoc Nguyen and Karin Verspoor. 20Convolutional neural networks for chemical-disease relation extraction are improved with character-based word embeddings. In BioNLP, pages 129–136, Melbourne, Australia. ACL.
    Google ScholarLocate open access versionFindings
  • Nagesh C Panyam, Karin Verspoor, Trevor Cohn, and Kotagiri Ramamohanarao. 2018. Exploiting graph kernels for high performance biomedical relation extraction. Journal of Biomedical Semantics, 9:7.
    Google ScholarLocate open access versionFindings
  • Michael Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. 2018. Modeling relational data with graph convolutional networks. In ESWC, pages 593–607, Heraklion, Greece. Springer.
    Google ScholarLocate open access versionFindings
  • Stephen Soderland, David Fisher, Jonathan Aseltine, and Wendy Lehnert. 1995. CRYSTAL: Inducing a conceptual dictionary. In IJCAI, pages 1314–1319, Montreal, Canada. Morgan Kaufmann Publishers.
    Google ScholarLocate open access versionFindings
  • Linfeng Song, Yue Zhang, Zhiguo Wang, and Daniel Gildea. 2018. N-ary relation extraction using graphstate LSTM. In EMNLP, pages 2226–2235, Brussels, Belgium. ACL.
    Google ScholarLocate open access versionFindings
  • Daniil Sorokin and Iryna Gurevych. 2017. Contextaware representations for knowledge base relation extraction. In EMNLP, pages 1784–1789, Copenhagen, Denmark. ACL.
    Google ScholarLocate open access versionFindings
  • Hengzhu Tang, Yanan Cao, Zhenyu Zhang, Jiangxia Cao, Fang Fang, Shi Wang, and Pengfei Yin. 2020. HIN: Hierarchical inference network for documentlevel relation extraction. In PAKDD, pages 197–209, Singapore. Springer.
    Google ScholarLocate open access versionFindings
  • Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS, pages 5998–6008, Long Beach, CA, USA. Curran Associates, Inc.
    Google ScholarLocate open access versionFindings
  • Nanyun Peng, Hoifung Poon, Chris Quirk, Kristina Toutanova, and Wen-tau Yih. 2017. Cross-sentence n-ary relation extraction with graph LSTMs. TACL, 5:101–115.
    Google ScholarLocate open access versionFindings
  • Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. GloVe: Global vectors for word representation. In EMNLP, pages 1532–1543, Doha, Qatar. ACL.
    Google ScholarLocate open access versionFindings
  • Patrick Verga, Emma Strubell, and Andrew McCallum. 2018. Simultaneously self-attending to all mentions for full-abstract biological relation extraction. In NAACL, pages 872–884, New Orleans, LA, USA. ACL.
    Google ScholarLocate open access versionFindings
  • Hong Wang, Christfried Focke, Rob Sylvester, Nilesh Mishra, and William Wang. 2019. Fine-tune Bert for DocRED with two-step process. arXiv:1909.11898.
    Findings
  • Lin Qiu, Hao Zhou, Yanru Qu, Weinan Zhang, Suoheng Li, Shu Rong, Dongyu Ru, Lihua Qian, Kewei Tu, and Yong Yu. 2018. QA4IE: A question answering based framework for information extraction. In ISWC, pages 198–216, Monterey, CA, USA. Springer.
    Google ScholarLocate open access versionFindings
  • Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R. Salakhutdinov, and Quoc V. Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. In NeurIPS, pages 5753–5763, Vancouver, Canada. Curran Associates, Inc.
    Google ScholarLocate open access versionFindings
  • Jianfeng Qu, Wen Hua, Dantong Ouyang, Xiaofang Zhou, and Ximing Li. 2019. A fine-grained and noise-aware method for neural relation extraction. In CIKM, pages 659–668, Beijing, China. ACM.
    Google ScholarLocate open access versionFindings
  • Chris Quirk and Hoifung Poon. 2017. Distant supervision for relation extraction beyond the sentence boundary. In EACL, pages 1171–1182, Valencia, Spain. ACL.
    Google ScholarLocate open access versionFindings
  • Sunil Kumar Sahu, Fenia Christopoulou, Makoto Miwa, and Sophia Ananiadou. 2019. Inter-sentence relation extraction with document-level graph convolutional neural network. In ACL, pages 4309–4316, Florence, Italy. ACL.
    Google ScholarLocate open access versionFindings
  • Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. 2019. DocRED: A large-scale document-level relation extraction dataset. In ACL, pages 764–777, Florence, Italy. ACL.
    Google ScholarLocate open access versionFindings
  • Deming Ye, Yankai Lin, Jiaju Du, Zhenghao Liu, Maosong Sun, and Zhiyuan Liu. 2020. Coreferential reasoning learning for language representation. In EMNLP, Online. ACL.
    Google ScholarLocate open access versionFindings
  • Daojian Zeng, Kang Liu, Yubo Chen, and Jun Zhao. 2015. Distant supervision for relation extraction via piecewise convolutional neural networks. In EMNLP, pages 1753–1762, Lisbon, Portugal. ACL.
    Google ScholarLocate open access versionFindings
  • Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, and Jun Zhao. 2014. Relation classification via convolutional deep neural network. In COLING, pages 2335–2344, Dublin, Ireland. ACL.
    Google ScholarLocate open access versionFindings
  • Ningyu Zhang, Shumin Deng, Zhanlin Sun, Guanying Wang, Xi Chen, Wei Zhang, and Huajun Chen. 2019. Long-tail relation extraction via knowledge graph embeddings and graph convolution networks. In NAACL-HLT, pages 3016–3025, Minneapolis, MN, USA. ACL.
    Google ScholarLocate open access versionFindings
  • Shu Zhang, Dequan Zheng, Xinchen Hu, and Ming Yang. 2015. Bidirectional long short-term memory networks for relation classification. In PACLIC, pages 73–78, Shanghai, China. ACL.
    Google ScholarLocate open access versionFindings
  • Yuhao Zhang, Peng Qi, and Christopher D Manning. 2018. Graph convolution over pruned dependency trees improves relation extraction. In EMNLP, pages 2205–2215, Brussels, Belgium. ACL.
    Google ScholarLocate open access versionFindings
  • Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017. Positionaware attention and supervised data improve slot filling. In EMNLP, pages 35–45, Copenhagen, Denmark. ACL.
    Google ScholarLocate open access versionFindings
  • Wei Zheng, Hongfei Lin, Zhiheng Li, Xiaoxia Liu, Zhengguang Li, Bo Xu, Yijia Zhang, Zhihao Yang, and Jian Wang. 2018. An effective neural model extracting document level chemical-induced disease relations from biomedical literature. Journal of Biomedical Informatics, 83:1–9.
    Google ScholarLocate open access versionFindings
  • Huiwei Zhou, Huijie Deng, Long Chen, Yunlong Yang, Chen Jia, and Degen Huang. 2016. Exploiting syntactic and semantics information for chemicaldisease relation extraction. Database, page baw048.
    Google ScholarLocate open access versionFindings
Author
Difeng Wang
Difeng Wang
Ermei Cao
Ermei Cao
Weijian Sun
Weijian Sun
Your rating :
0

 

Tags
Comments
数据免责声明
页面数据均来自互联网公开来源、合作出版商和通过AI技术自动分析结果,我们不对页面数据的有效性、准确性、正确性、可靠性、完整性和及时性做出任何承诺和保证。若有疑问,可以通过电子邮件方式联系我们:report@aminer.cn
小科