AI帮你理解科学

AI 生成解读视频

AI抽取解析论文重点内容自动生成视频


pub
生成解读视频

AI 溯源

AI解析本论文相关学术脉络


Master Reading Tree
生成 溯源树

AI 精读

AI抽取本论文的概要总结


微博一下
We demonstrate that the identified hierarchical discourse segments improve performance of the model on two tasks, automated essay scoring and assessing writing quality

Centering based Neural Coherence Modeling with Hierarchical Discourse Segments

EMNLP 2020, (2020)

被引用0|浏览9
下载 PDF 全文
引用
微博一下

摘要

Previous neural coherence models have focused on identifying semantic relations between adjacent sentences. However, they do not have the means to exploit structural information. In this work, we propose a coherence model which takes discourse structural information into account without relying on human annotations. We approximate a lingu...更多

代码

数据

0
简介
  • Coherence describes the semantic relation between elements of a text
  • It identifies a text passage as either a unified whole or a collection of unrelated sentences.
  • Prior studies of coherence have mainly focused on modeling local coherence in Centering theory (Barzilay and Lapata, 2008).
  • They aim to identify the semantic relations between adjacent sentences.
  • Incorporating structural information into the model has been beneficial for diverse downstream tasks including text summarization (Marcu, 2000), translation (Guzman et al, 2014), sentiment analysis (Bhatia et al, 2015), and text classification (Ji and Smith, 2017)
重点内容
  • Coherence describes the semantic relation between elements of a text
  • We propose a coherence model inspired by Centering theory which takes structural information into consideration
  • Our model achieves state-of-the-art performance on New York Times (NYT) among the models using the pretrained embedding layer, but it still shows lower performance than the model using the embedding layer trained on the target corpus
  • We propose a neural model of coherence inspired by Centering theory
  • Our model identifies the hierarchy of discourse segments without human annotations, and incorporates structural information into the model
  • We demonstrate that the identified hierarchical discourse segments improve performance of the model on two tasks, automated essay scoring and assessing writing quality
方法
  • The authors implement the model using the PyTorch library and use the Stanford Stanza library2 for sentence tokenization.
  • The authors employ XLNet for the pretrainedlanguage model.
  • For the baselines that do not use the pretrained language model, the authors use Glove for word embeddings, the pretrained word embeddings trained on Google News (Pennington et al, 2014).
  • The authors encode each sentence separately using XLNet instead of the whole document at once.
  • The authors' dataset consists of long documents i.e., journal articles with more than 3,000 tokens.
  • For employing the pretrained model, it is
结果
  • The authors' model achieves state-of-the-art performance on NYT among the models using the pretrained embedding layer, but it still shows lower performance than the model using the embedding layer trained on the target corpus.
  • This suggests that linguistic cues have the potential to improve this model further
结论
  • The authors propose a neural model of coherence inspired by Centering theory.
  • The intuition is that it describes coherence by tracking the changes of the focus between discourse segments.
  • The authors' model identifies the hierarchy of discourse segments without human annotations, and incorporates structural information into the model.
  • The authors demonstrate that the identified hierarchical discourse segments improve performance of the model on two tasks, automated essay scoring and assessing writing quality.
  • The authors find statistical differences of trees generated from texts of different quality
总结
  • Introduction:

    Coherence describes the semantic relation between elements of a text
  • It identifies a text passage as either a unified whole or a collection of unrelated sentences.
  • Prior studies of coherence have mainly focused on modeling local coherence in Centering theory (Barzilay and Lapata, 2008).
  • They aim to identify the semantic relations between adjacent sentences.
  • Incorporating structural information into the model has been beneficial for diverse downstream tasks including text summarization (Marcu, 2000), translation (Guzman et al, 2014), sentiment analysis (Bhatia et al, 2015), and text classification (Ji and Smith, 2017)
  • Methods:

    The authors implement the model using the PyTorch library and use the Stanford Stanza library2 for sentence tokenization.
  • The authors employ XLNet for the pretrainedlanguage model.
  • For the baselines that do not use the pretrained language model, the authors use Glove for word embeddings, the pretrained word embeddings trained on Google News (Pennington et al, 2014).
  • The authors encode each sentence separately using XLNet instead of the whole document at once.
  • The authors' dataset consists of long documents i.e., journal articles with more than 3,000 tokens.
  • For employing the pretrained model, it is
  • Results:

    The authors' model achieves state-of-the-art performance on NYT among the models using the pretrained embedding layer, but it still shows lower performance than the model using the embedding layer trained on the target corpus.
  • This suggests that linguistic cues have the potential to improve this model further
  • Conclusion:

    The authors propose a neural model of coherence inspired by Centering theory.
  • The intuition is that it describes coherence by tracking the changes of the focus between discourse segments.
  • The authors' model identifies the hierarchy of discourse segments without human annotations, and incorporates structural information into the model.
  • The authors demonstrate that the identified hierarchical discourse segments improve performance of the model on two tasks, automated essay scoring and assessing writing quality.
  • The authors find statistical differences of trees generated from texts of different quality
表格
  • Table1: Three types of centering transitions
  • Table2: TOEFL Accuracy performance comparison on the test sets (see Appendix D for more details)
  • Table3: Mean (standard deviation) accuracy performance of assessing writing quality on the test sets in NYT. We compare the performance of <a class="ref-link" id="cLiu_2018_a" href="#rLiu_2018_a">Liu and Lapata (2018</a>), reported in <a class="ref-link" id="cFerracane_et+al_2019_a" href="#rFerracane_et+al_2019_a">Ferracane et al (2019</a>) which uses an embedding layer trained on NYT and our implementation which uses a pretrained Glove embedding layer
  • Table4: Statistics for learned trees as labels by our model described as mean (standard deviation)
  • Table5: Top-10 most preferred centers (proportions) of essays submitted to the same prompt in TOEFL, a NYT article whose id is 1458761, and a NYT article whose id is 1516415 (see Appendix F for more details)
  • Table6: Proportion of function words determined as centers in essays submitted to the prompt 1 and 5 in TOEFL (T), a NYT article whose id is 1458761 (N14*), and a NYT article whose id is 1516415 (N-15*)
  • Table7: Dataset statistics on tokenization: each TOEFL prompt (T-P) and NYT
  • Table8: Topic description: TOEFL
  • Table9: TOEFL accuracy performance comparison on the test sets, described as mean (std)
  • Table10: TOEFL accuracy performance comparison on the validation sets, described as mean (std)
  • Table11: Top-10 most preferred centers (proportions) of essays submitted to the same prompt in TOEFL (see Appendix. A for given topics) and four articles in NYT whose id is 1458761, 1516415, 1705265, and 1254567, respectively. The title of NYT articles are as follows, 1458761: “Among 4 States, a Great Divide in Fortunes”, 1516415: “One Cosmic Question, Too Many Answers”, 1705265: “Which of These Foods Will Stop Cancer?”, and 1254567: “Quantum Theory Tugged, And All of Physics Unraveled”
Download tables as Excel
相关工作
  • While unsupervised approaches for discourse parser have been developed (Marcu and Echihabi, 2002; Ji et al, 2015), earlier work mostly adopted a supervised approach to identify discourse structure relying on human annotations. Subba and Di Eugenio (2009) incorporate various linguistic features, including compositional semantics and part-of-speech information, to propose a discourse parser based on Inductive Logic Programming. Hernault et al (2010) introduce a discourse parser which constructs discourse structure from a full input text. They train classifiers to identify discourse relations, and use them to build a tree structure of an input text. Feng and Hirst (2012) improve the tree building algorithm of this system by incorporating more linguistic features. Wang et al (2017) introduce an SVM-based model that consists of two stages, one identifying discourse structure, and the other classifying types of relations between units.
基金
  • This work has been funded by the Klaus Tschira Foundation, Heidelberg, Germany
  • The first author has been supported by a Heidelberg Institute for Theoretical Studies Ph.D. scholarship
研究对象与分析
articles: 4
TOEFL accuracy performance comparison on the validation sets, described as mean (std). Top-10 most preferred centers (proportions) of essays submitted to the same prompt in TOEFL (see Appendix. A for given topics) and four articles in NYT whose id is 1458761, 1516415, 1705265, and 1254567, respectively. The title of NYT articles are as follows, 1458761: “Among 4 States, a Great Divide in Fortunes”, 1516415: “One Cosmic Question, Too Many Answers”, 1705265: “Which of These Foods Will Stop Cancer?”, and 1254567: “Quantum Theory Tugged, And All of Physics Unraveled”. Our model architecture

引用论文
  • Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the ICLR Conference.
    Google ScholarLocate open access versionFindings
  • Regina Barzilay and Mirella Lapata. 2008. Modeling local coherence: An entity-based approach. Computational Linguistics, 34(1):1–34.
    Google ScholarLocate open access versionFindings
  • Parminder Bhatia, Yangfeng Ji, and Jacob Eisenstein. 2015. Better document-level sentiment analysis from RST discourse parsing. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2212–2218, Lisbon, Portugal. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Chloe Braud, Barbara Plank, and Anders Søgaard. 2016. Multi-view and multi-task training of RST discourse parsers. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1903–1913, Osaka, Japan. The COLING 2016 Organizing Committee.
    Google ScholarLocate open access versionFindings
  • Susan E Brennan, Marilyn W Friedman, and Carl Pollard. 1987. A centering approach to pronouns. In 25th Annual Meeting of the Association for Computational Linguistics, pages 155–162.
    Google ScholarLocate open access versionFindings
  • Jill Burstein, Joel Tetreault, and Slava Andreyev. 2010. Using entity-based features to model coherence in student essays. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 681–684, Los Angeles, California. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Jihun Choi, Kang Min Yoo, and Sang-goo Lee. 2018. Learning to compose task-specific tree structures. In Thirty-Second AAAI Conference on Artificial Intelligence.
    Google ScholarLocate open access versionFindings
  • Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? An analysis of BERT’s attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276–286, Florence, Italy. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 201BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A. Smith. 2019. Show your work: Improved reporting of experimental results. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2185– 2194, Hong Kong, China. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Fei Dong and Yue Zhang. 2016. Automatic features for essay scoring – an empirical study. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1072–1077, Austin, Texas. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Fei Dong, Yue Zhang, and Jie Yang. 2017. Attentionbased recurrent convolutional neural network for automatic essay scoring. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 153–162, Vancouver, Canada. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Vanessa Wei Feng and Graeme Hirst. 2012. Text-level discourse parsing with rich linguistic features. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 60–68, Jeju Island, Korea. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Vanessa Wei Feng, Ziheng Lin, and Graeme Hirst. 20The impact of deep hierarchical discourse structures in the evaluation of text coherence. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 940–949, Dublin, Ireland. Dublin City University and Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Elisa Ferracane, Greg Durrett, Junyi Jessy Li, and Katrin Erk. 2019. Evaluating discourse in structured text representations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 646–653, Florence, Italy. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Barbara J. Grosz, Aravind K. Joshi, and Scott Weinstein. 1995. Centering: A framework for modeling the local coherence of discourse. Computational Linguistics, 21(2):203–225.
    Google ScholarLocate open access versionFindings
  • Barbara J. Grosz and Candace L. Sidner. 1986.
    Google ScholarFindings
  • Francisco Guzman, Shafiq Joty, Lluıs Marquez, and Preslav Nakov. 2014. Using discourse structure improves machine translation evaluation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 687–698, Baltimore, Maryland. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Hugo Hernault, Helmut Prendinger, Mitsuru Ishizuka, et al. 2010. HILDA: A discourse parser using support vector machine classification. Dialogue & Discourse, 1(3).
    Google ScholarLocate open access versionFindings
  • Yangfeng Ji and Noah A. Smith. 2017. Neural discourse structure for text categorization. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 996–1005, Vancouver, Canada. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Yangfeng Ji, Gongbo Zhang, and Jacob Eisenstein. 2015. Closing the gap: Domain adaptation from explicit to implicit discourse relations. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2219– 2224, Lisbon, Portugal. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Jiwei Li, Rumeng Li, and Eduard Hovy. 2014. Recursive deep models for discourse parsing. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2061–2069, Doha, Qatar. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Qi Li, Tianshi Li, and Baobao Chang. 2016. Discourse parsing with attention-based hierarchical neural networks. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 362–371, Austin, Texas. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Yang Liu and Mirella Lapata. 2018. Learning structured text representations. Transactions of the Association for Computational Linguistics, 6:63–75.
    Google ScholarLocate open access versionFindings
  • Annie Louis and Ani Nenkova. 2013. What makes writing great? First experiments on article quality prediction in the science journalism domain. Transactions of the Association for Computational Linguistics, 1:341–352.
    Google ScholarLocate open access versionFindings
  • William C. Mann and Sandra A. Thompson. 1988. Rhetorical structure theory. Toward a functional theory of text organization. Text, 8(3):243–281.
    Google ScholarLocate open access versionFindings
  • Daniel Marcu. 2000. The Theory and Practice of Discourse Parsing and Summarization. MIT Press, Cambridge, MA, USA.
    Google ScholarFindings
  • Daniel Marcu and Abdessamad Echihabi. 2002. An unsupervised approach to recognizing discourse relations. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 368–375, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Mohsen Mesgar and Michael Strube. 2018. A neural local coherence model for text quality assessment. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4328–4339, Brussels, Belgium. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Cansu Sen, Thomas Hartvigsen, Biao Yin, Xiangnan Kong, and Elke Rundensteiner. 2020. Human attention maps for text classification: Do humans and neural networks focus on the same words? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4596– 4608, Online. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Rajen Subba and Barbara Di Eugenio. 2009. An effective discourse parser that uses rich linguistic information. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 566–574, Boulder, Colorado. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Kaveh Taghipour and Hwee Tou Ng. 2016. A neural approach to automated essay scoring. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1882–1891, Austin, Texas. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008.
    Google ScholarLocate open access versionFindings
  • Jesse Vig and Yonatan Belinkov. 2019. Analyzing the structure of attention in a transformer language model. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 63–76, Florence, Italy. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Yaushian Wang, Hung-Yi Lee, and Yun-Nung Chen. 2019. Tree transformer: Integrating tree structures into self-attention. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLPIJCNLP), pages 1061–1070, Hong Kong, China. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Yizhong Wang, Sujian Li, and Houfeng Wang. 2017. A two-stage parsing method for text-level discourse analysis. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 184–188, Vancouver, Canada. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Bonnie Webber, Rashmi Prasad, Alan Lee, and Aravind Joshi. 2019. The Penn discourse treebank 3.0 annotation manual.
    Google ScholarFindings
  • Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R. Salakhutdinov, and Quoc V. Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems, pages 5754– 5764.
    Google ScholarLocate open access versionFindings
  • Dani Yogatama, Phil Blunsom, Chris Dyer, Edward Grefenstette, and Wang Ling. 2017. Learning to compose words into sentences with reinforcement learning. In Proceedings of the ICLR Conference.
    Google ScholarLocate open access versionFindings
  • Nan Yu, Meishan Zhang, and Guohong Fu. 2018. Transition-based neural RST parsing with implicit syntax features. In Proceedings of the 27th International Conference on Computational Linguistics, pages 559–570, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
作者
Sungho Jeon
Sungho Jeon
Michael Strube
Michael Strube
您的评分 :
0

 

标签
评论
小科