Shape of synth to come: Why we should use synthetic data for English surface realization

Elder Henry
Elder Henry
Burke Robert
Burke Robert
O'Connor Alexander
O'Connor Alexander

ACL, pp. 7465-7471, 2020.

Cited by: 0|Views11
EI
Weibo:
We have argued for the use of synthetic data in English surface realization, justified by the fact that its use gives a significant performance boost on the shallow task, from 72.7 BLEU up to 80.1

Abstract:

The Surface Realization Shared Tasks of 2018 and 2019 were Natural Language Generation shared tasks with the goal of exploring approaches to surface realization from Universal-Dependency-like trees to surface strings for several languages. In the 2018 shared task there was very little difference in the absolute performance of systems tr...More

Code:

Data:

0
Full Text
Bibtex
Weibo
Introduction
  • The shallow task of the recent surface realization (SR) shared tasks (Belz et al, 2011; Mille et al, 2018, 2019) appears to be a relatively straightforward problem.
  • SR systems often struggle, even for a relatively fixed word order language such as English.
  • Improved performance would facilitate investigation of more complex versions of the shallow task, such as the deep task in which function words are pruned from the tree, which may be of more practical use in pipeline natural language generation (NLG) systems (Moryossef et al, 2019; Elder et al, 2019; come AP.
  • Synthetic data is created by taking an unlabelled sentence, parsing it with an open source universal dependency parser1 and transforming the result into the input representation
Highlights
  • The shallow task of the recent surface realization (SR) shared tasks (Belz et al, 2011; Mille et al, 2018, 2019) appears to be a relatively straightforward problem
  • We evaluate on the Surface Realization Shared Task (SRST) 2018 dataset (Mille et al, 2018) for English5, which was derived from the Universal Dependency English Web Treebank 2.06
  • We have argued for the use of synthetic data in English surface realization, justified by the fact that its use gives a significant performance boost on the shallow task, from 72.7 BLEU up to 80.1
  • While this is not yet at the level of reliability needed for neural natural language generation systems to be used commercially, it is a step in the right direction
  • The work described in this paper has focused on English
  • Another avenue of research would be to investigate the role of synthetic data in surface realization in other languages
Results
  • In Section 2.3, the authors described three improvements to the baseline system: random linearization, scoping and restricted beam search.
  • 3.1 The Effect of Synthetic Data.
  • The last row of Table 1 shows the effect of adding synthetic data.
  • To help understand why additional data makes such a substantial difference, the authors perform various analyses on the dev set, including examining the effect of the choice of unlabeled corpus and highlighting interesting differences between the systems trained with and without the synthetic data
Conclusion
  • The authors have argued for the use of synthetic data in English surface realization, justified by the fact that its use gives a significant performance boost on the shallow task, from 72.7 BLEU up to 80.1
  • While this is not yet at the level of reliability needed for neural NLG systems to be used commercially, it is a step in the right direction.
  • Another avenue of research would be to investigate the role of synthetic data in surface realization in other languages
Summary
  • Introduction:

    The shallow task of the recent surface realization (SR) shared tasks (Belz et al, 2011; Mille et al, 2018, 2019) appears to be a relatively straightforward problem.
  • SR systems often struggle, even for a relatively fixed word order language such as English.
  • Improved performance would facilitate investigation of more complex versions of the shallow task, such as the deep task in which function words are pruned from the tree, which may be of more practical use in pipeline natural language generation (NLG) systems (Moryossef et al, 2019; Elder et al, 2019; come AP.
  • Synthetic data is created by taking an unlabelled sentence, parsing it with an open source universal dependency parser1 and transforming the result into the input representation
  • Results:

    In Section 2.3, the authors described three improvements to the baseline system: random linearization, scoping and restricted beam search.
  • 3.1 The Effect of Synthetic Data.
  • The last row of Table 1 shows the effect of adding synthetic data.
  • To help understand why additional data makes such a substantial difference, the authors perform various analyses on the dev set, including examining the effect of the choice of unlabeled corpus and highlighting interesting differences between the systems trained with and without the synthetic data
  • Conclusion:

    The authors have argued for the use of synthetic data in English surface realization, justified by the fact that its use gives a significant performance boost on the shallow task, from 72.7 BLEU up to 80.1
  • While this is not yet at the level of reliability needed for neural NLG systems to be used commercially, it is a step in the right direction.
  • Another avenue of research would be to investigate the role of synthetic data in surface realization in other languages
Tables
  • Table1: Test set results for baselines trained on the original dataset and the final model which uses synthetic data
  • Table2: Dev set results for ablation of the baseline system plus improvements, trained only on the original dataset
  • Table3: Dev set results for the SR shared task data with additional synthetic data: the role of the corpus tions. However, all three make a meaningful, positive contribution
  • Table4: Error analysis breakdown for the 1,978 dev sentences. SRST is our system without synthetic data and Synth is our system with synthetic data
Download tables as Excel
Funding
  • This research is supported by Science Foundation Ireland in the ADAPT Centre for Digital Content Technology
  • The ADAPT Centre for Digital Content Technology is funded under the SFI Research Centres Programme (Grant 13/RC/2106) and is co-funded under the European Regional Development Fund
Reference
  • Anja Belz, Michael White, Dominic Espinosa, Eric Kow, Deirdre Hogan, and Amanda Stent. 201The First Surface Realisation Shared Task: Overview and Evaluation Results. In Proceedings of the European Workshop on Natural Language Generation, December, pages 217–226.
    Google ScholarLocate open access versionFindings
  • Bernd Bohnet, Leo Wanner, Simon Mille, and Alicia Burga. 2010. Broad Coverage Multilingual Deep Sentence Generation with a Stochastic Multi-Level Realizer. In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 98–106, Beijing, China. Coling 2010 Organizing Committee.
    Google ScholarLocate open access versionFindings
  • Thiago Castro Ferreira, Chris van der Lee, Emiel van Miltenburg, and Emiel Krahmer. 2019. Neural datato-text generation: A comparison between pipeline and end-to-end architectures. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 552–562, Stroudsburg, PA, USA. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Henry Elder, Jennifer Foster, James Barry, and Alexander OConnor. 2019. Designing a Symbolic Intermediate Representation for Neural Surface Realization. In Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation, pages 65–73, Stroudsburg, PA, USA. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Henry Elder and Chris Hokamp. 2018. Generating High-Quality Surface Realizations Using Data Augmentation and Factored Sequence Models. In Proceedings of the First Workshop on Multilingual Surface Realisation, pages 49–53, Stroudsburg, PA, USA. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching Machines to Read and Comprehend. In Advances in Neural Information Processing Systems 28, pages 1693–1701. Curran Associates, Inc.
    Google ScholarLocate open access versionFindings
  • Juraj Juraska, Panagiotis Karagiannis, Kevin Bowden, and Marilyn Walker. 2018. A Deep Ensemble Model with Slot Alignment for Sequence-toSequence Natural Language Generation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 152–162, Stroudsburg, PA, USA. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • David King and Michael White. 201The OSU Realizer for SRST 18: Neural Sequence-to-Sequence Inflection and Incremental Locality-Based Linearization. In Proceedings of the First Workshop on Multilingual Surface Realisation, 2009, pages 39–48, Stroudsburg, PA, USA. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander Rush. 2017. OpenNMT: Open-Source Toolkit for Neural Machine Translation. In Proceedings of ACL 2017, System Demonstrations, pages 67–72, Stroudsburg, PA, USA. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Ioannis Konstas, Srinivasan Iyer, Mark Yatskar, Yejin Choi, and Luke Zettlemoyer. 2017. Neural AMR: Sequence-to-Sequence Models for Parsing and Generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 146–157, Stroudsburg, PA, USA. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Percy Liang, Michael I. Jordan, and Dan Klein. 2009. Learning Semantic Correspondences with Less Supervision. Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, (August):91–99.
    Google ScholarLocate open access versionFindings
  • Christopher D Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J Bethard, and David McClosky. 2014. The {Stanford} {CoreNLP} Natural Language Processing Toolkit. In Association for Computational Linguistics (ACL) System Demonstrations, pages 55–60.
    Google ScholarLocate open access versionFindings
  • Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer Sentinel Mixture Models. In 5th International Conference on Learning Representations, {ICLR} 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
    Google ScholarLocate open access versionFindings
  • Simon Mille, Anja Belz, Bernd Bohnet, Yvette Graham, Emily Pitler, and Leo Wanner. 2018. The First Multilingual Surface Realisation Shared Task (SR’18): Overview and Evaluation Results. In Proceedings of the 1st Workshop on Multilingual Surface Realisation (MSR), 56th Annual Meeting of the Association for Computational Linguistics, pages 1– 10, Melbourne, Australia.
    Google ScholarLocate open access versionFindings
  • Simon Mille, Anja Belz, Bernd Bohnet, Yvette Graham, and Leo Wanner. 2019. The Second Multilingual Surface Realisation Shared Task (SR19): Overview and Evaluation Results. In Proceedings of the 2nd Workshop on Multilingual Surface Realisation (MSR 2019), Msr, pages 1–17, Stroudsburg, PA, USA. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Amit Moryossef, Yoav Goldberg, and Ido Dagan. 2019. Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation. In Proceedings of the 2019 Conference of the North, pages 2267– 2277, Stroudsburg, PA, USA. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Graham Neubig, Zi-Yi Dou, Junjie Hu, Paul Michel, Danish Pruthi, and Xinyi Wang. 2019. compare-mt: A Tool for Holistic Comparison of Language Generation Systems. In Meeting of the North American Chapter of the Association for Computational Linguistics (NAACL) Demo Track, Minneapolis, USA.
    Google ScholarLocate open access versionFindings
  • Kishore Papineni, Salim Roukos, Todd Ward, and WeiJing Zhu. 2002. BLEU: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL ’02, pages 311–318, Stroudsburg, PA, USA. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Ratish Puduppully, Yue Zhang, and Manish Shrivastava. 2016. Transition-Based Syntactic Linearization with Lookahead Features. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 488– 493, Stroudsburg, PA, USA. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Peng Qi, Timothy Dozat, Yuhao Zhang, and Christopher D Manning. 2018. Universal Dependency Parsing from Scratch. In Proceedings of the (CoNLL) 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 160–170, Brussels, Belgium. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get To The Point: Summarization with Pointer-Generator Networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073–1083, Stroudsburg, PA, USA. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Linfeng Song, Yue Zhang, Zhiguo Wang, and Daniel Gildea. 2018. A Graph-to-Sequence Model for AMR-to-Text Generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1616–1626, Stroudsburg, PA, USA. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), volume 2015, pages 1556–1566, Stroudsburg, PA, USA. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer Networks. In Advances in Neural Information Processing Systems 28, pages 2692–2700.
    Google ScholarLocate open access versionFindings
  • Xiang Yu, Agnieszka Falenska, Marina Haid, Ngoc Thang Vu, and Jonas Kuhn. 2019a. IMSurReal: IMS at the Surface Realization Shared Task
    Google ScholarFindings
  • 2019. In Proceedings of the 2nd Workshop on Multilingual Surface Realisation (MSR 2019), Msr, pages 50–58, Stroudsburg, PA, USA. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Xiang Yu, Agnieszka Falenska, Ngoc Thang Vu, and Jonas Kuhn. 2019b. Head-First Linearization with Tree-Structured Representation. In Proceedings of the 12th International Conference on Natural Language Generation, 2018, pages 279–289, Tokyo, Japan. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
Your rating :
0

 

Tags
Comments