Neural Syntactic Preordering for Controlled Paraphrase Generation

ACL, pp. 238-252, 2020.

Cited by: 0|Bibtex|Views21
EI
Other Links: arxiv.org|dblp.uni-trier.de|academic.microsoft.com
Weibo:
We propose a two-step framework for paraphrase generation: construction of diverse syntactic guides in the form of target reorderings followed by actual paraphrase generation that respects these reorderings

Abstract:

Paraphrasing natural language sentences is a multifaceted process: it might involve replacing individual words or short phrases, local rearrangement of content, or high-level restructuring like topicalization or passivization. Past approaches struggle to cover this space of paraphrase possibilities in an interpretable manner. Our work, ...More
Introduction
Highlights
  • Paraphrase generation (McKeown, 1983; Barzilay and Lee, 2003) has seen a recent surge of interest, both with large-scale dataset collection and curation (Lan et al, 2017; Wieting and Gimpel, 2018) and with modeling advances such as deep generative models (Gupta et al, 2018; Li et al, 2019)
  • Baselines We compare our model against the Syntactically Controlled Paraphrase Network (SCPN) model proposed in prior work (Iyyer et al, 2018). It produces 10 distinct paraphrase outputs conditioned on a pre-enumerated list of syntactic templates. This approach has been shown to outperform other paraphrase approaches that condition on interpretable intermediate structures (Chen et al, 2019b)
  • We evaluate our proposed approach against 3 systems: a) Monotone reordering {1, 2, . . . , n}. b) Random permutation, by randomly permuting the children of each node as we traverse down the constituency parse tree. c) Ground Truth, using the pseudo-ground truth rearrangement between the source and ground-truth target sentence
  • The source order rewriting model’s oracle perplexity is close to that of the ground truth reordering’s perplexity, showing that the proposed approach is capable of generating a diverse set of rearrangements such that one of them often comes close to the target rearrangement
  • We propose a two-step framework for paraphrase generation: construction of diverse syntactic guides in the form of target reorderings followed by actual paraphrase generation that respects these reorderings
  • Our experiments show that this approach can be used to produce paraphrases that achieve a better quality-diversity trade-off compared to previous methods and strong baselines
Methods
  • The authors' method centers around a sequence-tosequence model which can generate a paraphrase roughly respecting a particular ordering of the input tokens.
  • This is a model P (y | x, r).
  • In Section 2.3, the authors outline the reordering approach, including the source order rewriting (SOW) model, which produces the set of reorderings appropriate for a given input sentence x during inference (x → R)
Results
  • Setup As the main goal is to evaluate the model’s ability to generate diverse paraphrases, the authors obtain a set of paraphrases and compare these to sets of paraphrases produced by other methods.
  • Baselines The authors compare the model against the Syntactically Controlled Paraphrase Network (SCPN) model proposed in prior work (Iyyer et al, 2018)
  • It produces 10 distinct paraphrase outputs conditioned on a pre-enumerated list of syntactic templates.
  • Given a budget of 10 reorderings, the authors want to understand how close the SOW model comes to covering the target ordering
  • The authors do this by evaluating the REAP model in terms of oracle perplexity and oracle BLEU over these 10 orderings.
  • The comparatively high performance of the ground truth reorderings demonstrates that the positional embeddings are effective at guiding the REAP model’s generation
Conclusion
  • The authors propose a two-step framework for paraphrase generation: construction of diverse syntactic guides in the form of target reorderings followed by actual paraphrase generation that respects these reorderings.
  • The authors' experiments show that this approach can be used to produce paraphrases that achieve a better quality-diversity trade-off compared to previous methods and strong baselines
Summary
  • Introduction:

    Paraphrase generation (McKeown, 1983; Barzilay and Lee, 2003) has seen a recent surge of interest, both with large-scale dataset collection and curation (Lan et al, 2017; Wieting and Gimpel, 2018) and with modeling advances such as deep generative models (Gupta et al, 2018; Li et al, 2019).
  • The game was won by the Clippers.
  • Transformer seq2seq VBD NP Clippers won
  • Methods:

    The authors' method centers around a sequence-tosequence model which can generate a paraphrase roughly respecting a particular ordering of the input tokens.
  • This is a model P (y | x, r).
  • In Section 2.3, the authors outline the reordering approach, including the source order rewriting (SOW) model, which produces the set of reorderings appropriate for a given input sentence x during inference (x → R)
  • Results:

    Setup As the main goal is to evaluate the model’s ability to generate diverse paraphrases, the authors obtain a set of paraphrases and compare these to sets of paraphrases produced by other methods.
  • Baselines The authors compare the model against the Syntactically Controlled Paraphrase Network (SCPN) model proposed in prior work (Iyyer et al, 2018)
  • It produces 10 distinct paraphrase outputs conditioned on a pre-enumerated list of syntactic templates.
  • Given a budget of 10 reorderings, the authors want to understand how close the SOW model comes to covering the target ordering
  • The authors do this by evaluating the REAP model in terms of oracle perplexity and oracle BLEU over these 10 orderings.
  • The comparatively high performance of the ground truth reorderings demonstrates that the positional embeddings are effective at guiding the REAP model’s generation
  • Conclusion:

    The authors propose a two-step framework for paraphrase generation: construction of diverse syntactic guides in the form of target reorderings followed by actual paraphrase generation that respects these reorderings.
  • The authors' experiments show that this approach can be used to produce paraphrases that achieve a better quality-diversity trade-off compared to previous methods and strong baselines
Tables
  • Table1: Quality and diversity metrics for the different models. Our proposed approach outperforms other diverse models (SCPN and diverse decoding) in terms of all the quality metrics. These models exhibit higher diversity, but with many more rejected paraphrases, indicating that these models more freely generate bad paraphrases
  • Table2: Examples of paraphrases generated by our system and the baseline SCPN model. Our model successfully rearranges the different structural components of the input sentence to obtain meaningful rearrangements. SCPN conforms to pre-enumerated templates that may not align with a given input
  • Table3: Examples of our model’s rearrangements applied to a given input sentence. Parse tree level indicates the rule subtree’s depth from the root node of the sentence. The REAP model’s final generation considers the rule reordering at the higher levels of the tree but ignores the rearrangement within the lower sub-tree
  • Table4: Human annotated quality across different models. The evaluation was done on a 3 point quality scale, 2 = grammatical paraphrase, 1 = ungrammatical paraphrase, 0 = not a paraphrase
  • Table5: Comparison of different source reordering strategies. Our proposed approach outperforms baseline monotone and random rearrangement strategies
  • Table6: Examples of aligned phrase pairs with exactly two sub-phrases abstracted out and replaced with constituent labels. These phrase pairs are used to train the SOW MODEL
  • Table7: Example of input (I), syntactic exemplar (E), and the reference output (O) from the evaluation test set of (Chen et al, 2019b)
  • Table8: Examples of paraphrases generated by our system and the baseline SCPN model. The outputs from our model successfully rearranges the different structural components of the input sentence to obtain meaningful rearrangements. SCPN on the other hand tends to conform to pre-specified templates that are often not aligned with a given input
  • Table9: Hyperparameters used in the implementation of the REAP model
  • Table10: Hyperparameters used in the implementation of the SOW model
Download tables as Excel
Related work
Funding
  • This work was partially supported by NSF Grant IIS-1814522, a gift from Arm, and an equipment grant from NVIDIA
Reference
  • Zhecheng An and Sicong Liu. 2019. Towards Diverse Paraphrase Generation Using Multi-Class Wasserstein GAN. arXiv preprint arXiv:1909.13827.
    Findings
  • Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural Machine Translation by Jointly Learning to Align and Translate. CoRR, abs/1409.0473.
    Findings
  • Regina Barzilay and Lillian Lee. 200Learning to paraphrase: an unsupervised approach using multiple-sequence alignment. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume 1, pages 16– 23. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Ondrej Bojar, Ondrej Dusek, Tom Kocmi, Jindrich Libovicky, Michal Novak, Martin Popel, Roman Sudarikov, and Dusan Varis. 2016. CzEng 1.6: enlarged Czech-English parallel corpus with processing tools Dockered. In International Conference on Text, Speech, and Dialogue, pages 231–238. Springer.
    Google ScholarLocate open access versionFindings
  • Kehai Chen, Rui Wang, Masao Utiyama, and Eiichiro Sumita. 2020. Explicit Reordering for Neural Machine Translation. arXiv preprint arXiv:2004.03818.
    Findings
  • Mingda Chen, Qingming Tang, Sam Wiseman, and Kevin Gimpel. 2019a. A Multi-Task Approach for Disentangling Syntax and Semantics in Sentence Representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2453–2464.
    Google ScholarLocate open access versionFindings
  • Mingda Chen, Qingming Tang, Sam Wiseman, and Kevin Gimpel. 2019b. Controllable Paraphrase Generation with a Syntactic Exemplar. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5972–5984, Florence, Italy. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Jaemin Cho, Minjoon Seo, and Hannaneh Hajishirzi. 2019. Mixture Content Selection for Diverse Sequence Generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLPIJCNLP), pages 3112–3122.
    Google ScholarLocate open access versionFindings
  • Michael Collins, Philipp Koehn, and Ivona Kucerova. 2005. Clause restructuring for statistical machine translation. In Proceedings of the 43rd annual meeting on association for computational linguistics, pages 531–540. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • John DeNero and Jakob Uszkoreit. 2011. Inducing sentence structure from parallel corpora for reordering. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 193–203. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186.
    Google ScholarLocate open access versionFindings
  • Chris Dyer and Philip Resnik. 2010. Context-free reordering, finite-state translation. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 858–866, Los Angeles, California. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Anthony Fader, Luke Zettlemoyer, and Oren Etzioni. 20Paraphrase-driven learning for open question answering. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1608–1618.
    Google ScholarLocate open access versionFindings
  • Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical Neural Story Generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889–898.
    Google ScholarLocate open access versionFindings
  • Michel Galley, Mark Hopkins, Kevin Knight, and Daniel Marcu. 2004. What’s in a translation rule? In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pages 273–280, Boston, Massachusetts, USA. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Dmitriy Genzel. 2010. Automatically learning sourceside reordering rules for large scale machine translation. In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 376–384.
    Google ScholarLocate open access versionFindings
  • Ankush Gupta, Arvind Agarwal, Prawaan Singh, and Piyush Rai. 2018. A deep generative framework for paraphrase generation. In Thirty-Second AAAI Conference on Artificial Intelligence.
    Google ScholarLocate open access versionFindings
  • Sadid A Hasan, Kathy Lee, Vivek Datla, Ashequl Qadir, Joey Liu, Oladimeji Farri, et al. 2016. Neural Paraphrase Generation with Stacked Residual LSTM Networks. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2923–2934.
    Google ScholarLocate open access versionFindings
  • J. Edward Hu, Rachel Rudinger, Matt Post, and Benjamin Van Durme. 20ParaBank: Monolingual
    Google ScholarFindings
  • Bitext Generation and Sentential Paraphrasing via Lexically-constrained Neural Machine Translation. In Proceedings of AAAI.
    Google ScholarLocate open access versionFindings
  • Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. Adversarial Example Generation with Syntactically Controlled Paraphrase Networks. In Proceedings of NAACL-HLT, pages 1875– 1885.
    Google ScholarLocate open access versionFindings
  • Maxim Khalilov and Khalil Simaan. 2011. Contextsensitive syntactic source-reordering by statistical transduction. In Proceedings of 5th International Joint Conference on Natural Language Processing, pages 38–46.
    Google ScholarLocate open access versionFindings
  • Stanley Kok and Chris Brockett. 2010. Hitting the right paraphrases in good time. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 145–153. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Ashutosh Kumar, Satwik Bhattamishra, Manik Bhandari, and Partha Talukdar. 2019. Submodular Optimization-based Diverse Paraphrasing and its Effectiveness in Data Augmentation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3609–3619.
    Google ScholarLocate open access versionFindings
  • Wuwei Lan, Siyu Qiu, Hua He, and Wei Xu. 2017. A Continuously Growing Dataset of Sentential Paraphrases. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1224–1234, Copenhagen, Denmark. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Uri Lerner and Slav Petrov. 2013. Source-side classifier preordering for machine translation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 513–523.
    Google ScholarLocate open access versionFindings
  • Zichao Li, Xin Jiang, Lifeng Shang, and Hang Li. 2018. Paraphrase Generation with Deep Reinforcement Learning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3865–3878, Brussels, Belgium. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Zichao Li, Xin Jiang, Lifeng Shang, and Qun Liu. 2019. Decomposable Neural Paraphrase Generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3403–3414, Florence, Italy. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Kathleen R McKeown. 1983. Paraphrasing questions using given and new information. Computational Linguistics, 9(1):1–10.
    Google ScholarLocate open access versionFindings
  • Sunghyun Park, Seung-won Hwang, Fuxiang Chen, Jaegul Choo, Jung-Woo Ha, Sunghun Kim, and Jinyeong Yim. 2019. Paraphrase Diversification Using Counterfactual Debiasing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6883–6891.
    Google ScholarLocate open access versionFindings
  • Ellie Pavlick, Pushpendre Rastogi, Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2015. PPDB 2.0: Better paraphrase ranking, finegrained entailment relations, word embeddings, and style classification. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 425–430, Beijing, China. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of NAACL-HLT, pages 2227–2237.
    Google ScholarLocate open access versionFindings
  • Shrimai Prabhumoye, Yulia Tsvetkov, Ruslan Salakhutdinov, and Alan W Black. 2018. Style Transfer Through Back-Translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 866–876.
    Google ScholarLocate open access versionFindings
  • Lihua Qian, Lin Qiu, Weinan Zhang, Xin Jiang, and Yong Yu. 2019. Exploring Diverse Expressions for Paraphrase Generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3164–3173.
    Google ScholarLocate open access versionFindings
  • Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get To The Point: Summarization with Pointer-Generator Networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073–1083.
    Google ScholarLocate open access versionFindings
  • Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural Machine Translation of Rare Words with Subword Units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725.
    Google ScholarLocate open access versionFindings
  • Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2017. Style transfer from non-parallel text by cross-alignment. In Advances in neural information processing systems, pages 6830–6841.
    Google ScholarLocate open access versionFindings
  • Roy Tromble and Jason Eisner. 2009. Learning linear ordering problems for better translation. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 2-Volume 2, pages 1007–1016. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008.
    Google ScholarLocate open access versionFindings
  • Ashwin K Vijayakumar, Michael Cogswell, Ramprasaath R Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. 2018. Diverse beam search for improved description of complex scenes. In Thirty-Second AAAI Conference on Artificial Intelligence.
    Google ScholarLocate open access versionFindings
  • Chao Wang, Michael Collins, and Philipp Koehn. 2007. Chinese syntactic reordering for statistical machine translation. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 737–745.
    Google ScholarLocate open access versionFindings
  • Di Wang, Nebojsa Jojic, Chris Brockett, and Eric Nyberg. 2017. Steering Output Style and Topic in Neural Response Generation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2140–2150.
    Google ScholarLocate open access versionFindings
  • Su Wang, Rahul Gupta, Nancy Chang, and Jason Baldridge. 2019. A task in a suit and a tie: paraphrase generation with semantic augmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7176–7183.
    Google ScholarLocate open access versionFindings
  • John Wieting and Kevin Gimpel. 2018. ParaNMT50M: Pushing the limits of paraphrastic sentence embeddings with millions of machine translations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 451–462, Melbourne, Australia. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • John Wieting, Jonathan Mallinson, and Kevin Gimpel. 2017. Learning Paraphrastic Sentence Embeddings from Back-Translated Bitext. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 274–285.
    Google ScholarLocate open access versionFindings
  • Dekai Wu. 1997. Stochastic inversion transduction grammars and bilingual parsing of parallel corpora. Computational linguistics, 23(3):377–403.
    Google ScholarLocate open access versionFindings
  • Fei Xia and Michael McCord. 2004. Improving a statistical MT system with automatically learned rewrite patterns. In Proceedings of the 20th international conference on Computational Linguistics, page 508. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Qiongkai Xu, Juyan Zhang, Lizhen Qu, Lexing Xie, and Richard Nock. 2018. D-PAGE: Diverse Paraphrase Generation. CoRR, abs/1808.04364.
    Findings
  • Adams Wei Yu, David Dohan, Quoc Le, Thang Luong, Rui Zhao, and Kai Chen. 2018. QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension. In International Conference on Learning Representations.
    Google ScholarLocate open access versionFindings
  • Shiyue Zhang and Mohit Bansal. 2019. Addressing Semantic Drift in Question Generation for SemiSupervised Question Answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2495–2509, Hong Kong, China. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating Text Generation with BERT. In International Conference on Learning Representations.
    Google ScholarLocate open access versionFindings
Full Text
Your rating :
0

 

Tags
Comments