AI帮你理解科学

AI 生成解读视频

AI抽取解析论文重点内容自动生成视频


pub
生成解读视频

AI 溯源

AI解析本论文相关学术脉络


Master Reading Tree
生成 溯源树

AI 精读

AI抽取本论文的概要总结


微博一下
Peng et al train a language model to predict a representation of the semantic frame, entities, and sentiment of the fifth sentence given the representations of the previous sentences, take the more likely fifth sentence

Toward Better Storylines with Sentence-Level Language Models

ACL, pp.7472-7478, (2020)

被引用4|浏览119
EI
下载 PDF 全文
引用
微博一下

摘要

We propose a sentence-level language model which selects the next sentence in a story from a finite set of fluent alternatives. Since it does not need to model fluency, the sentence-level language model can focus on longer range dependencies, which are crucial for multi-sentence coherence. Rather than dealing with individual words, our ...更多
简介
  • Computer generation of stories and other kinds of creative writing is a challenging endeavor.
  • It entangles two difficult tasks: the generation of fluent natural language and the generation of a coherent storyline.
  • Rather than considering additional conditioning, the authors propose a model which takes as input several sentences of context and selects the best sentence within a large set of fluent candidate sentences.
  • Given the embeddings of the previous sentences of the story, the model learns to predict a likely embedding of the sentence
重点内容
  • Computer generation of stories and other kinds of creative writing is a challenging endeavor
  • The generation of coherent stories has recently been addressed with additional conditioning: Fan et al (2018) suggest conditioning on a story prompt, Clark et al (2018) propose collaboration between a generative model and a human writer, and Guan et al (2019) suggest attending to a commonsense graph relevant to the story plot
  • Rather than considering additional conditioning, we propose a model which takes as input several sentences of context and selects the best sentence within a large set of fluent candidate sentences
  • We propose a sentence-level language model: our model estimates P, the probability distribution for sentence st+1 given the t previous sentences, s1, . . . st
  • We only evaluate the Transformer on the task of predicting the 9th sentence so that evaluation results are directly comparable to the residual multi-layer perceptron
  • Peng et al (2017) train a language model to predict a representation of the semantic frame, entities, and sentiment of the fifth sentence given the representations of the previous sentences, take the more likely fifth sentence
方法
  • The authors propose a sentence-level language model: the model estimates P, the probability distribution for sentence st+1 given the t previous sentences, s1, . . . st.
  • Alternative sentence representations were considered, including embeddings from the universal sentence encoder (Cer et al, 2018) and a weighted mean of the BERT embeddings using inverse document frequency weighting (Zhang et al, 2019).
  • None of these alternatives improved the results
结果
  • Are not comparable to the unsupervised approach as they require training on the labeled validation set.
  • The introduction of the CSLoss which considers only context sentences as candidates improves accuracy compared to only using a loss over all possible fifth sentences.
  • Schenk and Chiarcos (2017) construct negative examples for their binary classification task by pairing contexts with random fifth sentences selected from the training set.
  • The authors achieve higher accuracy without relying on a task-specific architecture
结论
  • Context: The author's family got up one morning while on vacation. The authors loaded the boat onto a trailer and drove to the beach.
  • After loading up from the dock, the authors took off on the boat.
  • Rank: 1 Top scored: (30.23) She couldn’t wait to use the new words.
  • Context: Benjamin enjoyed going fishing with his grandfather as a kid.
  • They would pick a new location to go to every summer.
  • Benjamin liked seeing who would catch the biggest fish
  • Even after his grandfather passed he continued the tradition.
  • Rank: 2,281 Top ranked: (34.71) Greg grew to love golfing and is his favorite thing to do. (33.82) It was a tradition Tim continues with his own family. (33.63) Alex learned to be grateful of his family’s unique tradition. (33.40) Tom was sad that he would have to let his son down
表格
  • Table1: Accuracies (%) for the Story Cloze binary classification task. <a class="ref-link" id="cSchwartz_et+al_2017_a" href="#rSchwartz_et+al_2017_a">Schwartz et al (2017</a>) is a semi-supervised technique. GPT-2 refers to predicting the more likely ending according to the 355M parameter model, and GPT-2 finetuning was done on the ROC Stories train set
  • Table2: Precision@10 and mean-reciprocal rank on the 2018 valid set when considering all 5th sentences in the train and valid sets (98k total) as candidate endings
  • Table3: Precision@10 On Toronto Book Corpus for retrieving the correct next sentence (given the 8 previous sentences) when considering 10k or 100k distractor sentences, or all of the sentences from the same book as distractors
  • Table4: Top-scoring sentences (using resMLP without CSLoss) among 98k possible endings when using prompts from the validation set. Two success and two failures cases are shown
Download tables as Excel
基金
  • This research is based upon work supported in part by U.S DARPA KAIROS Program No FA875019-2-1004
引用论文
  • Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representation (ICLR).
    Google ScholarLocate open access versionFindings
  • Yoshua Bengio, Rejean Ducharme, Pascal Vincent, and Christian Jauvin. 2003. A neural probabilistic language model. Journal of Machine Learning Research (JMLR), 3(Feb):1137–1155.
    Google ScholarLocate open access versionFindings
  • Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. 2018. Universal sentence encoder. arXiv preprint arXiv:1803.11175.
    Findings
  • Khyathi Chandu, Eric Nyberg, and Alan W Black. 2019. Storyboarding of recipes: Grounded contextual generation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
    Google ScholarLocate open access versionFindings
  • Snigdha Chaturvedi, Haoruo Peng, and Dan Roth. 2017. Story comprehension for predicting what happens next. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1603–1614.
    Google ScholarLocate open access versionFindings
  • Elizabeth Clark, Anne Spencer Ross, Chenhao Tan, Yangfeng Ji, and Noah A Smith. 2018. Creative writing with a machine in the loop: Case studies on slogans and stories. In 23rd International Conference on Intelligent User Interfaces, pages 329–340. ACM.
    Google ScholarLocate open access versionFindings
  • Yiming Cui, Wanxiang Che, Wei-Nan Zhang, Ting Liu, Shijin Wang, and Guoping Hu. 2019. Discriminative sentence modeling for story ending prediction. CoRR, abs/1912.09008.
    Findings
  • Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186.
    Google ScholarLocate open access versionFindings
  • Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889–898, Melbourne, Australia. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Angela Fan, Mike Lewis, and Yann N. Dauphin. 2019. Strategies for structuring story generation. In Association for Computational Linguistics, ACL, pages 2650–2660. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Markus Freitag, Isaac Caswell, and Scott Roy. 2019. APE at scale and its implications on MT evaluation biases. In Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers), pages 34–44, Florence, Italy. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Jian Guan, Yansen Wang, and Minlie Huang. 2019. Story ending generation with incremental encoding and commonsense knowledge. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI, pages 6473–6480.
    Google ScholarLocate open access versionFindings
  • Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770– 778.
    Google ScholarLocate open access versionFindings
  • Ryan Kiros, Yukun Zhu, Russ R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in neural information processing systems, pages 3294–3302.
    Google ScholarLocate open access versionFindings
  • Mirella Lapata. 2003. Probabilistic text structuring: Experiments with sentence ordering. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics - Volume 1, ACL ’03, pages 545–552, Stroudsburg, PA, USA. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Zhongyang Li, Xiao Ding, and Ting Liu. 2019. Story ending prediction by transferable BERT. Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, pages 1800–1806.
    Google ScholarLocate open access versionFindings
  • Danyang Liu, Juntao Li, Meng-Hsuan Yu, Ziming Huang, Gongshen Liu, Dongyan Zhao, and Rui Yan. 2020. A character-centric neural model for automated story generation. In AAAI Conference on Artificial Intelligence.
    Google ScholarLocate open access versionFindings
  • Nelson F Liu, Matt Gardner, Yonatan Belinkov, Matthew E Peters, and Noah A Smith. 2019. Linguistic knowledge and transferability of contextual representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1073–1094.
    Google ScholarLocate open access versionFindings
  • Lajanugen Logeswaran and Honglak Lee. 2018. An efficient framework for learning sentence representations. In International Conference on Learning Representations.
    Google ScholarLocate open access versionFindings
  • Lara J Martin, Prithviraj Ammanabrolu, Xinyu Wang, William Hancock, Shruti Singh, Brent Harrison, and Mark O Riedl. 2018. Event representations for automated story generation with deep neural nets. In Thirty-Second AAAI Conference on Artificial Intelligence.
    Google ScholarLocate open access versionFindings
  • Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of the 26th International Conference on Neural Information Processing Systems.
    Google ScholarLocate open access versionFindings
  • Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 839–849, San Diego, California. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Haoruo Peng, Snigdha Chaturvedi, and Dan Roth. 2017. A joint model for semantic sequences: Frames, entities, sentiments. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 173–183.
    Google ScholarLocate open access versionFindings
  • Alec Radford, Jeffrey Wu Wu, Rewon Child, David Luan Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
    Google ScholarFindings
  • Niko Schenk and Christian Chiarcos. 2017. Resourcelean modeling of coherence in commonsense stories. In Proceedings of the 2nd Workshop on Linking Models of Lexical, Sentential and Discourse-level Semantics, pages 68–73.
    Google ScholarLocate open access versionFindings
  • Yingce Xia, Fei Tian, Lijun Wu, Jianxin Lin, Tao Qin, Nenghai Yu, and Tie-Yan Liu. 2017. Deliberation networks: Sequence generation beyond one-pass decoding. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 1784–1794. Curran Associates, Inc.
    Google ScholarLocate open access versionFindings
  • Lili Yao, Nanyun Peng, Ralph M. Weischedel, Kevin Knight, Dongyan Zhao, and Rui Yan. 2019. Planand-write: Towards better automatic storytelling. In AAAI Conference on Artificial Intelligence.
    Google ScholarLocate open access versionFindings
  • Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675.
    Findings
  • Mantong Zhou, Minlie Huang, and Xiaoyan Zhu. 2019. Story ending selection by finding hints from pairwise candidate endings. IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP), 27(4):719–729.
    Google ScholarLocate open access versionFindings
  • Roy Schwartz, Maarten Sap, Ioannis Konstas, Leila Zilles, Yejin Choi, and Noah A Smith. 2017. Story cloze task: Uw nlp system. In Proceedings of the 2nd Workshop on Linking Models of Lexical, Sentential and Discourse-level Semantics, pages 52–55.
    Google ScholarLocate open access versionFindings
  • Abigail See, Aneesh Pappu, Rohun Saxena, Akhila Yerukola, and Christopher D. Manning. 2019. Do massively pretrained language models make better storytellers? In Conference on Natural Language Learning CONLL.
    Google ScholarLocate open access versionFindings
  • Kai Sun, Dian Yu, Dong Yu, and Claire Cardie. 2019. Improving machine reading comprehension with general reading strategies. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 2633–2643.
    Google ScholarLocate open access versionFindings
  • Martin Sundermeyer, Ralf Schluter, and Hermann Ney. 2012. Lstm neural networks for language modeling. In Thirteenth annual conference of the international speech communication association.
    Google ScholarLocate open access versionFindings
  • Wilson L Taylor. 1953. cloze procedure: A new tool for measuring readability. Journalism Bulletin, 30(4):415–433.
    Google ScholarLocate open access versionFindings
  • Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008.
    Google ScholarLocate open access versionFindings
您的评分 :
0

 

标签
评论
数据免责声明
页面数据均来自互联网公开来源、合作出版商和通过AI技术自动分析结果,我们不对页面数据的有效性、准确性、正确性、可靠性、完整性和及时性做出任何承诺和保证。若有疑问,可以通过电子邮件方式联系我们:report@aminer.cn
小科