Peng et al train a language model to predict a representation of the semantic frame, entities, and sentiment of the fifth sentence given the representations of the previous sentences, take the more likely fifth sentence
Toward Better Storylines with Sentence-Level Language Models
ACL, pp.7472-7478, (2020)
We propose a sentence-level language model which selects the next sentence in a story from a finite set of fluent alternatives. Since it does not need to model fluency, the sentence-level language model can focus on longer range dependencies, which are crucial for multi-sentence coherence. Rather than dealing with individual words, our ...更多
下载 PDF 全文
- Computer generation of stories and other kinds of creative writing is a challenging endeavor.
- It entangles two difficult tasks: the generation of fluent natural language and the generation of a coherent storyline.
- Rather than considering additional conditioning, the authors propose a model which takes as input several sentences of context and selects the best sentence within a large set of fluent candidate sentences.
- Given the embeddings of the previous sentences of the story, the model learns to predict a likely embedding of the sentence
- Computer generation of stories and other kinds of creative writing is a challenging endeavor
- The generation of coherent stories has recently been addressed with additional conditioning: Fan et al (2018) suggest conditioning on a story prompt, Clark et al (2018) propose collaboration between a generative model and a human writer, and Guan et al (2019) suggest attending to a commonsense graph relevant to the story plot
- Rather than considering additional conditioning, we propose a model which takes as input several sentences of context and selects the best sentence within a large set of fluent candidate sentences
- We propose a sentence-level language model: our model estimates P, the probability distribution for sentence st+1 given the t previous sentences, s1, . . . st
- We only evaluate the Transformer on the task of predicting the 9th sentence so that evaluation results are directly comparable to the residual multi-layer perceptron
- Peng et al (2017) train a language model to predict a representation of the semantic frame, entities, and sentiment of the fifth sentence given the representations of the previous sentences, take the more likely fifth sentence
- The authors propose a sentence-level language model: the model estimates P, the probability distribution for sentence st+1 given the t previous sentences, s1, . . . st.
- Alternative sentence representations were considered, including embeddings from the universal sentence encoder (Cer et al, 2018) and a weighted mean of the BERT embeddings using inverse document frequency weighting (Zhang et al, 2019).
- None of these alternatives improved the results
- Are not comparable to the unsupervised approach as they require training on the labeled validation set.
- The introduction of the CSLoss which considers only context sentences as candidates improves accuracy compared to only using a loss over all possible fifth sentences.
- Schenk and Chiarcos (2017) construct negative examples for their binary classification task by pairing contexts with random fifth sentences selected from the training set.
- The authors achieve higher accuracy without relying on a task-specific architecture
- Context: The author's family got up one morning while on vacation. The authors loaded the boat onto a trailer and drove to the beach.
- After loading up from the dock, the authors took off on the boat.
- Rank: 1 Top scored: (30.23) She couldn’t wait to use the new words.
- Context: Benjamin enjoyed going fishing with his grandfather as a kid.
- They would pick a new location to go to every summer.
- Benjamin liked seeing who would catch the biggest fish
- Even after his grandfather passed he continued the tradition.
- Rank: 2,281 Top ranked: (34.71) Greg grew to love golfing and is his favorite thing to do. (33.82) It was a tradition Tim continues with his own family. (33.63) Alex learned to be grateful of his family’s unique tradition. (33.40) Tom was sad that he would have to let his son down
- Table1: Accuracies (%) for the Story Cloze binary classification task. <a class="ref-link" id="cSchwartz_et+al_2017_a" href="#rSchwartz_et+al_2017_a">Schwartz et al (2017</a>) is a semi-supervised technique. GPT-2 refers to predicting the more likely ending according to the 355M parameter model, and GPT-2 finetuning was done on the ROC Stories train set
- Table2: Precision@10 and mean-reciprocal rank on the 2018 valid set when considering all 5th sentences in the train and valid sets (98k total) as candidate endings
- Table3: Precision@10 On Toronto Book Corpus for retrieving the correct next sentence (given the 8 previous sentences) when considering 10k or 100k distractor sentences, or all of the sentences from the same book as distractors
- Table4: Top-scoring sentences (using resMLP without CSLoss) among 98k possible endings when using prompts from the validation set. Two success and two failures cases are shown
- This research is based upon work supported in part by U.S DARPA KAIROS Program No FA875019-2-1004
- Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representation (ICLR).
- Yoshua Bengio, Rejean Ducharme, Pascal Vincent, and Christian Jauvin. 2003. A neural probabilistic language model. Journal of Machine Learning Research (JMLR), 3(Feb):1137–1155.
- Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. 2018. Universal sentence encoder. arXiv preprint arXiv:1803.11175.
- Khyathi Chandu, Eric Nyberg, and Alan W Black. 2019. Storyboarding of recipes: Grounded contextual generation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
- Snigdha Chaturvedi, Haoruo Peng, and Dan Roth. 2017. Story comprehension for predicting what happens next. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1603–1614.
- Elizabeth Clark, Anne Spencer Ross, Chenhao Tan, Yangfeng Ji, and Noah A Smith. 2018. Creative writing with a machine in the loop: Case studies on slogans and stories. In 23rd International Conference on Intelligent User Interfaces, pages 329–340. ACM.
- Yiming Cui, Wanxiang Che, Wei-Nan Zhang, Ting Liu, Shijin Wang, and Guoping Hu. 2019. Discriminative sentence modeling for story ending prediction. CoRR, abs/1912.09008.
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186.
- Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889–898, Melbourne, Australia. Association for Computational Linguistics.
- Angela Fan, Mike Lewis, and Yann N. Dauphin. 2019. Strategies for structuring story generation. In Association for Computational Linguistics, ACL, pages 2650–2660. Association for Computational Linguistics.
- Markus Freitag, Isaac Caswell, and Scott Roy. 2019. APE at scale and its implications on MT evaluation biases. In Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers), pages 34–44, Florence, Italy. Association for Computational Linguistics.
- Jian Guan, Yansen Wang, and Minlie Huang. 2019. Story ending generation with incremental encoding and commonsense knowledge. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI, pages 6473–6480.
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770– 778.
- Ryan Kiros, Yukun Zhu, Russ R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in neural information processing systems, pages 3294–3302.
- Mirella Lapata. 2003. Probabilistic text structuring: Experiments with sentence ordering. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics - Volume 1, ACL ’03, pages 545–552, Stroudsburg, PA, USA. Association for Computational Linguistics.
- Zhongyang Li, Xiao Ding, and Ting Liu. 2019. Story ending prediction by transferable BERT. Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, pages 1800–1806.
- Danyang Liu, Juntao Li, Meng-Hsuan Yu, Ziming Huang, Gongshen Liu, Dongyan Zhao, and Rui Yan. 2020. A character-centric neural model for automated story generation. In AAAI Conference on Artificial Intelligence.
- Nelson F Liu, Matt Gardner, Yonatan Belinkov, Matthew E Peters, and Noah A Smith. 2019. Linguistic knowledge and transferability of contextual representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1073–1094.
- Lajanugen Logeswaran and Honglak Lee. 2018. An efficient framework for learning sentence representations. In International Conference on Learning Representations.
- Lara J Martin, Prithviraj Ammanabrolu, Xinyu Wang, William Hancock, Shruti Singh, Brent Harrison, and Mark O Riedl. 2018. Event representations for automated story generation with deep neural nets. In Thirty-Second AAAI Conference on Artificial Intelligence.
- Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of the 26th International Conference on Neural Information Processing Systems.
- Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 839–849, San Diego, California. Association for Computational Linguistics.
- Haoruo Peng, Snigdha Chaturvedi, and Dan Roth. 2017. A joint model for semantic sequences: Frames, entities, sentiments. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 173–183.
- Alec Radford, Jeffrey Wu Wu, Rewon Child, David Luan Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
- Niko Schenk and Christian Chiarcos. 2017. Resourcelean modeling of coherence in commonsense stories. In Proceedings of the 2nd Workshop on Linking Models of Lexical, Sentential and Discourse-level Semantics, pages 68–73.
- Yingce Xia, Fei Tian, Lijun Wu, Jianxin Lin, Tao Qin, Nenghai Yu, and Tie-Yan Liu. 2017. Deliberation networks: Sequence generation beyond one-pass decoding. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 1784–1794. Curran Associates, Inc.
- Lili Yao, Nanyun Peng, Ralph M. Weischedel, Kevin Knight, Dongyan Zhao, and Rui Yan. 2019. Planand-write: Towards better automatic storytelling. In AAAI Conference on Artificial Intelligence.
- Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675.
- Mantong Zhou, Minlie Huang, and Xiaoyan Zhu. 2019. Story ending selection by finding hints from pairwise candidate endings. IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP), 27(4):719–729.
- Roy Schwartz, Maarten Sap, Ioannis Konstas, Leila Zilles, Yejin Choi, and Noah A Smith. 2017. Story cloze task: Uw nlp system. In Proceedings of the 2nd Workshop on Linking Models of Lexical, Sentential and Discourse-level Semantics, pages 52–55.
- Abigail See, Aneesh Pappu, Rohun Saxena, Akhila Yerukola, and Christopher D. Manning. 2019. Do massively pretrained language models make better storytellers? In Conference on Natural Language Learning CONLL.
- Kai Sun, Dian Yu, Dong Yu, and Claire Cardie. 2019. Improving machine reading comprehension with general reading strategies. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 2633–2643.
- Martin Sundermeyer, Ralf Schluter, and Hermann Ney. 2012. Lstm neural networks for language modeling. In Thirteenth annual conference of the international speech communication association.
- Wilson L Taylor. 1953. cloze procedure: A new tool for measuring readability. Journalism Bulletin, 30(4):415–433.
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008.