Topic-Oriented Spoken Dialogue Summarization for Customer Service with Saliency-Aware Topic Modeling
Weibo:
Abstract:
In a customer service system, dialogue summarization can boost service efficiency by automatically creating summaries for long spoken dialogues in which customers and agents try to address issues about specific topics. In this work, we focus on topic-oriented dialogue summarization, which generates highly abstractive summaries that pres...More
Introduction
- In an active customer service system, massive dialogues conveying important information between customers and agents are generated in real time.
- With this background, how to efficiently consume dialogue information becomes a nontrivial issue.
- To better understand customers’ and agents’ intentions, in this work the authors focus on the topic-oriented dialogue summarization, which aims to extract semantically consistent topics and generate highly abstractive summaries to maintain the main ideas from dialogues
Highlights
- In an active customer service system, massive dialogues conveying important information between customers and agents are generated in real time
- To better understand customers’ and agents’ intentions, in this work we focus on the topic-oriented dialogue summarization, which aims to extract semantically consistent topics and generate highly abstractive summaries to maintain the main ideas from dialogues
- We propose a novel two-stage neural model jointly with an enhanced topic modeling approach for spoken dialogue summarization
- To better distinguish the underlying informative content from abundant common semantics and dialogue noise, we introduce a saliency-aware topic model (SATM), where topics are split into two groups: informative topics and other topics
- We propose a topic-augmented two-stage summarizer with a multi-role topic modeling mechanism for customer service dialogues, which can generate highly abstractive summaries that highlight role-specific information
- We introduce a novel training regime for topic modeling that directly learns word-saliency correspondences to alleviate the influence of uninformative content
Methods
- Seq2seq+Att PGNet TRF CopyTRF HiBERT* BERT+TRF* FastRL* TDS+NTM TDS+SATM TDS+NTM* TDS+SATM* RG-1 RG-2 RG-L BLEU.
- 6.89 9.77 9.87 9.78 9.89 10.19 10.40 9.87 10.34 10.77 11.24 training (90%), development (5%), and test (5%) set.
- Table 1 shows the detailed statistics of the collected dataset
Conclusion
- To better understand the influence of role information, saliency-aware topic modeling, and the topic-informed at- SATM Informative Topic SATM Other Topic
NTM General Topic
T1: deliver, time, order, address, modify, cancel, ship, return, refund, receive
T2: feedback, problem, submit, suggest, apply, complain, seller, quality, product, slow
T3: buy, account, pay, bind, phone, number, modify, check, username, message
T1: please, wait, service, sorry, really, thanks, bother, mean, find, welcome
T2: send, call, record, again, later, check, help, keep, contact, reply
T1: thanks, later, sorry, please, really, phone, feedback, deliver, number, returnConclusion and Future Work
In this paper, the authors propose a topic-augmented two-stage summarizer with a multi-role topic modeling mechanism for customer service dialogues, which can generate highly abstractive summaries that highlight role-specific information. - To better understand the influence of role information, saliency-aware topic modeling, and the topic-informed at- SATM Informative Topic SATM Other Topic.
- T3: buy, account, pay, bind, phone, number, modify, check, username, message.
- T1: please, wait, service, sorry, really, thanks, bother, mean, find, welcome.
- T1: thanks, later, sorry, please, really, phone, feedback, deliver, number, returnConclusion and Future Work.
- The authors propose a topic-augmented two-stage summarizer with a multi-role topic modeling mechanism for customer service dialogues, which can generate highly abstractive summaries that highlight role-specific information.
- Future directions may be the exploration of template-guided abstractive methods to make summaries more standardized and easier for reporting
Summary
Introduction:
In an active customer service system, massive dialogues conveying important information between customers and agents are generated in real time.- With this background, how to efficiently consume dialogue information becomes a nontrivial issue.
- To better understand customers’ and agents’ intentions, in this work the authors focus on the topic-oriented dialogue summarization, which aims to extract semantically consistent topics and generate highly abstractive summaries to maintain the main ideas from dialogues
Methods:
Seq2seq+Att PGNet TRF CopyTRF HiBERT* BERT+TRF* FastRL* TDS+NTM TDS+SATM TDS+NTM* TDS+SATM* RG-1 RG-2 RG-L BLEU.- 6.89 9.77 9.87 9.78 9.89 10.19 10.40 9.87 10.34 10.77 11.24 training (90%), development (5%), and test (5%) set.
- Table 1 shows the detailed statistics of the collected dataset
Conclusion:
To better understand the influence of role information, saliency-aware topic modeling, and the topic-informed at- SATM Informative Topic SATM Other Topic
NTM General Topic
T1: deliver, time, order, address, modify, cancel, ship, return, refund, receive
T2: feedback, problem, submit, suggest, apply, complain, seller, quality, product, slow
T3: buy, account, pay, bind, phone, number, modify, check, username, message
T1: please, wait, service, sorry, really, thanks, bother, mean, find, welcome
T2: send, call, record, again, later, check, help, keep, contact, reply
T1: thanks, later, sorry, please, really, phone, feedback, deliver, number, returnConclusion and Future Work
In this paper, the authors propose a topic-augmented two-stage summarizer with a multi-role topic modeling mechanism for customer service dialogues, which can generate highly abstractive summaries that highlight role-specific information.- To better understand the influence of role information, saliency-aware topic modeling, and the topic-informed at- SATM Informative Topic SATM Other Topic.
- T3: buy, account, pay, bind, phone, number, modify, check, username, message.
- T1: please, wait, service, sorry, really, thanks, bother, mean, find, welcome.
- T1: thanks, later, sorry, please, really, phone, feedback, deliver, number, returnConclusion and Future Work.
- The authors propose a topic-augmented two-stage summarizer with a multi-role topic modeling mechanism for customer service dialogues, which can generate highly abstractive summaries that highlight role-specific information.
- Future directions may be the exploration of template-guided abstractive methods to make summaries more standardized and easier for reporting
Tables
- Table1: Statistics of the customer service dataset
- Table2: Results of automatic metrics on the customer service dataset. RG-(1,2,L) represents the F1 score of ROUGE(1,2,L). TRF denotes the Transformer. Methods marked with * utilize BERT as the word-level encoder
- Table3: Human evaluation with system ranking results
- Table4: Ablation study of TDS+SATM with different kinds of topic modeling. Agent and Cust. represent topic modeling on agent utterances and customer utterances, respectively
- Table5: Top-10 words of example topics in different topic groups learned by joint training of TDS+Topic Model
Related work
- Dialogue Summarization
Dialogue summarization is a challenging task and has been widely explored in various scenarios. Previous works generally focus on summarizing dialogues by stringing key points to maintain an integral dialogue flow: Mehdad et al (2013) and Shang et al (2018) first group utterances that share similar semantics by community detection, and then generate a summary sentence for each utterance group. Liu et al (2019a) propose a hierarchical model to produce key point sequences and generate summaries at the same time for customer service dialogues. Duan et al (2019b) train the assignment of utterances to the corresponding controversy focuses to summarize court debate dialogues. Several works (Zechner 2001; Xie et al 2008; Oya et al 2014; Liu et al 2019b; Li et al 2019a) split dialogues into multiple segments by means of topic segmentation when conducting summarization. Different from above works, Pan et al (2018) first attempt to generate a highly abstractive summary for the entire dialogue, which produces a concise event/object description with a Transformer-based approach. By contrast, in this work, dialogue summaries generally highlight role-specific content, which requires the system to further focus on the
Funding
- This work was partially funded by China National Key R&D Program (No 2018YFC0831105), National Natural Science Foundation of China (No 61751201, 62076069, 61976056), Shanghai Municipal Science and Technology Major Project (No.2018SHZDZX01), Science and Technology Commission of Shanghai Municipality Grant (No.18DZ1201000, 17JC1420200)
- This work was supported by Alibaba Group through Alibaba Innovative Research Program
Study subjects and analysis
volunteers: 3
Hence, we further randomly sample 100 examples in the test set for human evaluation. We follow Narayan at al. (2018) to design the experiment, in which three volunteers are invited to compare summaries produced from PGNet, BERT+TRF, HiBERT, FastRL, our proposed TDS+SATM, and the gold sum-. TDS+SATM (w/o) Cust. (w/o) Agent (w/o) Agent & Cust
volunteers: 3
Each volunteer is presented with a dialogue and two summaries produced from two out of six systems and is asked to decide which summary is better in order of two dimensions: informativeness (which summary captures more important information in the dialogue?) and fluency (which summary is more fluent and well-formed?). We collect judgments from three volunteers for each comparison with the order of dialogues and summaries randomized. Table 3 gives the system ranking results of human evaluation
Reference
- Blei, D. M.; Ng, A. Y.; and Jordan, M. I. 2003. Latent dirichlet allocation. Journal of machine Learning research 3(Jan): 993–1022.
- Chen, Y.-C.; and Bansal, M. 2018. Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 675–686.
- Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4171–4186.
- Duan, X.; Yu, H.; Yin, M.; Zhang, M.; Luo, W.; and Zhang, Y. 2019a. Contrastive Attention Mechanism for Abstractive Sentence Summarization. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 3035–3044.
- Duan, X.; Zhang, Y.; Yuan, L.; Zhou, X.; Liu, X.; Wang, T.; Wang, R.; Zhang, Q.; Sun, C.; and Wu, F. 2019b. Legal Summarization for Multi-role Debate Dialogue via Controversy Focus Mining and Multi-task Learning. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, 1361–1370.
- Fu, X.; Wang, J.; Zhang, J.; Wei, J.; and Yang, Z. 2020. Document Summarization with VHTM: Variational Hierarchical Topic-Aware Mechanism. In AAAI, 7740–7747.
- Gillick, D.; Riedhammer, K.; Favre, B.; and Hakkani-Tur, D. 2009. A global optimization framework for meeting summarization. In 2009 IEEE International Conference on Acoustics, Speech and Signal Processing, 4769–4772. IEEE.
- Higashinaka, R.; Minami, Y.; Nishikawa, H.; Dohsaka, K.; Meguro, T.; Takahashi, S.; and Kikui, G. 2010. Learning to model domain-specific utterance sequences for extractive summarization of contact center dialogues. In Coling 2010: Posters, 400–408.
- Hofmann, T. 199Probabilistic latent semantic indexing. In Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, 50–57.
- Kingma, D. P.; and Welling, M. 2014. Auto-Encoding Variational Bayes. CoRR abs/1312.6114.
- Li, M.; Zhang, L.; Ji, H.; and Radke, R. J. 2019a. Keep meeting summaries on topic: Abstractive multi-modal meeting summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2190– 2196.
- Li, X.; Wang, Y.; Zhang, A.; Li, C.; Chi, J.; and Ouyang, J. 2018. Filtering out the noise in short text topic modeling. Information Sciences 456: 83–96.
- Li, X.; Zhang, J.; Ouyang, J.; et al. 2019b. Dirichlet Multinomial Mixture with Variational Manifold Regularization: Topic Modeling over Short Texts. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, 7884–7891.
- Lin, C.-Y. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, 74–81.
- Liu, C.; Wang, P.; Xu, J.; Li, Z.; and Ye, J. 2019a. Automatic Dialogue Summary Generation for Customer Service. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining.
- Liu, Y.; and Lapata, M. 2019. Text Summarization with Pretrained Encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 3721–3731.
- Liu, Z.; Ng, A.; Lee, S.; Aw, A. T.; and Chen, N. F. 2019b. Topic-aware pointer-generator networks for summarizing spoken conversations. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 814– 821. IEEE.
- Maaten, L. v. d.; and Hinton, G. 2008. Visualizing data using t-SNE. Journal of machine learning research 9(Nov): 2579– 2605.
- Mehdad, Y.; Carenini, G.; Tompa, F.; and Ng, R. 2013. Abstractive meeting summarization with entailment and fusion. In Proceedings of the 14th European Workshop on Natural Language Generation, 136–146.
- Miao, Y.; Grefenstette, E.; Blunsom, P.; et al. 2017. Discovering discrete latent topics with neural variational inference. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, 2410–2419.
- Nallapati, R.; Zhai, F.; Zhou, B.; et al. 2017. Summarunner: A recurrent neural network based sequence model for extractive summarization of documents. In Thirty-First AAAI Conference on Artificial Intelligence.
- Nallapati, R.; Zhou, B.; Gulcehre, C.; and Xiang, B. 2016. Abstractive Text Summarization using Sequence-tosequence RNNs and Beyond. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, 280–290.
- Narayan, S.; Cohen, S. B.; Lapata, M.; et al. 2018. Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 1797–1807.
- Oya, T.; Mehdad, Y.; Carenini, G.; and Ng, R. 2014. A template-based abstractive meeting summarization: Leveraging summary and source text relationships. In Proceedings of the 8th International Natural Language Generation Conference (INLG), 45–53.
- Pan, H.; Zhou, J.; Zhao, Z.; Liu, Y.; Cai, D.; and Yang, M. 2018. Dial2desc: end-to-end dialogue description generation. arXiv preprint arXiv:1811.00185.
- Papineni, K.; Roukos, S.; Ward, T.; and Zhu, W.-J. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, 311–318.
- Rambow, O.; Shrestha, L.; Chen, J.; and Lauridsen, C. 2004. Summarizing email threads. In Proceedings of HLT-NAACL 2004: Short Papers, 105–108.
- See, A.; Liu, P. J.; Manning, C. D.; et al. 2017. Get To The Point: Summarization with Pointer-Generator Networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, 1073–1083.
- Shang, G.; Ding, W.; Zhang, Z.; Tixier, A.; Meladianos, P.; Vazirgiannis, M.; and Lorre, J.-P. 2018. Unsupervised Abstractive Meeting Summarization with Multi-Sentence Compression and Budgeted Submodular Maximization. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 664–674.
- Sood, A.; Mohamed, T. P.; Varma, V.; et al. 2013. Topicfocused summarization of chat conversations. In European Conference on Information Retrieval, 800–803. Springer.
- Tixier, A.; Meladianos, P.; Vazirgiannis, M.; et al. 2017. Combining graph degeneracy and submodularity for unsupervised extractive summarization. In Proceedings of the workshop on new frontiers in summarization, 48–58.
- Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017. Attention is all you need. In Advances in neural information processing systems, 5998–6008.
- Vinyals, O.; Fortunato, M.; Jaitly, N.; et al. 2015. Pointer networks. In Advances in neural information processing systems, 2692–2700.
- Wang, J.; Wang, J.; Sun, C.; Li, S.; Liu, X.; Si, L.; Zhang, M.; and Zhou, G. 2020. Sentiment Classification in Customer Service Dialogue with Topic-Aware Multi-Task Learning. In Proceedings of the AAAI Conference on Artificial Intelligence.
- Wang, L.; and Cardie, C. 2012. Unsupervised topic modeling approaches to decision summarization in spoken meetings. In Proceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue, 40–49.
- Wang, L.; Yao, J.; Tao, Y.; Zhong, L.; Liu, W.; and Du, Q. 2018. A reinforced topic-aware convolutional sequence-tosequence model for abstractive text summarization. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, 4453–4460.
- Williams, R. J. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning 8(3-4): 229–256.
- Xie, S.; Liu, Y.; Lin, H.; et al. 2008. Evaluating the effectiveness of features and sampling in extractive meeting summarization. In 2008 IEEE Spoken Language Technology Workshop, 157–160. IEEE.
- Zechner, K. 2001. Automatic generation of concise summaries of spoken dialogues in unrestricted domains. In Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, 199–207.
- Zhang, S.; Lei, M.; Liu, Y.; and Li, W. 2019a. Investigation of modeling units for mandarin speech recognition using dfsmn-ctc-smbr. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 7085–7089. IEEE.
- Zhang, X.; Wei, F.; Zhou, M.; et al. 2019b. HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 5059–5069.
Full Text
Tags
Comments