Topic-Oriented Spoken Dialogue Summarization for Customer Service with Saliency-Aware Topic Modeling

Cited by: 0|Bibtex|Views32
Other Links: arxiv.org
Weibo:
We propose a topic-augmented two-stage summarizer with a multi-role topic modeling mechanism for customer service dialogues, which can generate highly abstractive summaries that highlight role-specific information

Abstract:

In a customer service system, dialogue summarization can boost service efficiency by automatically creating summaries for long spoken dialogues in which customers and agents try to address issues about specific topics. In this work, we focus on topic-oriented dialogue summarization, which generates highly abstractive summaries that pres...More

Code:

Data:

0
Introduction
  • In an active customer service system, massive dialogues conveying important information between customers and agents are generated in real time.
  • With this background, how to efficiently consume dialogue information becomes a nontrivial issue.
  • To better understand customers’ and agents’ intentions, in this work the authors focus on the topic-oriented dialogue summarization, which aims to extract semantically consistent topics and generate highly abstractive summaries to maintain the main ideas from dialogues
Highlights
  • In an active customer service system, massive dialogues conveying important information between customers and agents are generated in real time
  • To better understand customers’ and agents’ intentions, in this work we focus on the topic-oriented dialogue summarization, which aims to extract semantically consistent topics and generate highly abstractive summaries to maintain the main ideas from dialogues
  • We propose a novel two-stage neural model jointly with an enhanced topic modeling approach for spoken dialogue summarization
  • To better distinguish the underlying informative content from abundant common semantics and dialogue noise, we introduce a saliency-aware topic model (SATM), where topics are split into two groups: informative topics and other topics
  • We propose a topic-augmented two-stage summarizer with a multi-role topic modeling mechanism for customer service dialogues, which can generate highly abstractive summaries that highlight role-specific information
  • We introduce a novel training regime for topic modeling that directly learns word-saliency correspondences to alleviate the influence of uninformative content
Methods
  • Seq2seq+Att PGNet TRF CopyTRF HiBERT* BERT+TRF* FastRL* TDS+NTM TDS+SATM TDS+NTM* TDS+SATM* RG-1 RG-2 RG-L BLEU.
  • 6.89 9.77 9.87 9.78 9.89 10.19 10.40 9.87 10.34 10.77 11.24 training (90%), development (5%), and test (5%) set.
  • Table 1 shows the detailed statistics of the collected dataset
Conclusion
  • To better understand the influence of role information, saliency-aware topic modeling, and the topic-informed at- SATM Informative Topic SATM Other Topic

    NTM General Topic

    T1: deliver, time, order, address, modify, cancel, ship, return, refund, receive

    T2: feedback, problem, submit, suggest, apply, complain, seller, quality, product, slow

    T3: buy, account, pay, bind, phone, number, modify, check, username, message

    T1: please, wait, service, sorry, really, thanks, bother, mean, find, welcome

    T2: send, call, record, again, later, check, help, keep, contact, reply

    T1: thanks, later, sorry, please, really, phone, feedback, deliver, number, returnConclusion and Future Work

    In this paper, the authors propose a topic-augmented two-stage summarizer with a multi-role topic modeling mechanism for customer service dialogues, which can generate highly abstractive summaries that highlight role-specific information.
  • To better understand the influence of role information, saliency-aware topic modeling, and the topic-informed at- SATM Informative Topic SATM Other Topic.
  • T3: buy, account, pay, bind, phone, number, modify, check, username, message.
  • T1: please, wait, service, sorry, really, thanks, bother, mean, find, welcome.
  • T1: thanks, later, sorry, please, really, phone, feedback, deliver, number, returnConclusion and Future Work.
  • The authors propose a topic-augmented two-stage summarizer with a multi-role topic modeling mechanism for customer service dialogues, which can generate highly abstractive summaries that highlight role-specific information.
  • Future directions may be the exploration of template-guided abstractive methods to make summaries more standardized and easier for reporting
Summary
  • Introduction:

    In an active customer service system, massive dialogues conveying important information between customers and agents are generated in real time.
  • With this background, how to efficiently consume dialogue information becomes a nontrivial issue.
  • To better understand customers’ and agents’ intentions, in this work the authors focus on the topic-oriented dialogue summarization, which aims to extract semantically consistent topics and generate highly abstractive summaries to maintain the main ideas from dialogues
  • Methods:

    Seq2seq+Att PGNet TRF CopyTRF HiBERT* BERT+TRF* FastRL* TDS+NTM TDS+SATM TDS+NTM* TDS+SATM* RG-1 RG-2 RG-L BLEU.
  • 6.89 9.77 9.87 9.78 9.89 10.19 10.40 9.87 10.34 10.77 11.24 training (90%), development (5%), and test (5%) set.
  • Table 1 shows the detailed statistics of the collected dataset
  • Conclusion:

    To better understand the influence of role information, saliency-aware topic modeling, and the topic-informed at- SATM Informative Topic SATM Other Topic

    NTM General Topic

    T1: deliver, time, order, address, modify, cancel, ship, return, refund, receive

    T2: feedback, problem, submit, suggest, apply, complain, seller, quality, product, slow

    T3: buy, account, pay, bind, phone, number, modify, check, username, message

    T1: please, wait, service, sorry, really, thanks, bother, mean, find, welcome

    T2: send, call, record, again, later, check, help, keep, contact, reply

    T1: thanks, later, sorry, please, really, phone, feedback, deliver, number, returnConclusion and Future Work

    In this paper, the authors propose a topic-augmented two-stage summarizer with a multi-role topic modeling mechanism for customer service dialogues, which can generate highly abstractive summaries that highlight role-specific information.
  • To better understand the influence of role information, saliency-aware topic modeling, and the topic-informed at- SATM Informative Topic SATM Other Topic.
  • T3: buy, account, pay, bind, phone, number, modify, check, username, message.
  • T1: please, wait, service, sorry, really, thanks, bother, mean, find, welcome.
  • T1: thanks, later, sorry, please, really, phone, feedback, deliver, number, returnConclusion and Future Work.
  • The authors propose a topic-augmented two-stage summarizer with a multi-role topic modeling mechanism for customer service dialogues, which can generate highly abstractive summaries that highlight role-specific information.
  • Future directions may be the exploration of template-guided abstractive methods to make summaries more standardized and easier for reporting
Tables
  • Table1: Statistics of the customer service dataset
  • Table2: Results of automatic metrics on the customer service dataset. RG-(1,2,L) represents the F1 score of ROUGE(1,2,L). TRF denotes the Transformer. Methods marked with * utilize BERT as the word-level encoder
  • Table3: Human evaluation with system ranking results
  • Table4: Ablation study of TDS+SATM with different kinds of topic modeling. Agent and Cust. represent topic modeling on agent utterances and customer utterances, respectively
  • Table5: Top-10 words of example topics in different topic groups learned by joint training of TDS+Topic Model
Download tables as Excel
Related work
  • Dialogue Summarization

    Dialogue summarization is a challenging task and has been widely explored in various scenarios. Previous works generally focus on summarizing dialogues by stringing key points to maintain an integral dialogue flow: Mehdad et al (2013) and Shang et al (2018) first group utterances that share similar semantics by community detection, and then generate a summary sentence for each utterance group. Liu et al (2019a) propose a hierarchical model to produce key point sequences and generate summaries at the same time for customer service dialogues. Duan et al (2019b) train the assignment of utterances to the corresponding controversy focuses to summarize court debate dialogues. Several works (Zechner 2001; Xie et al 2008; Oya et al 2014; Liu et al 2019b; Li et al 2019a) split dialogues into multiple segments by means of topic segmentation when conducting summarization. Different from above works, Pan et al (2018) first attempt to generate a highly abstractive summary for the entire dialogue, which produces a concise event/object description with a Transformer-based approach. By contrast, in this work, dialogue summaries generally highlight role-specific content, which requires the system to further focus on the
Funding
  • This work was partially funded by China National Key R&D Program (No 2018YFC0831105), National Natural Science Foundation of China (No 61751201, 62076069, 61976056), Shanghai Municipal Science and Technology Major Project (No.2018SHZDZX01), Science and Technology Commission of Shanghai Municipality Grant (No.18DZ1201000, 17JC1420200)
  • This work was supported by Alibaba Group through Alibaba Innovative Research Program
Study subjects and analysis
volunteers: 3
Hence, we further randomly sample 100 examples in the test set for human evaluation. We follow Narayan at al. (2018) to design the experiment, in which three volunteers are invited to compare summaries produced from PGNet, BERT+TRF, HiBERT, FastRL, our proposed TDS+SATM, and the gold sum-. TDS+SATM (w/o) Cust. (w/o) Agent (w/o) Agent & Cust

volunteers: 3
Each volunteer is presented with a dialogue and two summaries produced from two out of six systems and is asked to decide which summary is better in order of two dimensions: informativeness (which summary captures more important information in the dialogue?) and fluency (which summary is more fluent and well-formed?). We collect judgments from three volunteers for each comparison with the order of dialogues and summaries randomized. Table 3 gives the system ranking results of human evaluation

Reference
  • Blei, D. M.; Ng, A. Y.; and Jordan, M. I. 2003. Latent dirichlet allocation. Journal of machine Learning research 3(Jan): 993–1022.
    Google ScholarLocate open access versionFindings
  • Chen, Y.-C.; and Bansal, M. 2018. Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 675–686.
    Google ScholarLocate open access versionFindings
  • Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4171–4186.
    Google ScholarLocate open access versionFindings
  • Duan, X.; Yu, H.; Yin, M.; Zhang, M.; Luo, W.; and Zhang, Y. 2019a. Contrastive Attention Mechanism for Abstractive Sentence Summarization. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 3035–3044.
    Google ScholarLocate open access versionFindings
  • Duan, X.; Zhang, Y.; Yuan, L.; Zhou, X.; Liu, X.; Wang, T.; Wang, R.; Zhang, Q.; Sun, C.; and Wu, F. 2019b. Legal Summarization for Multi-role Debate Dialogue via Controversy Focus Mining and Multi-task Learning. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, 1361–1370.
    Google ScholarLocate open access versionFindings
  • Fu, X.; Wang, J.; Zhang, J.; Wei, J.; and Yang, Z. 2020. Document Summarization with VHTM: Variational Hierarchical Topic-Aware Mechanism. In AAAI, 7740–7747.
    Google ScholarFindings
  • Gillick, D.; Riedhammer, K.; Favre, B.; and Hakkani-Tur, D. 2009. A global optimization framework for meeting summarization. In 2009 IEEE International Conference on Acoustics, Speech and Signal Processing, 4769–4772. IEEE.
    Google ScholarLocate open access versionFindings
  • Higashinaka, R.; Minami, Y.; Nishikawa, H.; Dohsaka, K.; Meguro, T.; Takahashi, S.; and Kikui, G. 2010. Learning to model domain-specific utterance sequences for extractive summarization of contact center dialogues. In Coling 2010: Posters, 400–408.
    Google ScholarLocate open access versionFindings
  • Hofmann, T. 199Probabilistic latent semantic indexing. In Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, 50–57.
    Google ScholarLocate open access versionFindings
  • Kingma, D. P.; and Welling, M. 2014. Auto-Encoding Variational Bayes. CoRR abs/1312.6114.
    Findings
  • Li, M.; Zhang, L.; Ji, H.; and Radke, R. J. 2019a. Keep meeting summaries on topic: Abstractive multi-modal meeting summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2190– 2196.
    Google ScholarLocate open access versionFindings
  • Li, X.; Wang, Y.; Zhang, A.; Li, C.; Chi, J.; and Ouyang, J. 2018. Filtering out the noise in short text topic modeling. Information Sciences 456: 83–96.
    Google ScholarLocate open access versionFindings
  • Li, X.; Zhang, J.; Ouyang, J.; et al. 2019b. Dirichlet Multinomial Mixture with Variational Manifold Regularization: Topic Modeling over Short Texts. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, 7884–7891.
    Google ScholarLocate open access versionFindings
  • Lin, C.-Y. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, 74–81.
    Google ScholarFindings
  • Liu, C.; Wang, P.; Xu, J.; Li, Z.; and Ye, J. 2019a. Automatic Dialogue Summary Generation for Customer Service. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining.
    Google ScholarLocate open access versionFindings
  • Liu, Y.; and Lapata, M. 2019. Text Summarization with Pretrained Encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 3721–3731.
    Google ScholarLocate open access versionFindings
  • Liu, Z.; Ng, A.; Lee, S.; Aw, A. T.; and Chen, N. F. 2019b. Topic-aware pointer-generator networks for summarizing spoken conversations. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 814– 821. IEEE.
    Google ScholarLocate open access versionFindings
  • Maaten, L. v. d.; and Hinton, G. 2008. Visualizing data using t-SNE. Journal of machine learning research 9(Nov): 2579– 2605.
    Google ScholarLocate open access versionFindings
  • Mehdad, Y.; Carenini, G.; Tompa, F.; and Ng, R. 2013. Abstractive meeting summarization with entailment and fusion. In Proceedings of the 14th European Workshop on Natural Language Generation, 136–146.
    Google ScholarLocate open access versionFindings
  • Miao, Y.; Grefenstette, E.; Blunsom, P.; et al. 2017. Discovering discrete latent topics with neural variational inference. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, 2410–2419.
    Google ScholarLocate open access versionFindings
  • Nallapati, R.; Zhai, F.; Zhou, B.; et al. 2017. Summarunner: A recurrent neural network based sequence model for extractive summarization of documents. In Thirty-First AAAI Conference on Artificial Intelligence.
    Google ScholarLocate open access versionFindings
  • Nallapati, R.; Zhou, B.; Gulcehre, C.; and Xiang, B. 2016. Abstractive Text Summarization using Sequence-tosequence RNNs and Beyond. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, 280–290.
    Google ScholarLocate open access versionFindings
  • Narayan, S.; Cohen, S. B.; Lapata, M.; et al. 2018. Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 1797–1807.
    Google ScholarLocate open access versionFindings
  • Oya, T.; Mehdad, Y.; Carenini, G.; and Ng, R. 2014. A template-based abstractive meeting summarization: Leveraging summary and source text relationships. In Proceedings of the 8th International Natural Language Generation Conference (INLG), 45–53.
    Google ScholarLocate open access versionFindings
  • Pan, H.; Zhou, J.; Zhao, Z.; Liu, Y.; Cai, D.; and Yang, M. 2018. Dial2desc: end-to-end dialogue description generation. arXiv preprint arXiv:1811.00185.
    Findings
  • Papineni, K.; Roukos, S.; Ward, T.; and Zhu, W.-J. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, 311–318.
    Google ScholarLocate open access versionFindings
  • Rambow, O.; Shrestha, L.; Chen, J.; and Lauridsen, C. 2004. Summarizing email threads. In Proceedings of HLT-NAACL 2004: Short Papers, 105–108.
    Google ScholarLocate open access versionFindings
  • See, A.; Liu, P. J.; Manning, C. D.; et al. 2017. Get To The Point: Summarization with Pointer-Generator Networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, 1073–1083.
    Google ScholarLocate open access versionFindings
  • Shang, G.; Ding, W.; Zhang, Z.; Tixier, A.; Meladianos, P.; Vazirgiannis, M.; and Lorre, J.-P. 2018. Unsupervised Abstractive Meeting Summarization with Multi-Sentence Compression and Budgeted Submodular Maximization. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 664–674.
    Google ScholarLocate open access versionFindings
  • Sood, A.; Mohamed, T. P.; Varma, V.; et al. 2013. Topicfocused summarization of chat conversations. In European Conference on Information Retrieval, 800–803. Springer.
    Google ScholarLocate open access versionFindings
  • Tixier, A.; Meladianos, P.; Vazirgiannis, M.; et al. 2017. Combining graph degeneracy and submodularity for unsupervised extractive summarization. In Proceedings of the workshop on new frontiers in summarization, 48–58.
    Google ScholarLocate open access versionFindings
  • Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017. Attention is all you need. In Advances in neural information processing systems, 5998–6008.
    Google ScholarLocate open access versionFindings
  • Vinyals, O.; Fortunato, M.; Jaitly, N.; et al. 2015. Pointer networks. In Advances in neural information processing systems, 2692–2700.
    Google ScholarLocate open access versionFindings
  • Wang, J.; Wang, J.; Sun, C.; Li, S.; Liu, X.; Si, L.; Zhang, M.; and Zhou, G. 2020. Sentiment Classification in Customer Service Dialogue with Topic-Aware Multi-Task Learning. In Proceedings of the AAAI Conference on Artificial Intelligence.
    Google ScholarLocate open access versionFindings
  • Wang, L.; and Cardie, C. 2012. Unsupervised topic modeling approaches to decision summarization in spoken meetings. In Proceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue, 40–49.
    Google ScholarLocate open access versionFindings
  • Wang, L.; Yao, J.; Tao, Y.; Zhong, L.; Liu, W.; and Du, Q. 2018. A reinforced topic-aware convolutional sequence-tosequence model for abstractive text summarization. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, 4453–4460.
    Google ScholarLocate open access versionFindings
  • Williams, R. J. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning 8(3-4): 229–256.
    Google ScholarLocate open access versionFindings
  • Xie, S.; Liu, Y.; Lin, H.; et al. 2008. Evaluating the effectiveness of features and sampling in extractive meeting summarization. In 2008 IEEE Spoken Language Technology Workshop, 157–160. IEEE.
    Google ScholarLocate open access versionFindings
  • Zechner, K. 2001. Automatic generation of concise summaries of spoken dialogues in unrestricted domains. In Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, 199–207.
    Google ScholarLocate open access versionFindings
  • Zhang, S.; Lei, M.; Liu, Y.; and Li, W. 2019a. Investigation of modeling units for mandarin speech recognition using dfsmn-ctc-smbr. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 7085–7089. IEEE.
    Google ScholarLocate open access versionFindings
  • Zhang, X.; Wei, F.; Zhou, M.; et al. 2019b. HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 5059–5069.
    Google ScholarLocate open access versionFindings
Full Text
Your rating :
0

 

Tags
Comments