Match$^2$: A Matching over Matching Model for Similar Question Identification

international acm sigir conference on research and development in information retrieval, 2020.

Cited by: 0|Bibtex|Views120
Other Links: arxiv.org|academic.microsoft.com
Weibo:
We proposed a novel matching over matching model, which consists of three main components, namely the representation-based similarity module, matching pa ern-based similarity module, and the aggregation module

Abstract:

Community Question Answering (CQA) has become a primary means for people to acquire knowledge, where people are free to ask questions or submit answers. To enhance the efficiency of the service, similar question identification becomes a core task in CQA which aims to find a similar question from the archived repository whenever a new qu...More

Code:

Data:

0
Introduction
  • Community estion Answering (CQA) services, such as StackExchange1 and ora2, have grown in popularity in recent years as a platform for people to exchange knowledge.
  • CQA services greatly bene t users with highquality human-generated answers for solving their problems, the e ciency becomes a big concern as the asker need to wait until someone submits the answer to his/her question.
  • To alleviate this problem, similar question identi cation becomes a core task in CQA which aims to nd a similar question from the archived repository whenever a new question is proposed.
  • As shown in Figure 2 Case B, these
Highlights
  • Community estion Answering (CQA) services, such as StackExchange1 and ora2, have grown in popularity in recent years as a platform for people to exchange knowledge
  • Based on the above idea, we propose a novel Matching over Matching Model, namely Match2 for short, which compares the matching pa erns of the two questions over the same answer for similar question identi cation
  • To incorporate the answer information in oraQP, we crawled archived answers of the corresponding questions from ora and enrich the benchmark into a new answer-expanded version, namely oraQP-a. e experimental results on these two benchmarks demonstrated that our method can signi cantly outperform those state-of-the-art methods on the similar question identi cation task
  • It can be observed that the relative improvement of the neural methods over TSUBAKI is much larger on oraQP-a than the CQADupStack. e reason might that the oraQP-a is much larger in size than the CQADupStack, where neural models are o en data hungry. e Bert achieves the best performance on
  • We introduced a two-side usage of the archived answer for similar question identi cation task by leveraging the answer as a bridge of the questions
  • We proposed a novel matching over matching (Match2) model, which consists of three main components, namely the representation-based similarity module, matching pa ern-based similarity module, and the aggregation module
Methods
  • CQADupStack is a benchmark dataset which is widely used in CQA [21].
  • It contains question threads sampled from twelve StackExchange subforums and annotated with similar question information.
  • The authors take the annotated best answer of the question as the archived answer.
  • The authors compare the proposed model with previous similar question identi cation methods, which could be classi ed into two categories based on the usage of answers, i.e., question-only methods and oneside methods.
  • The authors consider six existing methods which only rely on questions for similar question identi cation
Results
  • The authors show the main results of the Match2 model as well as baseline methods.
  • For the question-only methods, the authors can see that neural models (e.g., BiMPM, ESIM and etc.) achieve signi cant be er performance than traditional methods (i.e., TSUBAKI) on both datasets.
  • It can be observed that the relative improvement of the neural methods over TSUBAKI is much larger on oraQP-a than the CQADupStack.
  • The authors nd that the a ention method is relatively more e ective than the concatenation method, which indicates the possibility to improve the performance by carefully designed answer usage method
Conclusion
  • CONCLUSION AND FUTURE WORK

    In this paper, the authors introduced a two-side usage of the archived answer for similar question identi cation task by leveraging the answer as a bridge of the questions.
  • The authors proposed a novel matching over matching (Match2) model, which consists of three main components, namely the representation-based similarity module, matching pa ern-based similarity module, and the aggregation module.
  • Empirical experiments on two benchmarks demonstrate that the model can signi cantly outperform previous state-of-the-art methods.
  • The authors conducted rigorous experiments on the sub-modules to verify the e ectiveness of the model.
  • The authors would like to extend the model to leverage variant number of answers and take the answer quality into account
Summary
  • Introduction:

    Community estion Answering (CQA) services, such as StackExchange1 and ora2, have grown in popularity in recent years as a platform for people to exchange knowledge.
  • CQA services greatly bene t users with highquality human-generated answers for solving their problems, the e ciency becomes a big concern as the asker need to wait until someone submits the answer to his/her question.
  • To alleviate this problem, similar question identi cation becomes a core task in CQA which aims to nd a similar question from the archived repository whenever a new question is proposed.
  • As shown in Figure 2 Case B, these
  • Methods:

    CQADupStack is a benchmark dataset which is widely used in CQA [21].
  • It contains question threads sampled from twelve StackExchange subforums and annotated with similar question information.
  • The authors take the annotated best answer of the question as the archived answer.
  • The authors compare the proposed model with previous similar question identi cation methods, which could be classi ed into two categories based on the usage of answers, i.e., question-only methods and oneside methods.
  • The authors consider six existing methods which only rely on questions for similar question identi cation
  • Results:

    The authors show the main results of the Match2 model as well as baseline methods.
  • For the question-only methods, the authors can see that neural models (e.g., BiMPM, ESIM and etc.) achieve signi cant be er performance than traditional methods (i.e., TSUBAKI) on both datasets.
  • It can be observed that the relative improvement of the neural methods over TSUBAKI is much larger on oraQP-a than the CQADupStack.
  • The authors nd that the a ention method is relatively more e ective than the concatenation method, which indicates the possibility to improve the performance by carefully designed answer usage method
  • Conclusion:

    CONCLUSION AND FUTURE WORK

    In this paper, the authors introduced a two-side usage of the archived answer for similar question identi cation task by leveraging the answer as a bridge of the questions.
  • The authors proposed a novel matching over matching (Match2) model, which consists of three main components, namely the representation-based similarity module, matching pa ern-based similarity module, and the aggregation module.
  • Empirical experiments on two benchmarks demonstrate that the model can signi cantly outperform previous state-of-the-art methods.
  • The authors conducted rigorous experiments on the sub-modules to verify the e ectiveness of the model.
  • The authors would like to extend the model to leverage variant number of answers and take the answer quality into account
Tables
  • Table1: A summary of key notations in this work
  • Table2: Dataset statistics. # denotes the number of instances, —lenQ — and —lenA— denote the average length of the questions and answers, respectively
  • Table3: Results of di erent similarity functions in the matching pattern-based module on CQADupStack
  • Table4: Main Results on CQADupStack and oraQP-a. †indicates the statistically signi cant di erence over the best baseline model, where +/- indicates the statistically signi cant improvement/deterioration over the question-only counterpart with p < 0.01 [<a class="ref-link" id="c49" href="#r49">49</a>]
  • Table5: Ablation results on CQADupStack and oraQP-a. †indicates the statistically signi cant di erence over the Match2 model with p < 0.01 [<a class="ref-link" id="c49" href="#r49">49</a>]
  • Table6: Two cases from the CQADupStack data. MatchQ2 is the representation-based similarity module, and MatchA2 is the matching pattern-based similarity module
Download tables as Excel
Related work
  • In this section, we brie y review the most related topics to our work in CQA, i.e., question matching. estion matching which evaluates the similarity between two questions, could be further divided into the question deduplication task and the similar question identi cation task with regard to di erent application scenarios.

    2.1 estion Deduplication estion deduplication aims to merge or remove the redundant questions in the archived question threads. Early studies mainly focused on designing e ective features to measure the similarities between two questions, such as lexical features [4, 17, 23], syntactic features [8, 30, 42], or heuristic features [3, 13]. Many recent successes on this task have been achieved by advanced neural network models. For example, Pang et al [32] evaluated the question similarity from hierarchical levels. Wan et al [41] modeled the recursive structure between question pairs with spatial RNN. Tay et al [38] proposed a CSRAN model to learn ne-grained question matching details. Yang et al [48] built RE2 model with stacked alignment layers to keep the model fast while still yielding strong performance, and Devlin et al [11] pre-trained a stacked transformer network which can be used for question deduplication task a er ne-tuning.
Funding
  • is work was supported by the National Natural Science Foundation of China (NSFC) under Grants No 61722211, 61773362, 61872338, and 61902381, Beijing Academy of Arti cial Intelligence (BAAI) under Grants No BAAI2019ZD0306, and BAAI2020ZJ0303, the Youth Innovation Promotion Association CAS under Grants
Reference
  • Martın Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Je rey Dean, Ma hieu Devin, et al. 2016. Tensor ow: Large-scale machine learning on heterogeneous distributed systems. arXiv (2016).
    Google ScholarLocate open access versionFindings
  • Muhammad Ahasanuzzaman, Muhammad Asaduzzaman, Chanchal K Roy, and Kevin A Schneider. 2016. Mining duplicate questions in stack over ow. In MSR. ACM, 402–412.
    Google ScholarFindings
  • Alberto Barron-Cedeno, Simone Filice, Giovanni Da San Martino, Sha q Joty, Lluıs Marquez, Preslav Nakov, and Alessandro Moschi i. 2015. read-level information for comment classi cation in community question answering. In ACL. 687–693.
    Google ScholarLocate open access versionFindings
  • Andrei Z Broder. 1997. On the resemblance and containment of documents. In Proceedings. Compression and Complexity of SEQUENCES. IEEE, 21–29.
    Google ScholarLocate open access versionFindings
  • Li Cai, Guangyou Zhou, Kang Liu, and Jun Zhao. 2011. Learning the Latent Topics for estion Retrieval in Community QA. In IJCNLP.
    Google ScholarLocate open access versionFindings
  • Xin Cao, Gao Cong, Bin Cui, Christian S Jensen, and an Yuan. 2012. Approaches to exploring category information for question retrieval in community question-answer archives. TOIS 30, 2 (2012), 7.
    Google ScholarLocate open access versionFindings
  • Yunbo Cao, Huizhong Duan, Chin-Yew Lin, Yong Yu, and Hsiao-Wuen Hon. 2008. Recommending questions using the mdl-based tree cut model. In WWW. ACM, 81–90.
    Google ScholarLocate open access versionFindings
  • David Carmel, Avihai Mejer, Yuval Pinter, and Idan Szpektor. 2014. Improving term weighting for community question answering search using syntactic analysis. In CIKM. ACM, 351–360.
    Google ScholarLocate open access versionFindings
  • Qian Chen, Xiaodan Zhu, Zhenhua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2016. Enhanced lstm for natural language inference. arXiv (2016).
    Google ScholarLocate open access versionFindings
  • Kyunghyun Cho, Bart Van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv (2014).
    Google ScholarLocate open access versionFindings
  • Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. NAACL (2018).
    Google ScholarLocate open access versionFindings
  • Huizhong Duan, Yunbo Cao, Chin-Yew Lin, and Yong Yu. 2008. Searching questions by identifying question topic and question focus. In ACL. 156–164.
    Google ScholarLocate open access versionFindings
  • Simone Filice, Giovanni Da San Martino, and Alessandro Moschi i. 2017. KeLP at SemEval-2017 task 3: Learning pairwise pa erns in community question answering. In SemEval-2017. 326–333.
    Google ScholarLocate open access versionFindings
  • Bent Fuglede and Flemming Topsoe. 2004. Jensen-Shannon divergence and Hilbert space embedding. In ISIT. IEEE, 31.
    Google ScholarLocate open access versionFindings
  • Yichen Gong, Heng Luo, and Jian Zhang. 2017. Natural language inference over interaction space. arXiv (2017).
    Google ScholarLocate open access versionFindings
  • Sparsh Gupta and Vitor R Carvalho. 2019. FAQ Retrieval Using A entive Matching. In SIGIR. ACM, 929–932.
    Google ScholarLocate open access versionFindings
  • Dan Gus eld. 1997. Algorithms on strings, trees, and sequences: computer science and computational biology.
    Google ScholarFindings
  • Francisco Guzman, Lluıs Marquez, and Preslav Nakov. 2019. Machine translation evaluation meets community question answering. arXiv (2019).
    Google ScholarLocate open access versionFindings
  • Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR. 770–778.
    Google ScholarLocate open access versionFindings
  • Doris Hoogeveen, Andrew Benne, Yitong Li, Karin M Verspoor, and Timothy Baldwin. 2018. Detecting mis agged duplicate questions in community questionanswering archives. In AAAI.
    Google ScholarFindings
  • Doris Hoogeveen, Karin M Verspoor, and Timothy Baldwin. [n.d.]. CQADupStack: A benchmark data set for community question-answering research. In ADCS.
    Google ScholarFindings
  • Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. 2017. Densely connected convolutional networks. In CVPR. 4700–4708.
    Google ScholarFindings
  • Paul Jaccard. 1901. Etude comparative de la distribution orale dans une portion des Alpes et des Jura. Bull Soc Vaudoise Sci Nat 37 (1901), 547–579.
    Google ScholarLocate open access versionFindings
  • Sarthak Jain and Byron C. Wallace. 2019. A ention is not Explanation. arXiv:cs.CL/1902.10186
    Google ScholarFindings
  • Jiwoon Jeon, W Bruce Cro, and Joon Ho Lee. 2005. Finding similar questions in large question and answer archives. In CIKM. ACM, 84–90.
    Google ScholarLocate open access versionFindings
  • Zongcheng Ji, Fei Xu, Bin Wang, and Ben He. 2012. estion-answer topic model for question retrieval in community question answering. In CIKM. ACM.
    Google ScholarLocate open access versionFindings
  • Di Liang, Fubao Zhang, Weidong Zhang, Qi Zhang, Jinlan Fu, Minlong Peng, Tao Gui, and Xuanjing Huang. 2019. Adaptive Multi-A ention Network Incorporating Answer Information for Duplicate estion Detection. (2019).
    Google ScholarFindings
  • Min Lin, Qiang Chen, and Shuicheng Yan. 2013. Network in network. arXiv (2013).
    Google ScholarLocate open access versionFindings
  • Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. 2019. On the variance of the adaptive learning rate and beyond. arXiv (2019).
    Google ScholarLocate open access versionFindings
  • Alessandro Moschi i. 2006. E cient convolution kernels for dependency and constituent syntactic trees. In ECML. Springer, 318–329.
    Google ScholarFindings
  • Preslav Nakov, Lluıs Marquez, and Francisco Guzman. 2016. It takes three to tango: triangulation approach to answer ranking in community question answering. In EMNLP. 1586–1597.
    Google ScholarFindings
  • Liang Pang, Yanyan Lan, Jiafeng Guo, Jun Xu, Shengxian Wan, and Xueqi Cheng. 2016. Text matching as image recognition. In AAAI.
    Google ScholarFindings
  • Xipeng Qiu and Xuanjing Huang. 2015. Convolutional neural tensor network architecture for community-based question answering. In IJCAI.
    Google ScholarFindings
  • Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu, Mike Gatford, et al. 1995. Okapi at TREC-3. NIST SP 109 (1995), 109.
    Google ScholarFindings
  • Lorenza Romano, Milen Kouylekov, Idan Szpektor, Ido Dagan, and Alberto Lavelli. 2006. Investigating a generic paraphrase-based approach for relation extraction. In EACL.
    Google ScholarFindings
  • Wataru Sakata, Tomohide Shibata, Ribeka Tanaka, and Sadao Kurohashi. 2019. FAQ Retrieval using ery- estion Similarity and BERT-Based ery-Answer Relevance. SIGIR (2019).
    Google ScholarLocate open access versionFindings
  • Keiji Shinzato, Tomohide Shibata, Daisuke Kawahara, and Sadao Kurohashi. 2012. Tsubaki: An open search engine infrastructure for developing information access methodology. Journal of information processing 20, 1 (2012), 216–227.
    Google ScholarLocate open access versionFindings
  • Yi Tay, Luu Anh Tuan, and Siu Cheung Hui. 2018. Co-stack residual a nity networks with multi-level a ention re nement for matching text sequences. arXiv (2018).
    Google ScholarFindings
  • Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. A ention is all you need. In NIPS. 5998–6008.
    Google ScholarFindings
  • Shengxian Wan, Yanyan Lan, Jiafeng Guo, Jun Xu, Liang Pang, and Xueqi Cheng. 2016. A deep architecture for semantic matching with multiple positional sentence representations. In AAAI.
    Google ScholarFindings
  • Shengxian Wan, Yanyan Lan, Jun Xu, Jiafeng Guo, Liang Pang, and Xueqi Cheng. 2016. Match-srnn: Modeling the recursive matching structure with spatial rnn. IJCAI (2016).
    Google ScholarLocate open access versionFindings
  • Mengqiu Wang, Noah A Smith, and Teruko Mitamura. 2007. What is the Jeopardy model? A quasi-synchronous grammar for QA. In EMNLP-CoNLL. 22–32.
    Google ScholarFindings
  • Zhiguo Wang, Wael Hamza, and Radu Florian. 2017. Bilateral multi-perspective matching for natural language sentences. arXiv (2017).
    Google ScholarLocate open access versionFindings
  • Guoshun Wu, Yixuan Sheng, Man Lan, and Yuanbin Wu. 2017. ECNU at SemEval2017 Task 3: Using Traditional and Deep Learning Methods to Address Community estion Answering Task. In SemEval-2017. 365–369.
    Google ScholarLocate open access versionFindings
  • Wei Wu, Xu Sun, and Houfeng Wang. 2018. estion condensing networks for answer selection in community question answering. In ACL. 1746–1755.
    Google ScholarFindings
  • Xiaobing Xue, Jiwoon Jeon, and W Bruce Cro. 2008. Retrieval models for question and answer archives. In SIGIR. ACM, 475–482.
    Google ScholarLocate open access versionFindings
  • Liu Yang, Minghui Qiu, Swapna Go ipati, Feida Zhu, Jing Jiang, Huiping Sun, and Zhong Chen. 2013. Cqarank: jointly model topics and expertise in community question answering. In CIKM. ACM, 99–108.
    Google ScholarLocate open access versionFindings
  • Runqi Yang, Jianhai Zhang, Xing Gao, Feng Ji, and Haiqing Chen. 2019. Simple and E ective Text Matching with Richer Alignment Features. ACL (2019).
    Google ScholarLocate open access versionFindings
  • Alexander Yeh. 2000. More accurate tests for the statistical signi cance of result di erences. In COLING. ACL, 947–953.
    Google ScholarLocate open access versionFindings
  • Chengxiang Zhai and John La erty. 2004. A study of smoothing methods for language models applied to information retrieval. TOIS 22, 2 (2004), 179–214.
    Google ScholarLocate open access versionFindings
  • Kai Zhang, Wei Wu, Haocheng Wu, Zhoujun Li, and Ming Zhou. 2014. estion retrieval with high quality answers in community question answering. In CIKM.
    Google ScholarFindings
  • Guangyou Zhou, Li Cai, Jun Zhao, and Kang Liu. 2011. Phrase-based translation model for question retrieval in community question answer archives. In ACL. ACL, 653–662.
    Google ScholarLocate open access versionFindings
  • Guangyou Zhou, Yubo Chen, Daojian Zeng, and Jun Zhao. 2013. Towards faster and be er retrieval models for question search. In CIKM. ACM, 2139–2148.
    Google ScholarFindings
  • Guangyou Zhou, Yang Liu, Fang Liu, Daojian Zeng, and Jun Zhao. 2013. Improving question retrieval in community question answering using world knowledge. In IJCAI.
    Google ScholarFindings
  • Tom Chao Zhou, Chin-Yew Lin, Irwin King, Michael R Lyu, Young-In Song, and Yunbo Cao. 2011. Learning to suggest questions in online forums. In AAAI.
    Google ScholarFindings
Full Text
Your rating :
0

 

Tags
Comments