Match$^2$: A Matching over Matching Model for Similar Question Identification
international acm sigir conference on research and development in information retrieval, 2020.
Weibo:
Abstract:
Community Question Answering (CQA) has become a primary means for people to acquire knowledge, where people are free to ask questions or submit answers. To enhance the efficiency of the service, similar question identification becomes a core task in CQA which aims to find a similar question from the archived repository whenever a new qu...More
Code:
Data:
Introduction
- Community estion Answering (CQA) services, such as StackExchange1 and ora2, have grown in popularity in recent years as a platform for people to exchange knowledge.
- CQA services greatly bene t users with highquality human-generated answers for solving their problems, the e ciency becomes a big concern as the asker need to wait until someone submits the answer to his/her question.
- To alleviate this problem, similar question identi cation becomes a core task in CQA which aims to nd a similar question from the archived repository whenever a new question is proposed.
- As shown in Figure 2 Case B, these
Highlights
- Community estion Answering (CQA) services, such as StackExchange1 and ora2, have grown in popularity in recent years as a platform for people to exchange knowledge
- Based on the above idea, we propose a novel Matching over Matching Model, namely Match2 for short, which compares the matching pa erns of the two questions over the same answer for similar question identi cation
- To incorporate the answer information in oraQP, we crawled archived answers of the corresponding questions from ora and enrich the benchmark into a new answer-expanded version, namely oraQP-a. e experimental results on these two benchmarks demonstrated that our method can signi cantly outperform those state-of-the-art methods on the similar question identi cation task
- It can be observed that the relative improvement of the neural methods over TSUBAKI is much larger on oraQP-a than the CQADupStack. e reason might that the oraQP-a is much larger in size than the CQADupStack, where neural models are o en data hungry. e Bert achieves the best performance on
- We introduced a two-side usage of the archived answer for similar question identi cation task by leveraging the answer as a bridge of the questions
- We proposed a novel matching over matching (Match2) model, which consists of three main components, namely the representation-based similarity module, matching pa ern-based similarity module, and the aggregation module
Methods
- CQADupStack is a benchmark dataset which is widely used in CQA [21].
- It contains question threads sampled from twelve StackExchange subforums and annotated with similar question information.
- The authors take the annotated best answer of the question as the archived answer.
- The authors compare the proposed model with previous similar question identi cation methods, which could be classi ed into two categories based on the usage of answers, i.e., question-only methods and oneside methods.
- The authors consider six existing methods which only rely on questions for similar question identi cation
Results
- The authors show the main results of the Match2 model as well as baseline methods.
- For the question-only methods, the authors can see that neural models (e.g., BiMPM, ESIM and etc.) achieve signi cant be er performance than traditional methods (i.e., TSUBAKI) on both datasets.
- It can be observed that the relative improvement of the neural methods over TSUBAKI is much larger on oraQP-a than the CQADupStack.
- The authors nd that the a ention method is relatively more e ective than the concatenation method, which indicates the possibility to improve the performance by carefully designed answer usage method
Conclusion
- CONCLUSION AND FUTURE WORK
In this paper, the authors introduced a two-side usage of the archived answer for similar question identi cation task by leveraging the answer as a bridge of the questions. - The authors proposed a novel matching over matching (Match2) model, which consists of three main components, namely the representation-based similarity module, matching pa ern-based similarity module, and the aggregation module.
- Empirical experiments on two benchmarks demonstrate that the model can signi cantly outperform previous state-of-the-art methods.
- The authors conducted rigorous experiments on the sub-modules to verify the e ectiveness of the model.
- The authors would like to extend the model to leverage variant number of answers and take the answer quality into account
Summary
Introduction:
Community estion Answering (CQA) services, such as StackExchange1 and ora2, have grown in popularity in recent years as a platform for people to exchange knowledge.- CQA services greatly bene t users with highquality human-generated answers for solving their problems, the e ciency becomes a big concern as the asker need to wait until someone submits the answer to his/her question.
- To alleviate this problem, similar question identi cation becomes a core task in CQA which aims to nd a similar question from the archived repository whenever a new question is proposed.
- As shown in Figure 2 Case B, these
Methods:
CQADupStack is a benchmark dataset which is widely used in CQA [21].- It contains question threads sampled from twelve StackExchange subforums and annotated with similar question information.
- The authors take the annotated best answer of the question as the archived answer.
- The authors compare the proposed model with previous similar question identi cation methods, which could be classi ed into two categories based on the usage of answers, i.e., question-only methods and oneside methods.
- The authors consider six existing methods which only rely on questions for similar question identi cation
Results:
The authors show the main results of the Match2 model as well as baseline methods.- For the question-only methods, the authors can see that neural models (e.g., BiMPM, ESIM and etc.) achieve signi cant be er performance than traditional methods (i.e., TSUBAKI) on both datasets.
- It can be observed that the relative improvement of the neural methods over TSUBAKI is much larger on oraQP-a than the CQADupStack.
- The authors nd that the a ention method is relatively more e ective than the concatenation method, which indicates the possibility to improve the performance by carefully designed answer usage method
Conclusion:
CONCLUSION AND FUTURE WORK
In this paper, the authors introduced a two-side usage of the archived answer for similar question identi cation task by leveraging the answer as a bridge of the questions.- The authors proposed a novel matching over matching (Match2) model, which consists of three main components, namely the representation-based similarity module, matching pa ern-based similarity module, and the aggregation module.
- Empirical experiments on two benchmarks demonstrate that the model can signi cantly outperform previous state-of-the-art methods.
- The authors conducted rigorous experiments on the sub-modules to verify the e ectiveness of the model.
- The authors would like to extend the model to leverage variant number of answers and take the answer quality into account
Tables
- Table1: A summary of key notations in this work
- Table2: Dataset statistics. # denotes the number of instances, —lenQ — and —lenA— denote the average length of the questions and answers, respectively
- Table3: Results of di erent similarity functions in the matching pattern-based module on CQADupStack
- Table4: Main Results on CQADupStack and oraQP-a. †indicates the statistically signi cant di erence over the best baseline model, where +/- indicates the statistically signi cant improvement/deterioration over the question-only counterpart with p < 0.01 [<a class="ref-link" id="c49" href="#r49">49</a>]
- Table5: Ablation results on CQADupStack and oraQP-a. †indicates the statistically signi cant di erence over the Match2 model with p < 0.01 [<a class="ref-link" id="c49" href="#r49">49</a>]
- Table6: Two cases from the CQADupStack data. MatchQ2 is the representation-based similarity module, and MatchA2 is the matching pattern-based similarity module
Related work
- In this section, we brie y review the most related topics to our work in CQA, i.e., question matching. estion matching which evaluates the similarity between two questions, could be further divided into the question deduplication task and the similar question identi cation task with regard to di erent application scenarios.
2.1 estion Deduplication estion deduplication aims to merge or remove the redundant questions in the archived question threads. Early studies mainly focused on designing e ective features to measure the similarities between two questions, such as lexical features [4, 17, 23], syntactic features [8, 30, 42], or heuristic features [3, 13]. Many recent successes on this task have been achieved by advanced neural network models. For example, Pang et al [32] evaluated the question similarity from hierarchical levels. Wan et al [41] modeled the recursive structure between question pairs with spatial RNN. Tay et al [38] proposed a CSRAN model to learn ne-grained question matching details. Yang et al [48] built RE2 model with stacked alignment layers to keep the model fast while still yielding strong performance, and Devlin et al [11] pre-trained a stacked transformer network which can be used for question deduplication task a er ne-tuning.
Funding
- is work was supported by the National Natural Science Foundation of China (NSFC) under Grants No 61722211, 61773362, 61872338, and 61902381, Beijing Academy of Arti cial Intelligence (BAAI) under Grants No BAAI2019ZD0306, and BAAI2020ZJ0303, the Youth Innovation Promotion Association CAS under Grants
Reference
- Martın Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Je rey Dean, Ma hieu Devin, et al. 2016. Tensor ow: Large-scale machine learning on heterogeneous distributed systems. arXiv (2016).
- Muhammad Ahasanuzzaman, Muhammad Asaduzzaman, Chanchal K Roy, and Kevin A Schneider. 2016. Mining duplicate questions in stack over ow. In MSR. ACM, 402–412.
- Alberto Barron-Cedeno, Simone Filice, Giovanni Da San Martino, Sha q Joty, Lluıs Marquez, Preslav Nakov, and Alessandro Moschi i. 2015. read-level information for comment classi cation in community question answering. In ACL. 687–693.
- Andrei Z Broder. 1997. On the resemblance and containment of documents. In Proceedings. Compression and Complexity of SEQUENCES. IEEE, 21–29.
- Li Cai, Guangyou Zhou, Kang Liu, and Jun Zhao. 2011. Learning the Latent Topics for estion Retrieval in Community QA. In IJCNLP.
- Xin Cao, Gao Cong, Bin Cui, Christian S Jensen, and an Yuan. 2012. Approaches to exploring category information for question retrieval in community question-answer archives. TOIS 30, 2 (2012), 7.
- Yunbo Cao, Huizhong Duan, Chin-Yew Lin, Yong Yu, and Hsiao-Wuen Hon. 2008. Recommending questions using the mdl-based tree cut model. In WWW. ACM, 81–90.
- David Carmel, Avihai Mejer, Yuval Pinter, and Idan Szpektor. 2014. Improving term weighting for community question answering search using syntactic analysis. In CIKM. ACM, 351–360.
- Qian Chen, Xiaodan Zhu, Zhenhua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2016. Enhanced lstm for natural language inference. arXiv (2016).
- Kyunghyun Cho, Bart Van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv (2014).
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. NAACL (2018).
- Huizhong Duan, Yunbo Cao, Chin-Yew Lin, and Yong Yu. 2008. Searching questions by identifying question topic and question focus. In ACL. 156–164.
- Simone Filice, Giovanni Da San Martino, and Alessandro Moschi i. 2017. KeLP at SemEval-2017 task 3: Learning pairwise pa erns in community question answering. In SemEval-2017. 326–333.
- Bent Fuglede and Flemming Topsoe. 2004. Jensen-Shannon divergence and Hilbert space embedding. In ISIT. IEEE, 31.
- Yichen Gong, Heng Luo, and Jian Zhang. 2017. Natural language inference over interaction space. arXiv (2017).
- Sparsh Gupta and Vitor R Carvalho. 2019. FAQ Retrieval Using A entive Matching. In SIGIR. ACM, 929–932.
- Dan Gus eld. 1997. Algorithms on strings, trees, and sequences: computer science and computational biology.
- Francisco Guzman, Lluıs Marquez, and Preslav Nakov. 2019. Machine translation evaluation meets community question answering. arXiv (2019).
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR. 770–778.
- Doris Hoogeveen, Andrew Benne, Yitong Li, Karin M Verspoor, and Timothy Baldwin. 2018. Detecting mis agged duplicate questions in community questionanswering archives. In AAAI.
- Doris Hoogeveen, Karin M Verspoor, and Timothy Baldwin. [n.d.]. CQADupStack: A benchmark data set for community question-answering research. In ADCS.
- Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. 2017. Densely connected convolutional networks. In CVPR. 4700–4708.
- Paul Jaccard. 1901. Etude comparative de la distribution orale dans une portion des Alpes et des Jura. Bull Soc Vaudoise Sci Nat 37 (1901), 547–579.
- Sarthak Jain and Byron C. Wallace. 2019. A ention is not Explanation. arXiv:cs.CL/1902.10186
- Jiwoon Jeon, W Bruce Cro, and Joon Ho Lee. 2005. Finding similar questions in large question and answer archives. In CIKM. ACM, 84–90.
- Zongcheng Ji, Fei Xu, Bin Wang, and Ben He. 2012. estion-answer topic model for question retrieval in community question answering. In CIKM. ACM.
- Di Liang, Fubao Zhang, Weidong Zhang, Qi Zhang, Jinlan Fu, Minlong Peng, Tao Gui, and Xuanjing Huang. 2019. Adaptive Multi-A ention Network Incorporating Answer Information for Duplicate estion Detection. (2019).
- Min Lin, Qiang Chen, and Shuicheng Yan. 2013. Network in network. arXiv (2013).
- Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. 2019. On the variance of the adaptive learning rate and beyond. arXiv (2019).
- Alessandro Moschi i. 2006. E cient convolution kernels for dependency and constituent syntactic trees. In ECML. Springer, 318–329.
- Preslav Nakov, Lluıs Marquez, and Francisco Guzman. 2016. It takes three to tango: triangulation approach to answer ranking in community question answering. In EMNLP. 1586–1597.
- Liang Pang, Yanyan Lan, Jiafeng Guo, Jun Xu, Shengxian Wan, and Xueqi Cheng. 2016. Text matching as image recognition. In AAAI.
- Xipeng Qiu and Xuanjing Huang. 2015. Convolutional neural tensor network architecture for community-based question answering. In IJCAI.
- Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu, Mike Gatford, et al. 1995. Okapi at TREC-3. NIST SP 109 (1995), 109.
- Lorenza Romano, Milen Kouylekov, Idan Szpektor, Ido Dagan, and Alberto Lavelli. 2006. Investigating a generic paraphrase-based approach for relation extraction. In EACL.
- Wataru Sakata, Tomohide Shibata, Ribeka Tanaka, and Sadao Kurohashi. 2019. FAQ Retrieval using ery- estion Similarity and BERT-Based ery-Answer Relevance. SIGIR (2019).
- Keiji Shinzato, Tomohide Shibata, Daisuke Kawahara, and Sadao Kurohashi. 2012. Tsubaki: An open search engine infrastructure for developing information access methodology. Journal of information processing 20, 1 (2012), 216–227.
- Yi Tay, Luu Anh Tuan, and Siu Cheung Hui. 2018. Co-stack residual a nity networks with multi-level a ention re nement for matching text sequences. arXiv (2018).
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. A ention is all you need. In NIPS. 5998–6008.
- Shengxian Wan, Yanyan Lan, Jiafeng Guo, Jun Xu, Liang Pang, and Xueqi Cheng. 2016. A deep architecture for semantic matching with multiple positional sentence representations. In AAAI.
- Shengxian Wan, Yanyan Lan, Jun Xu, Jiafeng Guo, Liang Pang, and Xueqi Cheng. 2016. Match-srnn: Modeling the recursive matching structure with spatial rnn. IJCAI (2016).
- Mengqiu Wang, Noah A Smith, and Teruko Mitamura. 2007. What is the Jeopardy model? A quasi-synchronous grammar for QA. In EMNLP-CoNLL. 22–32.
- Zhiguo Wang, Wael Hamza, and Radu Florian. 2017. Bilateral multi-perspective matching for natural language sentences. arXiv (2017).
- Guoshun Wu, Yixuan Sheng, Man Lan, and Yuanbin Wu. 2017. ECNU at SemEval2017 Task 3: Using Traditional and Deep Learning Methods to Address Community estion Answering Task. In SemEval-2017. 365–369.
- Wei Wu, Xu Sun, and Houfeng Wang. 2018. estion condensing networks for answer selection in community question answering. In ACL. 1746–1755.
- Xiaobing Xue, Jiwoon Jeon, and W Bruce Cro. 2008. Retrieval models for question and answer archives. In SIGIR. ACM, 475–482.
- Liu Yang, Minghui Qiu, Swapna Go ipati, Feida Zhu, Jing Jiang, Huiping Sun, and Zhong Chen. 2013. Cqarank: jointly model topics and expertise in community question answering. In CIKM. ACM, 99–108.
- Runqi Yang, Jianhai Zhang, Xing Gao, Feng Ji, and Haiqing Chen. 2019. Simple and E ective Text Matching with Richer Alignment Features. ACL (2019).
- Alexander Yeh. 2000. More accurate tests for the statistical signi cance of result di erences. In COLING. ACL, 947–953.
- Chengxiang Zhai and John La erty. 2004. A study of smoothing methods for language models applied to information retrieval. TOIS 22, 2 (2004), 179–214.
- Kai Zhang, Wei Wu, Haocheng Wu, Zhoujun Li, and Ming Zhou. 2014. estion retrieval with high quality answers in community question answering. In CIKM.
- Guangyou Zhou, Li Cai, Jun Zhao, and Kang Liu. 2011. Phrase-based translation model for question retrieval in community question answer archives. In ACL. ACL, 653–662.
- Guangyou Zhou, Yubo Chen, Daojian Zeng, and Jun Zhao. 2013. Towards faster and be er retrieval models for question search. In CIKM. ACM, 2139–2148.
- Guangyou Zhou, Yang Liu, Fang Liu, Daojian Zeng, and Jun Zhao. 2013. Improving question retrieval in community question answering using world knowledge. In IJCAI.
- Tom Chao Zhou, Chin-Yew Lin, Irwin King, Michael R Lyu, Young-In Song, and Yunbo Cao. 2011. Learning to suggest questions in online forums. In AAAI.
Full Text
Tags
Comments