AI帮你理解科学

AI 生成解读视频

AI抽取解析论文重点内容自动生成视频


pub
生成解读视频

AI 溯源

AI解析本论文相关学术脉络


Master Reading Tree
生成 溯源树

AI 精读

AI抽取本论文的概要总结


微博一下
We address the problem of detecting duplicate questions in forums, which is an important step towards automating the process of answering new questions

Adversarial Domain Adaptation for Duplicate Question Detection.

EMNLP, (2018): 1056-1063

被引用30|浏览242
EI
下载 PDF 全文
引用
微博一下

摘要

We address the problem of detecting duplicate questions in forums, which is an important step towards automating the process of answering new questions. As finding and annotating such potential duplicates manually is very tedious and costly, automatic methods based on machine learning are a viable alternative. However, many forums do not ...更多

代码

数据

简介
  • Recent years have seen the rise of community question answering forums, which allow users to ask questions and to get answers in a collaborative fashion.
  • Duplicate question detection is a special case of the more general problem of question-question similarity
  • The latter was addressed using a variety of textual similarity measures, topic modeling (Cao et al, 2008; Zhang et al, 2014), and syntactic structure (Wang et al, 2009; Filice et al, 2016; Da San Martino et al, 2016; Barrón-Cedeño et al, 2016; Filice et al, 2017).
  • Translation models have been popular (Zhou et al, 2011; Jeon et al, 2005; Guzmán et al, 2016a,b)
重点内容
  • Recent years have seen the rise of community question answering forums, which allow users to ask questions and to get answers in a collaborative fashion
  • Duplicate question detection is a special case of the more general problem of question-question similarity. The latter was addressed using a variety of textual similarity measures, topic modeling (Cao et al, 2008; Zhang et al, 2014), and syntactic structure (Wang et al, 2009; Filice et al, 2016; Da San Martino et al, 2016; Barrón-Cedeño et al, 2016; Filice et al, 2017)
  • The above work assumes labeled training data, which exists for question-question similarity, e.g., from SemEval-2016/2017 (Agirre et al, 2016; Nakov et al, 2016b, 2017), and for duplicate question detection, e.g., SemEval-2017 task 3 featured four StackExchange forums, Android, English, Gaming, and Wordpress, from CQADupStack (Hoogeveen et al, 2015, 2016). Such annotation is not available for many other forums, e.g., the Apple community on StackExchange. We address this lack of annotation using adversarial domain adaptation (ADA) to effectively use labeled data from another forum
  • We show that adversarial domain adaptation can be efficient even for unseen target domains, given some similarity of the target domain with the source one and with the regularizing adversarial domain
  • On the StackExchange family of forums, our model outperforms the best baseline with an average relative improvement of 5.6% across all domain pairs
  • We describe the design choices considered for our domain adaptation model in the following two subsections
方法
  • The authors' ADA model has three components: (i) question encoder, (ii) similarity function, and (iii) domain adaptation component, as shown in Figure 1.
  • The similarity function f takes two question vectors, v1 and v2, and predicts whether the corresponding questions are duplicates.
  • The domain classifier g takes a question vector v and predicts whether the question is from the source or from the target domain.
  • The authors train the encoder to do well on the task for the source data, and to fool the domain classifier, as shown in Algorithm 1.
  • The authors describe the design choices considered for the domain adaptation model in the following two subsections
结果
  • As the datasets may contain some duplicate question pairs, which were not discovered and not annotated, the authors end up having false negatives.
  • Metrics such as MAP and MRR are not suitable in this situation.
  • The authors compute the area integrating the false positive rate (x-axis) from 0 up to a threshold t, and the authors normalize the area to [0, 1]
  • This score is known as AUC(t).
  • It is more stable than MRR and MAP in the case when there could be several false negatives.3
结论
  • Conclusion and Future Work

    The authors have applied and analyzed adversarial methods for domain transfer for the task of duplicate question detection; to the best of the knowledge, this is the first such work.
  • One idea is to try source-pivot-target transfer, to the way this is done for machine translation (Wu and Wang, 2007).
  • Another promising direction is to have an attention mechanism (Luong et al, 2015) for question similarity which can be adapted across domains.4
总结
  • Introduction:

    Recent years have seen the rise of community question answering forums, which allow users to ask questions and to get answers in a collaborative fashion.
  • Duplicate question detection is a special case of the more general problem of question-question similarity
  • The latter was addressed using a variety of textual similarity measures, topic modeling (Cao et al, 2008; Zhang et al, 2014), and syntactic structure (Wang et al, 2009; Filice et al, 2016; Da San Martino et al, 2016; Barrón-Cedeño et al, 2016; Filice et al, 2017).
  • Translation models have been popular (Zhou et al, 2011; Jeon et al, 2005; Guzmán et al, 2016a,b)
  • Methods:

    The authors' ADA model has three components: (i) question encoder, (ii) similarity function, and (iii) domain adaptation component, as shown in Figure 1.
  • The similarity function f takes two question vectors, v1 and v2, and predicts whether the corresponding questions are duplicates.
  • The domain classifier g takes a question vector v and predicts whether the question is from the source or from the target domain.
  • The authors train the encoder to do well on the task for the source data, and to fool the domain classifier, as shown in Algorithm 1.
  • The authors describe the design choices considered for the domain adaptation model in the following two subsections
  • Results:

    As the datasets may contain some duplicate question pairs, which were not discovered and not annotated, the authors end up having false negatives.
  • Metrics such as MAP and MRR are not suitable in this situation.
  • The authors compute the area integrating the false positive rate (x-axis) from 0 up to a threshold t, and the authors normalize the area to [0, 1]
  • This score is known as AUC(t).
  • It is more stable than MRR and MAP in the case when there could be several false negatives.3
  • Conclusion:

    Conclusion and Future Work

    The authors have applied and analyzed adversarial methods for domain transfer for the task of duplicate question detection; to the best of the knowledge, this is the first such work.
  • One idea is to try source-pivot-target transfer, to the way this is done for machine translation (Wu and Wang, 2007).
  • Another promising direction is to have an attention mechanism (Luong et al, 2015) for question similarity which can be adapted across domains.4
表格
  • Table1: Statistics about the datasets. The table shows the number of question pairs that have been manually marked as similar/duplicates by the forum users (i.e., positive pairs). We further add 100 negative question pairs per duplicate question by randomly sampling from the full corpus of questions
  • Table2: Proportion of n-grams that are shared between the source and the target domains
  • Table3: Duplicate question detection: direct transfer vs
  • Table4: Domain adaptation for the StackExchange sourcetarget domain pairs when using the Direct approach, BM25, and our adaptation model, measured with AUC(0.05)
  • Table5: Domain adaptation results when using Sprint and Quora as the source domains with the Direct approach, BM25, and our adaptation model, measured with AUC(0.05)
  • Table6: AUC(0.05) of ADA to unseen domains, with
Download tables as Excel
基金
  • Our experiments with StackExchange data show an average improvement of 5.6% over the best baseline across multiple pairs of domains
  • • on the StackExchange family of forums, our model outperforms the best baseline with an average relative improvement of 5.6% (up to 14%) across all domain pairs
  • • For almost all source–target domain pairs from the StackExchange family, domain adaptation improves over both baselines, with an average relative improvement of 5.6%
  • This improvement goes up to 14% for the AskUbuntu–Android source–target domain pair
研究对象与分析
negative question pairs: 100
. Statistics about the datasets. The table shows the number of question pairs that have been manually marked as similar/duplicates by the forum users (i.e., positive pairs). We further add 100 negative question pairs per duplicate question by randomly sampling from the full corpus of questions. Proportion of n-grams that are shared between the source and the target domains

引用论文
  • Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2016. SemEval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation. In Proceedings of the International Workshop on Semantic Evaluation, SemEval ’16, pages 497–511, San Diego, CA, USA.
    Google ScholarLocate open access versionFindings
  • Martín Arjovsky and Léon Bottou. 2017. Towards principled methods for training generative adversarial networks. In Proceedings of the 5th International Conference on Learning Representations, ICLR ’17, Toulon, France.
    Google ScholarLocate open access versionFindings
  • Martín Arjovsky, Soumith Chintala, and Léon Bottou. 2017. Wasserstein generative adversarial networks. In Proceedings of the 34th International Conference on Machine Learning, ICML ’17, pages 214–223, Sydney, NSW, Australia.
    Google ScholarLocate open access versionFindings
  • Alberto Barrón-Cedeño, Giovanni Da San Martino, Shafiq Joty, Alessandro Moschitti, Fahad A. Al Obaidli, Salvatore Romeo, Kateryna Tymoshenko, and Antonio Uva. 2016. ConvKN at SemEval-2016 Task 3: Answer and question selection for question answering on Arabic and English fora. In Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval ’16, pages 896–903, San Diego, CA, USA.
    Google ScholarLocate open access versionFindings
  • Daniele Bonadiman, Antonio Uva, and Alessandro Moschitti. 2017. Effective shared representations with multitask learning for community question answering. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, EACL ’17, pages 726–732, Valencia, Spain.
    Google ScholarLocate open access versionFindings
  • Yunbo Cao, Huizhong Duan, Chin-Yew Lin, Yong Yu, and Hsiao-Wuen Hon. 2008. Recommending questions using the MDL-based tree cut model. In Proceedings of the International Conference on World Wide Web, WWW ’08, pages 81–90, Beijing, China.
    Google ScholarLocate open access versionFindings
  • Xilun Chen, Yu Sun, Ben Athiwaratkun, Claire Cardie, and Kilian Weinberger. 2016. Adversarial deep averaging networks for cross-lingual sentiment classification. arXiv preprint arXiv:1606.01614.
    Findings
  • Giovanni Da San Martino, Alberto Barrón-Cedeño, Salvatore Romeo, Antonio Uva, and Alessandro Moschitti. 2016. Learning to re-rank questions in community question answering using advanced features. In Proceedings of the ACM Conference on Information and Knowledge Management, CIKM ’16, pages 1997–2000, Indianapolis, IN, USA.
    Google ScholarLocate open access versionFindings
  • Giovanni Da San Martino, Salvatore Romeo, Alberto Barroón-Cedeño, Shafiq Joty, Lluís Màrquez, Alessandro Moschitti, and Preslav Nakov. 2017. Cross-language question re-ranking. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’17, pages 1145–1148, Tokyo, Japan.
    Google ScholarLocate open access versionFindings
  • Simone Filice, Danilo Croce, Alessandro Moschitti, and Roberto Basili. 2016. KeLP at SemEval-2016 task 3: Learning semantic relations between questions and answers. In Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval ’16, pages 1116–1123, San Diego, CA, USA.
    Google ScholarLocate open access versionFindings
  • Simone Filice, Giovanni Da San Martino, and Alessandro Moschitti. 2017. KeLP at SemEval-2017 task 3: Learning pairwise patterns in community question answering. In Proceedings of the 11th International Workshop on Semantic Evaluation, SemEval ’17, pages 327–334, Vancouver, Canada.
    Google ScholarLocate open access versionFindings
  • Yaroslav Ganin and Victor Lempitsky. 2015. Unsupervised domain adaptation by backpropagation. In Proceedings of the International Conference on Machine Learning, ICML ’15, pages 1180–1189, Lille, France.
    Google ScholarLocate open access versionFindings
  • Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. 2016. Domain-adversarial training of neural networks. Journal of Machine Learning Research, 17(1):2096–2030.
    Google ScholarLocate open access versionFindings
  • Francisco Guzmán, Lluís Màrquez, and Preslav Nakov. 2016a. Machine translation evaluation meets community question answering. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL ’16, pages 460–466, Berlin, Germany.
    Google ScholarLocate open access versionFindings
  • Francisco Guzmán, Lluís Màrquez, and Preslav Nakov. 2016b. MTE-NN at SemEval-2016 Task 3: Can machine translation evaluation help community question answering? In Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval ’16, pages 887–895, San Diego, CA, USA.
    Google ScholarLocate open access versionFindings
  • Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780.
    Google ScholarLocate open access versionFindings
  • Doris Hoogeveen, Karin Verspoor, and Timothy Baldwin. 2016. CQADupStack: Gold or silver? In Proceedings of the SIGIR Workshop on Web Question Answering Beyond Factoids, WebQA ’16, Pisa, Italy.
    Google ScholarLocate open access versionFindings
  • Doris Hoogeveen, Karin M. Verspoor, and Timothy Baldwin. 2015. CQADupStack: A benchmark data set for community question-answering research. In Proceedings of the 20th Australasian Document Computing Symposium, ADCS ’15, pages 3:1–3:8, Parramatta, NSW, Australia.
    Google ScholarLocate open access versionFindings
  • Doris Hoogeveen, Li Wang, Timothy Baldwin, and Karin M. Verspoor. 2018. Web forum retrieval and text analytics: A survey. Foundations and Trends in Information Retrieval, 12(1):1–163.
    Google ScholarLocate open access versionFindings
  • Jiwoon Jeon, W. Bruce Croft, and Joon Ho Lee. 2005. Finding similar questions in large question and answer archives. In Proceedings of the ACM Conference on Information and Knowledge Management, CIKM ’05, pages 84–90, Bremen, Germany.
    Google ScholarLocate open access versionFindings
  • Shafiq Joty, Lluís Màrquez, and Preslav Nakov. 2018. Joint multitask learning for community question answering using task-specific embeddings. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP ’18, Brussels, Belgium.
    Google ScholarLocate open access versionFindings
  • Shafiq Joty, Preslav Nakov, Lluís Màrquez, and Israa Jaradat. 2017. Cross-language learning with adversarial neural networks. In Proceedings of the 21st Conference on Computational Natural Language Learning, CoNLL ’17, pages 226–237, Vancouver, Canada.
    Google ScholarLocate open access versionFindings
  • Tuan Manh Lai, Trung Bui, and Sheng Li. 2018. A review on deep learning techniques applied to answer selection. In Proceedings of the International Conference on Computational Linguistics, COLING ’18, pages 2132–2144, Santa Fe, NM, USA.
    Google ScholarLocate open access versionFindings
  • Tao Lei, Hrishikesh Joshi, Regina Barzilay, Tommi S. Jaakkola, Kateryna Tymoshenko, Alessandro Moschitti, and Lluís Màrquez. 2016. Semi-supervised question retrieval with gated convolutions. In Proceedings of the 15th Annual Conference of the North American Chapter of the Association for Computational Linguistics, NAACL-HLT ’16, pages 1279– 1289, San Diego, CA, USA.
    Google ScholarLocate open access versionFindings
  • Zheng Li, Yu Zhang, Ying Wei, Yuxiang Wu, and Qiang Yang. 2017. End-to-end adversarial memory network for cross-domain sentiment classification. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, IJCAI’17, pages 2237–2243, Melbourne, Australia.
    Google ScholarLocate open access versionFindings
  • Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2017. Adversarial multi-task learning for text classification. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL ’17, pages 1–10, Vancouver, Canada.
    Google ScholarLocate open access versionFindings
  • Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP ’15, pages 1412– 1421, Lisbon, Portugal.
    Google ScholarLocate open access versionFindings
  • Michael McCandless, Erik Hatcher, and Otis Gospodnetic. 2010. Lucene in Action: Covers Apache Lucene 3.0. Manning Publications Co.
    Google ScholarFindings
  • Jerome L Myers, Arnold Well, and Robert Frederick Lorch. 2010. Research design and statistical analysis. Routledge.
    Google ScholarFindings
  • Preslav Nakov, Doris Hoogeveen, Lluís Màrquez, Alessandro Moschitti, Hamdy Mubarak, Timothy Baldwin, and Karin Verspoor. 2017. SemEval-2017 task 3: Community question answering. In Proceedings of the International Workshop on Semantic Evaluation, SemEval ’17, pages 525–545, Vancouver, Canada.
    Google ScholarLocate open access versionFindings
  • Preslav Nakov, Lluís Màrquez, and Francisco Guzmán. 2016a. It takes three to tango: Triangulation approach to answer ranking in community question answering. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP ’16, pages 1586–1597, Austin, TX, USA.
    Google ScholarLocate open access versionFindings
  • Preslav Nakov, Lluís Màrquez, Alessandro Moschitti, Walid Magdy, Hamdy Mubarak, Abed Alhakim Freihat, Jim Glass, and Bilal Randeree. 2016b. SemEval-2016 task 3: Community question answering. In Proceedings of the International Workshop on Semantic Evaluation, SemEval ’16, pages 525– 545, San Diego, CA, USA.
    Google ScholarLocate open access versionFindings
  • Massimo Nicosia and Alessandro Moschitti. 2017. Accurate sentence matching with hybrid Siamese networks. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, CIKM ’17, pages 2235–2238, Singapore.
    Google ScholarLocate open access versionFindings
  • Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP ’14, pages 1532–1543, Doha, Qatar.
    Google ScholarLocate open access versionFindings
  • Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and beyond. Found. Trends Inf. Retr., 3(4):333–389.
    Google ScholarLocate open access versionFindings
  • Salvatore Romeo, Giovanni Da San Martino, Alberto Barrón-Cedeño, Alessandro Moschitti, Yonatan Belinkov, Wei-Ning Hsu, Yu Zhang, Mitra Mohtarami, and James Glass. 2016. Neural attention for learning to rank questions in community question answering. In Proceedings of the 26th International Conference on Computational Linguistics, COLING ’16, pages 1734–1745, Osaka, Japan.
    Google ScholarLocate open access versionFindings
  • Cicero dos Santos, Luciano Barbosa, Dasha Bogdanova, and Bianca Zadrozny. 2015. Learning hybrid representations to retrieve semantically equivalent questions. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, ACLIJCNLP ’15, pages 694–699, Beijing, China.
    Google ScholarLocate open access versionFindings
  • Antonio Uva, Daniele Bonadiman, and Alessandro Moschitti. 2018. Injecting relational structural representation in neural networks for question similarity. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL ’18, pages 285–291, Melbourne, Australia.
    Google ScholarLocate open access versionFindings
  • Kai Wang, Zhaoyan Ming, and Tat-Seng Chua. 2009. A syntactic tree matching approach to finding similar questions in community-based QA services. In Proceedings of the 32nd International ACM SIGIR
    Google ScholarLocate open access versionFindings
您的评分 :
0

 

标签
评论
数据免责声明
页面数据均来自互联网公开来源、合作出版商和通过AI技术自动分析结果,我们不对页面数据的有效性、准确性、正确性、可靠性、完整性和及时性做出任何承诺和保证。若有疑问,可以通过电子邮件方式联系我们:report@aminer.cn
小科