Zero-shot Text-to-SQL Learning with Auxiliary Task

national conference on artificial intelligence, 2020.

Cited by: 0|Bibtex|Views112
Other Links: academic.microsoft.com|arxiv.org
Weibo:
The generation model is first improved by an attentive pooling inside the question, and bi-directional attention flow to improve the interaction between the question and table schema

Abstract:

Recent years have seen great success in the use of neural seq2seq models on the text-to-SQL task. However, little work has paid attention to how these models generalize to realistic unseen data, which naturally raises a question: does this impressive performance signify a perfect generalization model, or are there still some limitations...More

Code:

Data:

0
Introduction
  • Text-to-SQL has recently attracted much attention as a sequence-to-sequence learning problem due to its practical usage for search and question answering (Dong and Lapata, 2016; Zhong et al, 2017; Xu et al, 2017; Cai et al, 2018; Yu et al, 2018a; Dong and Lapata, 2018; FineganDollak et al, 2018; Yu et al, 2018b; Wang et al, 2017b; Shi et al, 2017).
  • Most of the previous text-to-SQL tasks assumed that all questions came from a fixed database and share one global table schema.
  • (Yu et al, 2018a) proposed to utilize high-level type information to better understand rare entities and numbers in the natural language questions and encode these information from the input
  • These type information come from either external knowledge graph, a column or a number.
  • These decoders can be classified two types: CLS for classifier, and PT for pointer
Highlights
  • Text-to-SQL has recently attracted much attention as a sequence-to-sequence learning problem due to its practical usage for search and question answering (Dong and Lapata, 2016; Zhong et al, 2017; Xu et al, 2017; Cai et al, 2018; Yu et al, 2018a; Dong and Lapata, 2018; FineganDollak et al, 2018; Yu et al, 2018b; Wang et al, 2017b; Shi et al, 2017)
  • As pointed out in (Finegan-Dollak et al, 2018), when evaluating models on text-to-SQL tasks, we need to measure how well the models generalize to realistic unseen data, which is very common in the real applications
  • Encoding Layer The question Q and corresponding table schema C are first translated into the hidden representation by a BiLSTM sentence encoder: hqt = BiLSTM(→−h qt−1, ←h−qt+1, qt, θ) hCt = BiLSTM(→−h Ct−1, ←h−Ct+1Ct, θ) where qt is embedding of question word qt and Ct is the representation of a column name Ct which consists of words c1t, · · ·, c|tCt|
  • We propose a novel auxiliary mapping task for zero-shot text-to-SQL learning
  • The generation model is first improved by an attentive pooling inside the question, and bi-directional attention flow to improve the interaction between the question and table schema
  • The mapping model serves as an enhancement model to text-to-SQL task as well as regularization to the generation model to increase its generalization
Methods
  • WikiSQL has over 20K tables and 80K questions corresponding to these tables
  • This dataset was designed for translating natural language questions to SQL queries using the corresponding table columns without access to the table content.
  • This dataset is further split into training and testing sets that are separately obtained from different Wiki pages, assuming there is no overlap of tables between training and testing sets.
  • Different football teams have their own Wiki page but each one have a table with the same schema recording match information
Results
  • Table 1 shows the overall and breakdown results on full WikiSQL dataset.
  • The authors compare the models with strong baseline models on the original WikiSQL test data.
  • All these models have no access to table content following (Zhong et al, 2017).
  • First the Gen-model with enhanced encoder/decoder improves over the baseline coarse-to-fine model by 1.6% in accuracy of both.
  • Ablation test shows the improvement is attributed to the attentive pooling in SEL decoding
Conclusion
  • Conclusions and Future Work

    In this paper, the authors propose a novel auxiliary mapping task for zero-shot text-to-SQL learning.
  • Traditional seq2seq generation model is augmented with an explicit mapping model from question words to table schema.
  • The mapping model serves as an enhancement model to text-to-SQL task as well as regularization to the generation model to increase its generalization.
  • Even though the generation model is already augmented with bi-directional attention to enhance the interaction between question and table, the results and analysis demonstrate that the explicitly mapping task can further increase the capability of generalization to unseen tables
Summary
  • Introduction:

    Text-to-SQL has recently attracted much attention as a sequence-to-sequence learning problem due to its practical usage for search and question answering (Dong and Lapata, 2016; Zhong et al, 2017; Xu et al, 2017; Cai et al, 2018; Yu et al, 2018a; Dong and Lapata, 2018; FineganDollak et al, 2018; Yu et al, 2018b; Wang et al, 2017b; Shi et al, 2017).
  • Most of the previous text-to-SQL tasks assumed that all questions came from a fixed database and share one global table schema.
  • (Yu et al, 2018a) proposed to utilize high-level type information to better understand rare entities and numbers in the natural language questions and encode these information from the input
  • These type information come from either external knowledge graph, a column or a number.
  • These decoders can be classified two types: CLS for classifier, and PT for pointer
  • Methods:

    WikiSQL has over 20K tables and 80K questions corresponding to these tables
  • This dataset was designed for translating natural language questions to SQL queries using the corresponding table columns without access to the table content.
  • This dataset is further split into training and testing sets that are separately obtained from different Wiki pages, assuming there is no overlap of tables between training and testing sets.
  • Different football teams have their own Wiki page but each one have a table with the same schema recording match information
  • Results:

    Table 1 shows the overall and breakdown results on full WikiSQL dataset.
  • The authors compare the models with strong baseline models on the original WikiSQL test data.
  • All these models have no access to table content following (Zhong et al, 2017).
  • First the Gen-model with enhanced encoder/decoder improves over the baseline coarse-to-fine model by 1.6% in accuracy of both.
  • Ablation test shows the improvement is attributed to the attentive pooling in SEL decoding
  • Conclusion:

    Conclusions and Future Work

    In this paper, the authors propose a novel auxiliary mapping task for zero-shot text-to-SQL learning.
  • Traditional seq2seq generation model is augmented with an explicit mapping model from question words to table schema.
  • The mapping model serves as an enhancement model to text-to-SQL task as well as regularization to the generation model to increase its generalization.
  • Even though the generation model is already augmented with bi-directional attention to enhance the interaction between question and table, the results and analysis demonstrate that the explicitly mapping task can further increase the capability of generalization to unseen tables
Tables
  • Table1: Overall and break down results on full WikiSQL dataset. ACCqm, ACCex are accuracy numbers of query match (ignore the order of conditions) and execution result, and ACCagg, ACCsel, ACCwhere are the accuracy of AGG, SEL, WHERE clauses. The upper part are baseline models, and the lower part are our generation model Genmodel and the whole model Full-model which is the Gen-model with the auxiliary mapping model. Gen-model w/o AP is the generation model without attentive pooling
  • Table2: Statisitics of WikiSQL test set. W-full is original WikiSQL test set and W-0, W-1,· · · , W-6 are subsets split by the number of shots (number of a table occurrences in the training data)
  • Table3: Number of samples in each error categories
Download tables as Excel
Related work
Reference
  • Ankur Bapna, Gokhan Tur, Dilek Hakkani-Tur, and Larry Heck. 2017. Towards zero-shot frame semantic parsing for domain scaling. arXiv preprint arXiv:1707.02363.
    Findings
  • Ruichu Cai, Boyan Xu, Xiaoyan Yang, Zhenjie Zhang, and Zijian Li. 2018. An encoder-decoder framework translating natural language to database queries. In Proceedings of the 27th International Joint Conference on Artificial Intelligence.
    Google ScholarLocate open access versionFindings
  • Deborah A. Dahl, Madeleine Bates, Michael Brown, William Fisher, Kate Hunicke-Smith, David Pallett, Christine Pao, Alexander Rudnicky, and Elizabeth Shriberg. 1994. Expanding the scope of the atis task: The atis-3 corpus. In Proceedings of the Workshop on Human Language Technology, pages 43–48.
    Google ScholarLocate open access versionFindings
  • Li Dong and Mirella Lapata. 2016. Language to logical form with neural attention. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 33–43.
    Google ScholarLocate open access versionFindings
  • Li Dong and Mirella Lapata. 2018. Coarse-to-fine decoding for neural semantic parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 731–742.
    Google ScholarLocate open access versionFindings
  • Cathetine Finegan-Dollak, Jonathan K. Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasivam, Rui Zhang, and Dragomir Radev. 2018. Improving text-to-sql evaluation methodology. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 351–360.
    Google ScholarLocate open access versionFindings
  • Jonathan Herzig and Jonathan Berant. 2018. Decoupling structure and lexicon for zero-shot semantic parsing. arXiv preprint arXiv:1804.07918.
    Findings
  • Po-Sen Huang, Chenglong Wang, Rishabh Singh, Wentau Yih, and Xiaodong He. 201Natural language to structured query generation via meta-learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 732–738.
    Google ScholarLocate open access versionFindings
  • Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, Jayant Krishnamurthy, and Luke Zettlemoyer. 2017. Learning a neural semantic parser from user feedback. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 963–973.
    Google ScholarLocate open access versionFindings
  • David Nadeau and Satoshi Sekine. 2007. A survey of named entity recognition and classification. Lingvisticae Investigationes, 30(1):3–26.
    Google ScholarLocate open access versionFindings
  • Arvind Neelakantan, Quoc V Le, Martin Abadi, Andrew MacCallum, and Dario Amodei. 2017. Learning a natural language interface with neural programmer. In Proceedings of the 5th International Conference on Learning Representations, pages 1– 10.
    Google ScholarLocate open access versionFindings
  • Cıcero Nogueira dos Santos, Ming Tan, Bing Xiang, and Bowen Zhou. 2016. Attentive pooling networks. CoRR.
    Google ScholarLocate open access versionFindings
  • MinJoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional attention flow for machine comprehension. In Proceedings of the 5th International Conference on Learning Representations.
    Google ScholarLocate open access versionFindings
  • Tianze Shi, Kedar Tatwawadi, Kaushik Chakrabarti, Yi Mao, and and Chen Weizhu Polozov, Oleksandr. 2017. Incsql: Training incremental text-tosql parsers with non-deterministic oracles. arXiv preprint arXiv:1809.05054v2.
    Findings
  • Chenglong Wang, Alvin Cheung, and Rastislav Bodik. 2017a. Synthesizing highly expressive sql queries from input-output examples. In ACM SIGPLAN Notices, volume 52, pages 452–466. ACM.
    Google ScholarLocate open access versionFindings
  • Chenglong Wang, Tatwawadi Kedar, Brockschmidt Marc, Huang Po-Sen, Mao Yi, Polozov Oleksandr, and Singh Rishabh. 2017b. Robust text-to-sql generation with execution-guided decoding. arXiv preprint arXiv:1807.03100v3.
    Findings
  • Xiaojun Xu, Chang Liu, and Dawn Song. 20Sqlnet: Generating structured queries from natural language without reinforcement learning. arXiv preprint arXiv:1711.04436.
    Findings
  • Pengcheng Yin and Graham Neubig. 2017. A syntactic neural model for general-purpose code generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 440– 450.
    Google ScholarLocate open access versionFindings
  • Tao Yu, Zifan Li, Zilin Zhang, Rui Zhang, and Dragomir Radev. 2018a. Typesql: knowledge-based type-aware neural text-to-sql generation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 588–594.
    Google ScholarLocate open access versionFindings
  • Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018b. Spider: A large-scale human-labeled dataset for complex and crossdomain semantic parsing and text-to-sql task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3911–3921.
    Google ScholarLocate open access versionFindings
  • John M. Zelle and Raymond J. Mooney. 1996. Learning to parse database queries using inductive logic programming. In Proceedings of the Thirteenth National Conference on Artificial Intelligence - Volume 2, pages 1050–1055.
    Google ScholarLocate open access versionFindings
  • Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2sql: Generating structured queries from natural language using reinforcement learning. arXiv preprint arXiv:1709.00103.
    Findings
  • WHERE genre = drama AND year = 1973
    Google ScholarFindings
  • 11480171-1 WHERE genre = drama AND year = 1973
    Google ScholarFindings
Full Text
Your rating :
0

 

Tags
Comments