Photon: A Robust Cross-Domain Text-to-SQL System

Cited by: 0|Bibtex|Views99
Other Links: arxiv.org
Weibo:
We plan to improve the performance of core models in PHOTON, such as semantic parsing, response generation and context-aware user interaction

Abstract:

Natural language interfaces to databases (NLIDB) democratize end user access to relational data. Due to fundamental differences between natural language communication and programming, it is common for end users to issue questions that are ambiguous to the system or fall outside the semantic scope of its underlying query language. We pre...More

Code:

Data:

0
Introduction
Highlights
  • Natural language interfaces to databases (Popescu et al, 2003; Li and Jagadish, 2014) democratize end user access to relational data and have attracted significant research attention for decades (Hemphill et al, 1990; Dahl et al, 1994; Zelle and Mooney, 1996; Popescu et al, 2003; Bertomeu et al, 2006; Zhong et al, 2017; Yu et al, 2018, 2019a)
  • We evaluate the performance of the proposed neural semantic parser of PHOTON on the original Spider dataset
  • We present PHOTON, a robust modular crossdomain text-to-SQL system, consisting of semantic parser, untranslatable question detector, human-inthe-loop question corrector, and natural language response generator
  • PHOTON has the potential to scale up to hundreds of different domains. It is the first cross-domain text-to-SQL system designed towards industrial applications with rich features, and bridges the demand of sophisticated database analysis and people without any SQL background knowledge
  • We plan to improve the performance of core models in PHOTON, such as semantic parsing, response generation and context-aware user interaction
Methods
  • SpiderUTran is the modified dataset to evaluate robustness, created by injecting the untranslatable questions into Spider.
  • The authors obtained 5,330 additional untranslatable questions (4,733 for training and 597 for development) from the original Spider dataset.
  • To ensure the quality of the synthetic dataset, the authors hired SQL experts from Upwork to annotate the auto-generated untranslatable examples in the dev set.
  • The authors conduct the evaluation by following the database split setting, as illustrated in Table 1.
  • The split follows the original dataset there is no test set of SpiderUTran
Results
  • Confusion Detection.
  • The authors examine the robustness of PHOTON by evaluating the performance of the confusion detection module in handling ambiguous and untranslatable input.
  • The authors aim to examine if PHOTON is effective in handling untranslatable questions by measuring its translatability detection accuracy and confusing span prediction accuracy & F1 score.
  • The authors compare to a baseline that uses a single-layer attentive bidirectional LSTM (“Att-biLSTM”).
  • Table 2 shows the evaluation results on the SpiderUTran dataseet
Conclusion
  • Conclusion and Future Work

    The authors present PHOTON, a robust modular crossdomain text-to-SQL system, consisting of semantic parser, untranslatable question detector, human-inthe-loop question corrector, and natural language response generator.
  • PHOTON has the potential to scale up to hundreds of different domains.
  • It is the first cross-domain text-to-SQL system designed towards industrial applications with rich features, and bridges the demand of sophisticated database analysis and people without any SQL background knowledge.
  • The authors plan to improve the performance of core models in PHOTON, such as semantic parsing, response generation and context-aware user interaction.
  • A comprehensive evaluation will be conducted among the users of the system
Summary
  • Introduction:

    Natural language interfaces to databases (Popescu et al, 2003; Li and Jagadish, 2014) democratize end user access to relational data and have attracted significant research attention for decades (Hemphill et al, 1990; Dahl et al, 1994; Zelle and Mooney, 1996; Popescu et al, 2003; Bertomeu et al, 2006; Zhong et al, 2017; Yu et al, 2018, 2019a).
  • Most existing NLIDBs adopt a modular architecture consisting of rule-based natural language parsing, ambiguity detection and pragmatics modeling (Li and
  • Objectives:

    The authors aim to examine if PHOTON is effective in handling untranslatable questions by measuring its translatability detection accuracy and confusing span prediction accuracy & F1 score.
  • Methods:

    SpiderUTran is the modified dataset to evaluate robustness, created by injecting the untranslatable questions into Spider.
  • The authors obtained 5,330 additional untranslatable questions (4,733 for training and 597 for development) from the original Spider dataset.
  • To ensure the quality of the synthetic dataset, the authors hired SQL experts from Upwork to annotate the auto-generated untranslatable examples in the dev set.
  • The authors conduct the evaluation by following the database split setting, as illustrated in Table 1.
  • The split follows the original dataset there is no test set of SpiderUTran
  • Results:

    Confusion Detection.
  • The authors examine the robustness of PHOTON by evaluating the performance of the confusion detection module in handling ambiguous and untranslatable input.
  • The authors aim to examine if PHOTON is effective in handling untranslatable questions by measuring its translatability detection accuracy and confusing span prediction accuracy & F1 score.
  • The authors compare to a baseline that uses a single-layer attentive bidirectional LSTM (“Att-biLSTM”).
  • Table 2 shows the evaluation results on the SpiderUTran dataseet
  • Conclusion:

    Conclusion and Future Work

    The authors present PHOTON, a robust modular crossdomain text-to-SQL system, consisting of semantic parser, untranslatable question detector, human-inthe-loop question corrector, and natural language response generator.
  • PHOTON has the potential to scale up to hundreds of different domains.
  • It is the first cross-domain text-to-SQL system designed towards industrial applications with rich features, and bridges the demand of sophisticated database analysis and people without any SQL background knowledge.
  • The authors plan to improve the performance of core models in PHOTON, such as semantic parsing, response generation and context-aware user interaction.
  • A comprehensive evaluation will be conducted among the users of the system
Tables
  • Table1: Data split of Spider and SpiderUTran. Q represents the all the questions, UTran Q represents the untranslatable questions
  • Table2: Translatability prediction accuracy (“Tran Acc”) and the confusing spans prediction accuracy and F1 on our SpiderUTran dataset (%)
  • Table3: Experimental results on the Spider Dev set (%). EM Acc. denotes the exact set match accuracy
  • Table4: Types of untranslatable questions in text-to-SQL identified from manual analysis of CoSQL (<a class="ref-link" id="cYu_et+al_2019_a" href="#rYu_et+al_2019_a">Yu et al, 2019a</a>) and Multi-WOZ (<a class="ref-link" id="cBudzianowski_et+al_2018_a" href="#rBudzianowski_et+al_2018_a">Budzianowski et al, 2018</a>). A question span that is problematic for the translation is highlighted when applicable
  • Table5: Examples of question-side and schema-side transformations for generating training data for untranslatable question detection. Let Q denote the question and S denote the schema. For each transformation, we provide two examples, i.e., (Q1, S1) and (Q2, S2). The italic and bold fonts highlight phrases before and after transformations
Download tables as Excel
Related work
  • Natural Language Interfaces to Databases. NLIDBs has been studied extensively in the past decades. Thanks to the availability of large-scale datasets (Zhong et al, 2017; Finegan-Dollak et al, 2018; Yu et al, 2018), data-driven approaches have dominated the field, in which deep learning based models achieve the best performance in both strongly (Hwang et al, 2019; Zhang et al, 2019; Guo et al, 2019) and weakly (Liang et al, 2017; Min et al, 2019) supervised settings. However, most of existing text-to-SQL datasets include only questions that can be translated into a valid SQL query. Spider (Finegan-Dollak et al, 2018) specifically controlled question clarify during data collection to exclude poorly phrased and ambiguous questions. WikiSQL (Zhong et al, 2017) was constructed on top of manually written synchronous grammars, and the mapping between its questions and SQL queries can be effectively resolved via lexical matching in vector space (Hwang et al, 2019). CoSQL (Yu et al, 2019a) is by far the only existing corpus to our knowledge which entables data-driven modeling and evaluation of untranslatable question detection. Yet the dataset is of context-dependent nature and contains untranslatable questions of limited variety. We fill in this gap by proposing PHOTON to cover a diverse set of untranslatable user input in text-to-SQL.
Funding
  • We found picklist augmentation results in an absolute performance improvement of 1% on the Spider dev set
  • We also plan to improve the performance of core models in PHOTON, such as semantic parsing (text-to-SQL), response generation (table-to-text) and context-aware user interaction (text-to-text)
Study subjects and analysis
students: 4
SQL: SELECT COUNT(*), Courses.course_name FROM Student_Course _Registrations WHERE. There are 4 students registered in statistics. It is an invalid query, please check the tables and ask again. non-executable

Reference
  • Diana Archangeli. 1988. Aspects of underspecification theory. Phonology, 5(2):183–207.
    Google ScholarLocate open access versionFindings
  • Philip Arthur, Graham Neubig, Sakriani Sakti, and Satoshi Nakamura. 2016. Semantic parsing of ambiguous input using multi synchronous grammars.
    Google ScholarFindings
  • Philip Arthur, Graham Neubig, Sakriani Sakti, Tomoki Toda, and Satoshi Nakamura. 2015. Semantic parsing of ambiguous input through paraphrasing and verification. TACL, 3:571–584.
    Google ScholarLocate open access versionFindings
  • Nuria Bertomeu, Hans Uszkoreit, Anette Frank, HansUlrich Krieger, and Brigitte Jorg. 2006. Contextual phenomena and thematic relations in database QA dialogues: results from a wizard-of-Oz experiment. In Proceedings of the Interactive Question Answering Workshop at HLT-NAACL 2006, pages 1–8, New York, NY, USA. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Ben Bogin, Jonathan Berant, and Matt Gardner. 2019a. Representing schema structure with graph neural networks for text-to-sql parsing. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Vol. 1, pages 4560–4565.
    Google ScholarLocate open access versionFindings
  • Ben Bogin, Matt Gardner, and Jonathan Berant. 2019b. Global reasoning over database structures for text-tosql parsing. CoRR, abs/1908.11214.
    Findings
  • Pawel Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gasic. 2018. Multiwoz - A largescale multi-domain wizard-of-oz dataset for taskoriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 5016–5026.
    Google ScholarLocate open access versionFindings
  • DongHyun Choi, Myeong Cheol Shin, EungGyun Kim, and Dong Ryeol Shin. 2020. RYANSQL: recursively applying sketch-based slot fillings for complex text-to-sql in cross-domain databases. CoRR, abs/2004.03125.
    Findings
  • Deborah A. Dahl, Madeleine Bates, Michael Brown, William M. Fisher, Kate Hunicke-Smith, David S. Pallett, Christine Pao, Alexander I. Rudnicky, and Elizabeth Shriberg. 1994. Expanding the scope of the ATIS task: The ATIS-3 corpus. In Human Language Technology, Proceedings of a Workshop held at Plainsboro, New Jerey, USA, March 8-11, 1994.
    Google ScholarLocate open access versionFindings
  • Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1, pages 4171–4186.
    Google ScholarLocate open access versionFindings
  • Li Dong and Mirella Lapata. 2016. Language to logical form with neural attention. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics.
    Google ScholarLocate open access versionFindings
  • Li Dong and Mirella Lapata. 2018. Coarse-to-fine decoding for neural semantic parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 731–742.
    Google ScholarLocate open access versionFindings
  • Li Dong, Chris Quirk, and Mirella Lapata. 2018. Confidence modeling for neural semantic parsing. In (Gurevych and Miyao, 2018), pages 743–753.
    Google ScholarLocate open access versionFindings
  • Catherine Finegan-Dollak, Jonathan K. Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasivam, Rui Zhang, and Dragomir R. Radev. 2018. Improving text-to-sql evaluation methodology. In (Gurevych and Miyao, 2018), pages 351–360.
    Google ScholarLocate open access versionFindings
  • Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao, Jian-Guang Lou, Ting Liu, and Dongmei Zhang. 2019. Towards complex text-to-sql in cross-domain database with intermediate representation. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4524–4535.
    Google ScholarLocate open access versionFindings
  • Iryna Gurevych and Yusuke Miyao, editors. 2018. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Charles T. Hemphill, John J. Godfrey, and George R. Doddington. 1990. The ATIS spoken language systems pilot corpus. In Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, USA, June 24-27, 1990.
    Google ScholarLocate open access versionFindings
  • Wonseok Hwang, Jinyeung Yim, Seunghyun Park, and Minjoon Seo. 2019. A comprehensive exploration on wikisql with table-aware word contextualization. CoRR, abs/1902.01069.
    Findings
  • Amol Kelkar, Rohan Relan, Vaishali Bhardwaj, Saurabh Vaichal, and Peter Relan. 2020. Bertranddr: Improving text-to-sql using a discriminative reranker. arXiv preprint arXiv:2002.00557.
    Findings
  • Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
    Google ScholarLocate open access versionFindings
  • Anna Korhonen, David R. Traum, and Lluıs Marquez, editors. 2019. Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Fei Li and Hosagrahar Visvesvaraya Jagadish. 2014. Nalir: an interactive natural language interface for querying relational databases. In International Conference on Management of Data, SIGMOD 2014, Snowbird, UT, USA, June 22-27, 2014, pages 709– 712. ACM.
    Google ScholarLocate open access versionFindings
  • Chen Liang, Jonathan Berant, Quoc V. Le, Kenneth D. Forbus, and Ni Lao. 2017. Neural symbolic machines: Learning semantic parsers on freebase with weak supervision. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 August 4, Volume 1: Long Papers, pages 23–33. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Abigail See, Peter J. Liu, and Christopher D. Manning. 2017b. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073–1083, Vancouver, Canada. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Vidya Setlur, Sarah E. Battersby, Melanie Tory, Rich Gossweiler, and Angel X. Chang. 2016. Eviza: A natural language interface for visual analysis. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology, UIST 2016, Tokyo, Japan, October 16-19, 2016, pages 365–377. ACM.
    Google ScholarLocate open access versionFindings
  • Kevin Lin, Ben Bogin, Mark Neumann, Jonathan Berant, and Matt Gardner. 2019. Grammar-based neural text-to-sql generation. CoRR, abs/1905.13326.
    Findings
  • Tianyu Liu, Fuli Luo, Pengcheng Yang, Wei Wu, Baobao Chang, and Zhifang Sui. 2019. Towards comprehensive description generation from factual attribute-value tables. In (Korhonen et al., 2019), pages 5985–5996.
    Google ScholarLocate open access versionFindings
  • Sewon Min, Danqi Chen, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2019. A discrete hard EM approach for weakly supervised question answering. CoRR, abs/1909.04849.
    Findings
  • Ana-Maria Popescu, Oren Etzioni, and Henry A. Kautz. 2003. Towards a theory of natural language interfaces to databases. In Proceedings of the 8th International Conference on Intelligent User Interfaces, IUI 2003, Miami, FL, USA, January 12-15, 2003, pages 149–157. ACM.
    Google ScholarLocate open access versionFindings
  • Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for squad. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 1520, 2018, Volume 2: Short Papers, pages 784–789.
    Google ScholarLocate open access versionFindings
  • Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100, 000+ questions for machine comprehension of text. ArXiv, abs/1606.05250.
    Findings
  • Kirk Roberts and Braja Gopal Patra. 2017. A semantic parsing method for mapping clinical questions to logical forms. In AMIA 2017, American Medical Informatics Association Annual Symposium, Washington, DC, USA, November 4-8, 2017. AMIA.
    Google ScholarFindings
  • Abigail See, Peter J. Liu, and Christopher D. Manning. 2017a. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers, pages 1073–1083.
    Google ScholarLocate open access versionFindings
  • Vidya Setlur, Melanie Tory, and Alex Djalali. 2019. Inferencing underspecified natural language utterances in visual analysis. In Proceedings of the 24th International Conference on Intelligent User Interfaces, IUI 2019, Marina del Ray, CA, USA, March 17-20, 2019, pages 40–51. ACM.
    Google ScholarLocate open access versionFindings
  • Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pages 3104–3112.
    Google ScholarLocate open access versionFindings
  • Achille C Varzi. 2001.
    Google ScholarFindings
  • Chenglong Wang, Po-Sen Huang, Alex Polozov, Marc Brockschmidt, and Rishabh Singh. 2018. Execution-guided neural program decoding. CoRR, abs/1807.03100.
    Findings
  • Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.
    Findings
  • Ziyu Yao, Yu Su, Huan Sun, and Wen-tau Yih. 2019. Model-based interactive semantic parsing: A unified framework and A text-to-sql case study. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 5446–5457. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Tao Yu, Rui Zhang, Heyang Er, Suyi Li, Eric Xue, Bo Pang, Xi Victoria Lin, Yi Chern Tan, Tianze Shi, Zihan Li, Youxuan Jiang, Michihiro Yasunaga, Sungrok Shim, Tao Chen, Alexander Richard Fabbri, Zifan Li, Luyao Chen, Yuwen Zhang, Shreya Dixit, Vincent Zhang, Caiming Xiong, Richard Socher, Walter S. Lasecki, and Dragomir R. Radev. 2019a. Cosql: A conversational text-to-sql challenge towards cross-domain natural language interfaces to databases. CoRR, abs/1909.05378.
    Findings
  • Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir R. Radev. 2018. Spider: A largescale human-labeled dataset for complex and crossdomain semantic parsing and text-to-sql task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, Oct 31 - Nov 4, 2018, pages 3911–3921.
    Google ScholarLocate open access versionFindings
  • Tao Yu, Rui Zhang, Michihiro Yasunaga, Yi Chern Tan, Xi Victoria Lin, Suyi Li, Heyang Er, Irene Li, Bo Pang, Tao Chen, Emily Ji, Shreya Dixit, David Proctor, Sungrok Shim, Jonathan Kraft, Vincent Zhang, Caiming Xiong, Richard Socher, and Dragomir R. Radev. 2019b. Sparc: Cross-domain semantic parsing in context. In (Korhonen et al., 2019), pages 4511–4523.
    Google ScholarLocate open access versionFindings
  • John M. Zelle and Raymond J. Mooney. 1996. Learning to parse database queries using inductive logic programming. In Proceedings of the Thirteenth National Conference on Artificial Intelligence and Eighth Innovative Applications of Artificial Intelligence Conference, AAAI 96, IAAI 96, Portland, Oregon, USA, August 4-8, 1996, Volume 2., pages 1050– 1055.
    Google ScholarLocate open access versionFindings
  • Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. SWAG: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 93– 104. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Luke S. Zettlemoyer and Michael Collins. 2005. Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. In UAI ’05, Proceedings of the 21st Conference in Uncertainty in Artificial Intelligence, Edinburgh, Scotland, July 26-29, 2005, pages 658–666. AUAI Press.
    Google ScholarLocate open access versionFindings
  • Rui Zhang, Tao Yu, Heyang Er, Sungrok Shim, Eric Xue, Xi Victoria Lin, Tianze Shi, Caiming Xiong, Richard Socher, and Dragomir R. Radev. 2019. Editing-based SQL query generation for cross-domain context-dependent questions. CoRR, abs/1909.00786.
    Findings
  • Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2sql: Generating structured queries from natural language using reinforcement learning. CoRR, abs/1709.00103.
    Findings
  • Table 4 shows a summary of different types of untranslatable questions based on analysis of CoSQL (Yu et al., 2019a) and Multi-WOZ (Budzianowski et al., 2018).
    Google ScholarFindings
Full Text
Your rating :
0

 

Tags
Comments