The Wrecking SQL Incremental Validation Methodology.

Large language models (LLMs) have made significant strides in text-to-SQL performance, transforming natural language queries into SQL queries. The BIRD benchmark, a cross-domain dataset with 12,751 question-SQL pairs across 95 databases, is currently the most challenging benchmark in the field. However, its leaderboard is dominated by solutions that rely on closed-source LLMs, creating a financial barrier for researchers. This paper focuses on leveraging open-source models and presents an incremental validation methodology, called Wrecking SQL, that in six steps, incrementally modifies the schema of datasets by replacing meaningful column and table names with meaningless ones-a realworld problem found in legacy SQL systems. We explore how meaningless names affect LLM accuracy and demonstrate that inferring meaningful names improves translation accuracy.