ChatGPT for GTFS: benchmarking LLMs on GTFS semantics... and retrieval

The General Transit Feed Specification (GTFS) standard for publishing transit data is ubiquitous. With the advent of LLMs being used widely, this research explores the possibility of extracting transit information from GTFS through natural language instructions. To evaluate the capabilities and limitations of LLMs, we introduce two benchmarks, namely “GTFS Semantics” and “GTFS Retrieval” that test how well LLMs can “understand” GTFS standards and retrieve relevant transit information. We benchmark OpenAI’s GPT-3.5 Turbo and GPT-4 LLMs, which are backends for the ChatGPT interface. In particular, we use zero-shot, one-shot, chain of thought, and program synthesis techniques with prompt engineering. For our multiple questions, GPT-3.5 Turbo answers 59.7