ETM: Modern Insights into Perspective on Text-to-SQL Evaluation in the Age of Large Language Models

Benjamin G. Ascoli (), Yasoda Sai Ram Kandikonda and Jinho D. Choi ()
Additional contact information
Benjamin G. Ascoli: Department of Computer Science, Emory University, Atlanta, GA 30322, USA
Yasoda Sai Ram Kandikonda: Department of Computer Science, Emory University, Atlanta, GA 30322, USA
Jinho D. Choi: Department of Computer Science, Emory University, Atlanta, GA 30322, USA

Future Internet, 2025, vol. 17, issue 8, 1-16

Abstract: The task of Text-to-SQL enables anyone to retrieve information from SQL databases using natural language. While this task has made substantial progress, the two primary evaluation metrics—Execution Accuracy (EXE) and Exact Set Matching Accuracy (ESM)—suffer from inherent limitations that can misrepresent performance. Specifically, ESM’s rigid matching overlooks semantically correct but stylistically different queries, whereas EXE can overestimate correctness by ignoring structural errors that yield correct outputs. These shortcomings become especially problematic when assessing outputs from large language model (LLM)-based approaches without fine-tuning, which vary more in style and structure compared with their fine-tuned counterparts. Thus, we introduce a new metric, Enhanced Tree Matching (ETM), which mitigates these issues by comparing queries using both syntactic and semantic elements. Through evaluating nine LLM-based models, we show that EXE and ESM can produce false positive and negative rates as high as 23.0% and 28.9%, while ETM reduces these rates to 0.3% and 2.7%, respectively. We release our ETM script as open-source, offering the community a more robust and reliable approach to evaluating Text-to-SQL.

Keywords: text-to-SQL; evaluation metrics; large language models (search for similar items in EconPapers)
JEL-codes: O3 (search for similar items in EconPapers)
Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/1999-5903/17/8/325/pdf (application/pdf)
https://www.mdpi.com/1999-5903/17/8/325/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jftint:v:17:y:2025:i:8:p:325-:d:1708433

Access Statistics for this article

Future Internet is currently edited by Ms. Grace You

More articles in Future Internet from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().