Optimizing Customer Support Using Text2SQL to Query Natural Language Databases

Michal Maj, Damian Pliszczuk, Patryk Marek, Weronika Wilczewska, Bartosz Przysucha and Tomasz Rymarczyk

European Research Studies Journal, 2024, vol. XXVII, issue Special B, 426-438

Abstract: Purpose: This paper explores the challenges and potential solutions associated with integrating Text2SQL technology into customer support operations. By leveraging large language models (LLMs) and tools like Vanna.AI, the study aims to enhance the efficiency and accuracy of handling customer queries without requiring specialized SQL knowledge. Design/Methodology/Approach: A comprehensive analysis was conducted comparing the effectiveness of three large language models—Llama3:70b-instruct, Gemma2:27b, and Codegemma—in generating correct SQL queries from natural language questions. The models were trained with identical datasets and evaluated using six benchmark questions over two iterations, with and without detailed database schema information. Performance metrics included correctness of the generated queries and response times. Findings: The results indicated that while Llama3 and Gemma2 initially demonstrated higher accuracy, the addition of detailed database schema information did not improve model performance. Instead, it led to decreased accuracy and increased response times, particularly for Llama3. Codegemma showed shorter response times but slightly lower accuracy. The study highlights that excessive contextual information can overwhelm LLMs, suggesting the need for optimized context provision. Practical Implications: The findings suggest that simplifying database schema information and focusing on essential contextual data can enhance the performance of LLMs in generating SQL queries. Implementing tools like Vanna.AI, which utilize Retrieval Augmented Generation (RAG), can improve customer support processes by enabling quick and accurate data access without specialized SQL expertise. Originality/Value: This paper provides valuable insights into the practical challenges of implementing Text2SQL technology in customer support. It offers recommendations for balancing context provision and model capabilities, contributing to the optimization of LLM performance in real-world applications.

Keywords: Text2SQL; Customer Support; Large Language Models; Vanna.AI; Retrieval Augmented Generation; SQL Query Generation. (search for similar items in EconPapers)
JEL-codes: C45 C61 D83 L8 M31 (search for similar items in EconPapers)
Date: 2024
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://ersj.eu/journal/3498/download (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:ers:journl:v:xxvii:y:2024:i:specialb:p:426-438

Access Statistics for this article

More articles in European Research Studies Journal from European Research Studies Journal
Bibliographic data for series maintained by Marios Agiomavritis ().