Enterprise Conversational AI with Retrieval-Augmented Generation: A CRM Case Study

Martin Sasinka, Martin Kotyrba, Eva Volna and Martin Pavlicek

Acta Informatica Pragensia, vol. preprint, No 1635

Abstract: Background: The integration of Large Language Models (LLMs) into enterprise information systems represents a major challenge in Artificial Intelligence (AI) and digital transformation. Retrieval-Augmented Generation (RAG) has emerged as a promising approach to improve knowledge management and reduce information overload in enterprise environments.Objective: This study aimed to design, implement, and evaluate a conversational assistant that combines RAG with fine-tuned LLMs in a Customer Relationship Management (CRM) environment. The research further sought to identify an optimal balance between performance, operational efficiency, and data governance.Methods: Guided by Task-Technology Fit theory, four open-source models (LLaMA 3.2 3B, Mistral 7B, Qwen2.5 3B, and Gemma 3 4B) were fine-tuned using QLoRA on domain-specific CRM data and compared with the commercial baseline GPT-4o-mini. Evaluation included semantic metrics (BERTScore and embedding cosine similarity), traditional n-gram metrics (BLEU and ROUGE-L), and operational indicators such as inference latency and memory footprint.Results: GPT-4o-mini achieved the highest semantic performance (BERTScore: 0.731; embedding similarity: 0.781) while maintaining low inference latency (1.89 s). Among the open-source alternatives, Mistral 7B achieved the best results (BERTScore: 0.711; similarity: 0.699). The findings demonstrate that open-source models can provide competitive performance while offering advantages in deployment flexibility and governance.Conclusion: The study provides a practical framework for organizations deploying conversational AI in enterprise information systems, supporting decisions related to Total Cost of Ownership, operational efficiency, and data governance. Future research should further validate retrieval-aware metrics against human expert assessments and user-based task evaluations to better assess the business value of enterprise RAG systems.

Keywords: RAG; Large language model; LLM; Fine-tuning; Customer relationship management; Enterprise information systems; Task-technology fit (search for similar items in EconPapers)
References: Add references at CitEc
Citations:

Downloads: (external link)
http://aip.vse.cz/doi/10.18267/j.aip.320.html (text/html)
free of charge

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:prg:jnlaip:v:preprint:id:320

Ordering information: This journal article can be ordered from
Redakce Acta Informatica Pragensia, Katedra systémové analýzy, Vysoká škola ekonomická v Praze, nám. W. Churchilla 4, 130 67 Praha 3
http://aip.vse.cz

DOI: 10.18267/j.aip.320

Access Statistics for this article

Acta Informatica Pragensia is currently edited by Editorial Office

More articles in Acta Informatica Pragensia from Prague University of Economics and Business Contact information at EDIRC.
Bibliographic data for series maintained by Stanislav Vojir ().