Retriever or reasoner? Decomposing retrieval-augmented generation performance in external audit supervision

Andrés Alonso-Robisco, José Manuel Carbó, Carlos José García, Jorge Quintana and Javier Tarancón
Additional contact information
Andrés Alonso-Robisco: Banco de España
José Manuel Carbó: Banco de España
Carlos José García: Banco de España
Jorge Quintana: Banco de España
Javier Tarancón: Banco de España

No 2613, Occasional Papers from Banco de España

Abstract: Supervisory reviews of external audit reports require structured evidence extraction and judgement under strict confidentiality and governance constraints. We evaluate retrieval-augmented generation pipelines for this task, comparing lexical, semantic, hybrid, and oracle retrieval across on-premise open-weight models (Llama 3B, Mistral 7B, Llama 70B) and proprietary cloud models (Kimi, Claude Sonnet 4.6). Using 20 bank audit reports and a standardized central bank template of 30 questions, we score operational correctness against supervisor-provided ground truth with an independent LLM-as-judge, complemented by expert back-to-back checks. The paired design holds each question report pair fixed, separating gains driven by retrieval quality from those driven by model capability, and identifying when they operate as complements rather than substitutes. Semantic retrieval yields a sizeable and statistically robust uplift within fixed models. Under symmetric strong retrieval, Llama 70B becomes statistically indistinguishable from the best cloud benchmark, while smaller on-premise models remain constrained on higher complexity judgement questions. The results point to a capacity threshold for practical substitution and provide evidence to guide deployment trade-offs in regulated settings.

Keywords: large language models; retrieval-augmented generation; financial supervision; audit (search for similar items in EconPapers)
JEL-codes: C88 G28 M42 (search for similar items in EconPapers)
Pages: 37 pages
Date: 2026-07
References: Add references at CitEc
Citations:

Downloads: (external link)
https://www.bde.es/f/webbe/SES/Secciones/Publicaci ... 26/Files/do2613e.pdf First version, July 2026 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:bde:opaper:2613e

DOI: 10.53479/43986

Access Statistics for this paper

More papers in Occasional Papers from Banco de España Contact information at EDIRC.
Bibliographic data for series maintained by Ángel Rodríguez. Electronic Dissemination of Information Unit. Research Department. Banco de España ().