A novel multi-step-prompt approach for LLM-based Q&As on banking supervisory regulations

Daniele Licari (), Canio Benedetto (), Daniele Bovi (), Praveen Bushipaka (), Alessandro De Gregorio (), Marco De Leonardis () and Tommaso Cucinotta ()
Additional contact information
Daniele Licari: Bank of Italy
Daniele Bovi: Bank of Italy
Praveen Bushipaka: Sant'Anna School of Advanced Studies
Alessandro De Gregorio: Bank of Italy
Marco De Leonardis: Bank of Italy
Tommaso Cucinotta: Sant'Anna School of Advanced Studies

No 935, Questioni di Economia e Finanza (Occasional Papers) from Bank of Italy, Economic Research and International Relations Area

Abstract: This paper investigates the use of large language models (LLMs) in analysing and answering questions related to banking supervisory regulations. We propose a multi-step-prompt approach that enriches the context provided to the LLM with relevant articles from the Capital Requirements Regulation (CRR). We compare our method against standard 'zero-shot' prompting, where the LLM answers are solely based on its pre-trained knowledge, and a standard 'few-shot' prompting, where the LLM is given only a limited number of examples of questions and answers to draw on each time. To assess the quality of the answers returned by the LLM, we also build an 'LLM evaluator' which, for each question, compares the correctness and completeness of the answer resulting from our multi-step prompt approach and from the two standard prompting methods with the official answer made available by the European Banking Authority (EBA), which is taken as a benchmark. Our findings on inquiries concerning Liquidity Risk rules indicate that our multi-step approach significantly improves the quality of LLM-generated answers, offering the analyst a valuable starting point to formulate appropriate answers to particularly complex questions.

Keywords: regulatory Q&A; banking supervisory regulation; Artificial Intelligence; GenAI; GPT-4o; RAG; LLM evaluator (search for similar items in EconPapers)
JEL-codes: C45 C63 G21 G28 (search for similar items in EconPapers)
Date: 2025-04
References: Add references at CitEc
Citations:

Downloads: (external link)
https://www.bancaditalia.it/pubblicazioni/qef/2025-0935/QEF_935_25.pdf (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:bdi:opques:qef_935_25

Access Statistics for this paper

More papers in Questioni di Economia e Finanza (Occasional Papers) from Bank of Italy, Economic Research and International Relations Area Contact information at EDIRC.
Bibliographic data for series maintained by ().