A novel multi-step-prompt approach for LLM-based Q&As on banking supervisory regulations
Daniele Licari (),
Canio Benedetto (),
Daniele Bovi (),
Praveen Bushipaka (),
Alessandro De Gregorio (),
Marco De Leonardis () and
Tommaso Cucinotta ()
Additional contact information
Daniele Licari: Bank of Italy
Daniele Bovi: Bank of Italy
Praveen Bushipaka: Sant'Anna School of Advanced Studies
Alessandro De Gregorio: Bank of Italy
Marco De Leonardis: Bank of Italy
Tommaso Cucinotta: Sant'Anna School of Advanced Studies
No 935, Questioni di Economia e Finanza (Occasional Papers) from Bank of Italy, Economic Research and International Relations Area
Abstract:
This paper investigates the use of large language models (LLMs) in analysing and answering questions related to banking supervisory regulations. We propose a multi-step-prompt approach that enriches the context provided to the LLM with relevant articles from the Capital Requirements Regulation (CRR). We compare our method against standard 'zero-shot' prompting, where the LLM answers are solely based on its pre-trained knowledge, and a standard 'few-shot' prompting, where the LLM is given only a limited number of examples of questions and answers to draw on each time. To assess the quality of the answers returned by the LLM, we also build an 'LLM evaluator' which, for each question, compares the correctness and completeness of the answer resulting from our multi-step prompt approach and from the two standard prompting methods with the official answer made available by the European Banking Authority (EBA), which is taken as a benchmark. Our findings on inquiries concerning Liquidity Risk rules indicate that our multi-step approach significantly improves the quality of LLM-generated answers, offering the analyst a valuable starting point to formulate appropriate answers to particularly complex questions.
Keywords: regulatory Q&A; banking supervisory regulation; Artificial Intelligence; GenAI; GPT-4o; RAG; LLM evaluator (search for similar items in EconPapers)
JEL-codes: C45 C63 G21 G28 (search for similar items in EconPapers)
Date: 2025-04
References: Add references at CitEc
Citations:
Downloads: (external link)
https://www.bancaditalia.it/pubblicazioni/qef/2025-0935/QEF_935_25.pdf (application/pdf)
Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.
Export reference: BibTeX
RIS (EndNote, ProCite, RefMan)
HTML/Text
Persistent link: https://EconPapers.repec.org/RePEc:bdi:opques:qef_935_25
Access Statistics for this paper
More papers in Questioni di Economia e Finanza (Occasional Papers) from Bank of Italy, Economic Research and International Relations Area Contact information at EDIRC.
Bibliographic data for series maintained by ().