Evaluating Local Language Models: An Application to Bank Earnings Calls

Thomas Cook, Sophia Kazinnik, Anne Lundgaard Hansen and Peter McAdam
Additional contact information
Anne Lundgaard Hansen: https://www.richmondfed.org/banking/qsr/hansen

No RWP 23-12, Research Working Paper from Federal Reserve Bank of Kansas City

Abstract: This study evaluates the performance of local large language models (LLMs) in interpreting financial texts, compared with closed-source, cloud-based models. We first introduce new benchmarking tasks for assessing LLM performance in analyzing financial and economic texts and explore the refinements needed to improve its performance. Our benchmarking results suggest local LLMs are a viable tool for general natural language processing analysis of these texts. We then leverage local LLMs to analyze the tone and substance of bank earnings calls in the post-pandemic era, including calls conducted during the banking stress of early 2023. We analyze remarks in bank earnings calls in terms of topics discussed, overall sentiment, temporal orientation, and vagueness. We find that after the banking stress in early 2023, banks tended to converge to a similar set of topics for discussion and to espouse a distinctly less positive sentiment.

Keywords: data; large language models; quantitative methods; banking and finance (search for similar items in EconPapers)
JEL-codes: C45 G21 (search for similar items in EconPapers)
Pages: 53
Date: 2023-11-06
New Economics Papers: this item is included in nep-ain, nep-ban, nep-big and nep-cmp
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (2)

Downloads: (external link)
https://www.kansascityfed.org/Research%20Working%2 ... nnikhansenmcadam.pdf Full Text (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:fip:fedkrw:97255

Ordering information: This working paper can be ordered from

Access Statistics for this paper

More papers in Research Working Paper from Federal Reserve Bank of Kansas City Contact information at EDIRC.
Bibliographic data for series maintained by Zach Kastens ().