How Effectively Can Current LLMs Analyze Macrofinancial Issues?

Paola Ganum and Tohid Atashbar

No 2026/035, IMF Working Papers from International Monetary Fund

Abstract: This paper empirically evaluates the ability of current Large Language Models (LLMs) to analyze macrofinancial coverage in IMF Article IV staff reports, using human economists' assessments as a benchmark. We test several GPT models on reports from 2016-2024, assessing their performance on both qualitative ratings and binary questions. Our findings indicate that the latest models can meaningfully assist economists, achieving an average accuracy of 71-75% on ratings and an average exact match rate of 76-81% on binary questions in 2024 across advanced GPT models. However, we find that LLMs tend to assign higher, less-dispersed ratings than human experts and struggle with open-ended questions that require deep contextual judgment. The paper provides quantitative evidence on current LLM accuracy in this domain, explores the drivers of its performance, and discusses key limitations such as optimistic bias.

Keywords: AI; Large Language Model; Textual Analysis; Macrofinancial Surveillance; IMF Staff Reports; Human-AI Comparison; IMF working papers; IMF seminar; IMF article IV; IMF staff; Large Language Models; Financial Sector Assessment Program; Macrofinancial analysis; Systemic risk (search for similar items in EconPapers)
Pages: 36
Date: 2026-02-27
New Economics Papers: this item is included in nep-ain, nep-big and nep-cmp
References: Add references at CitEc
Citations:

Downloads: (external link)
http://www.imf.org/external/pubs/cat/longres.aspx?sk=574289 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:imf:imfwpa:2026/035

Ordering information: This working paper can be ordered from
http://www.imf.org/external/pubs/pubs/ord_info.htm

Access Statistics for this paper

More papers in IMF Working Papers from International Monetary Fund International Monetary Fund, Washington, DC USA. Contact information at EDIRC.
Bibliographic data for series maintained by Akshay Modi ().