Total Recall? Evaluating the Macroeconomic Knowledge of Large Language Models

Leland Crane, Akhil Karra and Paul E. Soto
Additional contact information
Paul E. Soto: https://www.federalreserve.gov/econres/paul-e-soto.htm

No 2025-044, Finance and Economics Discussion Series from Board of Governors of the Federal Reserve System (U.S.)

Abstract: We evaluate the ability of large language models (LLMs) to estimate historical macroeconomic variables and data release dates. We find that LLMs have precise knowledge of some recent statistics, but performance degrades as we go farther back in history. We highlight two particularly important kinds of recall errors: mixing together first print data with subsequent revisions (i.e., smoothing across vintages) and mixing data for past and future reference periods (i.e., smoothing within vintages). We also find that LLMs can often recall individual data release dates accurately, but aggregating across series shows that on any given day the LLM is likely to believe it has data in hand which has not been released. Our results indicate that while LLMs have impressively accurate recall, their errors point to some limitations when used for historical analysis or to mimic real time forecasters.

Keywords: Artificial intelligence; Forecasting; Large language models; Real-time data (search for similar items in EconPapers)
JEL-codes: C53 C80 E37 (search for similar items in EconPapers)
Pages: 48 p.
Date: 2025-06-25
New Economics Papers: this item is included in nep-ain and nep-his
References: Add references at CitEc
Citations:

Downloads: (external link)
https://www.federalreserve.gov/econres/feds/files/2025044pap.pdf (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:fip:fedgfe:2025-44

DOI: 10.17016/FEDS.2025.044

Access Statistics for this paper

More papers in Finance and Economics Discussion Series from Board of Governors of the Federal Reserve System (U.S.) Contact information at EDIRC.
Bibliographic data for series maintained by Ryan Wolfslayer ; Keisha Fournillier ().