Validating Large Language Model Annotations

Anne Lundgaard Hansen
Additional contact information
Anne Lundgaard Hansen: https://www.richmondfed.org/banking/qsr/hansen

No 2026-020, Finance and Economics Discussion Series from Board of Governors of the Federal Reserve System (U.S.)

Abstract: This paper proposes a validation framework for LLM-generated measurements when reliable benchmarks are unavailable. Validity is established by testing whether an LLM can reconstruct passages from annotated labels while maintaining semantic consistency with the original text. The framework avoids circular reasoning by establishing testable prerequisite properties that must be met for a validation to be considered successful. Application to news article data demonstrates that the framework serves as a practical alternative to human benchmarking, which offers advantages in objectivity, scalability, and cost-effectiveness while identifying cases where LLMs capture economic meaning that human evaluators miss.

Keywords: Economic measurement; Machine learning; Unstructured data; Sentiment; Computational techniques (search for similar items in EconPapers)
JEL-codes: C18 C45 C80 (search for similar items in EconPapers)
Pages: 44 p.
Date: 2026-03-30
References: Add references at CitEc
Citations:

Downloads: (external link)
https://www.federalreserve.gov/econres/feds/files/2026020pap.pdf Full text (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:fip:fedgfe:103001

DOI: 10.17016/FEDS.2026.020

Access Statistics for this paper

More papers in Finance and Economics Discussion Series from Board of Governors of the Federal Reserve System (U.S.) Contact information at EDIRC.
Bibliographic data for series maintained by Ryan Wolfslayer ; Keisha Fournillier ().