Can AI Solve the Peer Review Crisis? A Large-Scale Experiment on LLM's Performance and Biases in Evaluating Economics Papers

Pat Pataranutaporn, Nattavudh Powdthavee and Pattie Maes
Additional contact information
Pat Pataranutaporn: Massachusetts Institute of Technology
Pattie Maes: Massachusetts Institute of Technology

No 17659, IZA Discussion Papers from Institute of Labor Economics (IZA)

Abstract: We investigate whether artificial intelligence can address the peer review crisis in economics by analyzing 27,090 evaluations of 9,030 unique submissions using a large language model (LLM). The experiment systematically varies author characteristics (e.g., affiliation, reputation, gender) and publication quality (e.g., top-tier, mid-tier, low-tier, AI-generated papers). The results indicate that LLMs effectively distinguish paper quality but exhibit biases favoring prominent institutions, male authors, and renowned economists. Additionally, LLMs struggle to differentiate high-quality AI-generated papers from genuine top-tier submissions. While LLMs offer efficiency gains, their susceptibility to bias necessitates cautious integration and hybrid peer review models to balance equity and accuracy.

Keywords: Artificial Intelligence; peer review; large language model (LLM); bias in academia; economics publishing; equity-efficiency trade-off (search for similar items in EconPapers)
JEL-codes: A11 C63 I23 O33 (search for similar items in EconPapers)
Pages: 73 pages
Date: 2025-01
New Economics Papers: this item is included in nep-ain, nep-big, nep-cmp, nep-exp, nep-sea and nep-sog
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://docs.iza.org/dp17659.pdf (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:iza:izadps:dp17659

Ordering information: This working paper can be ordered from
IZA, Margard Ody, P.O. Box 7240, D-53072 Bonn, Germany

Access Statistics for this paper

More papers in IZA Discussion Papers from Institute of Labor Economics (IZA) IZA, P.O. Box 7240, D-53072 Bonn, Germany. Contact information at EDIRC.
Bibliographic data for series maintained by Holger Hinte ().