Does Multi-Agent Debate Improve AI Feedback on Research Papers?

Tomas Havranek and Zuzana Irsova

No 2026/19, Working Papers IES from Charles University Prague, Faculty of Social Sciences, Institute of Economic Studies

Abstract: Probably not, at least for meta-analyses in economics. In a pre-registered, identity-masked, within-paper experiment, the authors of 44 meta-analyses ranked three AI reports on their own paper by usefulness for improving it: a single pass by a frontier model against two multi-agent debate tools we built and expected to win. All reports were held to a common length and template. The authors preferred the single pass, by 0.66 rank points over mad-research (95% CI 0.32 to 1.00) and 0.57 over paper-workshop (0.16 to 0.95), though paper-workshop spent roughly thirty times the tokens. Authors who recalled their journal referee report usually placed it first and never last; in a separate exercise, three AI judges almost always placed the real journal referee report last. Among the three AI reports, Gemini (the judge whose model family wrote none of the reports) would have ranked paper-workshop first in the authors’ place, reversing the single-pass preference. The reversal warns against substituting an AI judge for the author. We measure perceived usefulness for finished papers; whether AI should referee papers is a separate question.

Keywords: LLM-as-a-judge; meta-science; multi-agent debate; pre-registration; test-time compute (search for similar items in EconPapers)
JEL-codes: C18 C93 O33 (search for similar items in EconPapers)
Pages: 33 pages
Date: 2026-07, Revised 2026-07
References: Add references at CitEc
Citations:

Downloads: (external link)
https://ies.fsv.cuni.cz/en/does-multi-agent-debate ... back-research-papers (application/pdf)

Related works:
Working Paper: Does Multi-Agent Debate Improve AI Feedback on Research Papers? (2026)
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:fau:wpaper:wp2026_19

Access Statistics for this paper

More papers in Working Papers IES from Charles University Prague, Faculty of Social Sciences, Institute of Economic Studies Contact information at EDIRC.
Bibliographic data for series maintained by Natalie Svarcova ().