Field Experiments in the Science of Science: Lessons from Peer Review and the Evaluation of New Knowledge

Kevin Boudreau

No 34811, NBER Working Papers from National Bureau of Economic Research, Inc

Abstract: Scientific evaluation and peer review govern the allocation of resources and the certification of knowledge in science, yet have rarely been studied causally. This chapter is an analytic synthesis of randomized experiments, re-evaluation studies, controlled comparisons and audit studies of scientific evaluation, most conducted within live evaluation systems at journals, conferences, and funding agencies and others in evaluations reconstructed for research purposes, published through 2025. I reconstruct each design in detail where the record allows, and representative designs where studies cluster, to establish what each experiment can and cannot identify. I develop a Q–A–R–S framework for analyzing the studies’ designs and results in common terms. It separates submission quality, author attributes, reviewer expertise, and the evaluation system, and foregrounds a basic constraint: new knowledge must be judged with an existing and incomplete stock of knowledge. Read together, the experiments suggest three claims of different standing: the first describes what the record establishes, the second rests on experimentally identified effects, and the third is a comparative hypothesis assembled across designs. First, system interventions often change participant behavior, and sometimes scores and decisions; the experiments establish these responses more clearly than any improvement in evaluative judgment. Second, who reviews, and what they know relative to the submission, systematically shapes evaluations of the same work. Third, the evidence is consistent with peer review performing more dependably as a filter of weak submissions than as a ranker of strong ones. The record leaves much of the design space unexplored, especially how individual evaluations are combined into decisions. I set out an agenda for a more cumulative experimental science of evaluation, including the use of artificial intelligence to experiment with evaluation design itself.

JEL-codes: A14 C93 D02 D83 G18 I23 O31 O38 (search for similar items in EconPapers)
Date: 2026-02
New Economics Papers: this item is included in nep-exp and nep-sog
Note: PR
References: Add references at CitEc
Citations:

Forthcoming: Field Experiments in the Science of Science: Lessons from Peer Review and the Evaluation of New Knowledge , Kevin J. Boudreau. in The Economics of Science: Taking Stock and Looking Ahead , MacGarvie and Veugelers. 2026

Downloads: (external link)
http://www.nber.org/papers/w34811.pdf (application/pdf)
Access to the full text is generally limited to series subscribers, however if the top level domain of the client browser is in a developing country or transition economy free access is provided. More information about subscriptions and free access is available at http://www.nber.org/wwphelp.html. Free access is also available to older working papers.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:nbr:nberwo:34811

Ordering information: This working paper can be ordered from
http://www.nber.org/papers/w34811
The price is Paper copy available by mail.

Access Statistics for this paper

More papers in NBER Working Papers from National Bureau of Economic Research, Inc National Bureau of Economic Research, 1050 Massachusetts Avenue Cambridge, MA 02138, U.S.A.. Contact information at EDIRC.
Bibliographic data for series maintained by ().