Variability and negligence: grant peer review panels evaluating impact ex ante

Kristin Oxley and Magnus Gulbrandsen

Science and Public Policy, 2025, vol. 52, issue 2, 254-268

Abstract: Building on extensive observations of grant review panels and interviews with panellists in five funding organizations, we explore how such panels assess societal impact. We find that impact assessments are accorded little weight and are prone to substantial reviewer variability unless competitions feature dedicated societal impact criteria with clearly assigned weight, which are scored collectively based on highly structured discussions. These findings cast new light on the general phenomenon of variability in grant assessments. While earlier literature has tended to see inter-rater variability as a flaw in the review process that should be rectified, an unavoidable part of the process, or a desirable feature, we advocate a more nuanced understanding. Introducing the concepts of productive and unproductive variability, we show the dual nature of variability and provide concrete policy recommendations for how unproductive variability, caused by differing interpretations or gaming of the organizational framework, can be reduced.

Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
http://hdl.handle.net/10.1093/scipol/scae081 (application/pdf)
Access to full text is restricted to subscribers.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:oup:scippl:v:52:y:2025:i:2:p:254-268.

Access Statistics for this article

Science and Public Policy is currently edited by Nicoletta Corrocher, Jeong-Dong Lee, Mireille Matt and Nicholas Vonortas

More articles in Science and Public Policy from Oxford University Press
Bibliographic data for series maintained by Oxford University Press ().