Peer-review of grant proposals. An analysis of Economic and Social Research Council grant applications

John Jerrim ()
Additional contact information
John Jerrim: Department of Social Science, Institute of Education, University College London

No 19-05, DoQSS Working Papers from Quantitative Social Science - UCL Social Research Institute, University College London

Abstract: Peer-review is widely used throughout academia, most notably in the publication of journal articles and the allocation of research grants. Yet peer-review has been subject to much criticism, including being slow, unreliable, subjective and potentially prone to bias. This paper contributes to this literature by investigating the consistency of peer-reviews and the impact they have upon a high-stakes outcome (whether a research grant is funded). Analysing data from 4,000 social science grant proposals and 15,000 reviews, this paper illustrates how the peer-review scores assigned by different reviewers have only low levels of consistency (a correlation between reviewer scores of only 0.2). Reviews provided by ‘nominated reviewers’ (i.e. reviewers selected by the grant applicant) appear to be overly generous and do not correlate with the evaluations provided by independent reviewers. Yet a positive review from a nominated reviewer is strongly linked to whether a grant is awarded. Finally, a single negative peer-review is shown to reduce the chances of a proposal being funding from around 55% to around 25% (even when it has otherwise been rated highly).

Keywords: Peer-review; reliability; grants; scientific funding (search for similar items in EconPapers)
Date: 2019-07-19
New Economics Papers: this item is included in nep-sog
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://repec.ucl.ac.uk/REPEc/pdf/qsswp1905.pdf (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:qss:dqsswp:1905

Access Statistics for this paper

More papers in DoQSS Working Papers from Quantitative Social Science - UCL Social Research Institute, University College London Quantitative Social Science, Social Research Institute, 55-59 Gordon Square, London WC1H 0NU. Contact information at EDIRC.
Bibliographic data for series maintained by Dr Neus Bover Fonts ().