A new reader trial approach to peer review in funding research grants: An Australian experiment

Upali W. Jayasinghe, Herbert W. Marsh and Nigel Bond
Additional contact information
Upali W. Jayasinghe: University of Western Sydney
Herbert W. Marsh: University of Western Sydney; Department of Educational Studies, University of Oxford
Nigel Bond: University of Western Sydney

Scientometrics, 2006, vol. 69, issue 3, No 8, 606 pages

Abstract: Summary Peer reviews are highly valued in academic life, but are notoriously unreliable. A major problem is the substantial measurement error due to the idiosyncratic responses when large numbers of different assessors each evaluate only a single or a few submissions (e.g., journal articles, grants, etc.). To address this problem, the main funding body of academic research in Australia trialed a “reader system” in which each of a small number of senior academics read all proposals within their subdiscipline. The traditional peer review process for 1996 (2,989 proposals, 6,233 assessors) resulted in unacceptably low reliabilities comparable with those found in other research (0.475 for research project, 0.572 for researcher). For proposals from psychology and education in 1997, the new reader system resulted in substantially higher reliabilities: 0.643 and 0.881, respectively. In comparison to the traditional peer review approach, the new reader system is substantially more reliable, timely, and cost efficient - and applicable to many peer review situations.

Date: 2006
References: Add references at CitEc
Citations: View citations in EconPapers (10)

Downloads: (external link)
http://link.springer.com/10.1007/s11192-006-0171-4 Abstract (text/html)
Access to the full text of the articles in this series is restricted.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:scient:v:69:y:2006:i:3:d:10.1007_s11192-006-0171-4

Ordering information: This journal article can be ordered from
http://www.springer.com/economics/journal/11192

DOI: 10.1007/s11192-006-0171-4

Access Statistics for this article

Scientometrics is currently edited by Wolfgang Glänzel

More articles in Scientometrics from Springer, Akadémiai Kiadó
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().