A statistical approach to calibrating the scores of biased reviewers of scientific papers

Wiltrud Kuhlisch, Magnus Roos (), Jörg Rothe, Joachim Rudolph, Björn Scheuermann and Dietrich Stoyan

Metrika: International Journal for Theoretical and Applied Statistics, 2016, vol. 79, issue 1, 37-57

Abstract: Peer reviewing is the key ingredient of evaluating the quality of scientific work. Based on the review scores assigned by individual reviewers to papers, program committees of conferences and journal editors decide which papers to accept for publication and which to reject. A similar procedure is part of the selection process of grant applications and, among other fields, in sports. It is well known that the reviewing process suffers from measurement errors due to a lack of agreement among multiple reviewers of the same paper. And if not all papers are reviewed by all reviewers, the naive approach of averaging the scores is biased. Several statistical methods are proposed for aggregating review scores, which all can be realized by standard statistical software. The simplest method uses the well-known fixed-effects two-way classification with identical variances, while a more advanced method assumes different variances. As alternatives a mixed linear model and a generalized linear model are employed. The application of these methods implies an evaluation of the reviewers, which may help to improve reviewing processes. An application example with real conference data shows the potential of these statistical methods. Copyright Springer-Verlag Berlin Heidelberg 2016

Keywords: Analysis of variance; Model with main effects; Design matrix; Peer reviewing; Review scores; 62J10; 91B82 (search for similar items in EconPapers)
Date: 2016
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
http://hdl.handle.net/10.1007/s00184-015-0542-z (text/html)
Access to full text is restricted to subscribers.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:metrik:v:79:y:2016:i:1:p:37-57

Ordering information: This journal article can be ordered from
http://www.springer.com/statistics/journal/184/PS2

DOI: 10.1007/s00184-015-0542-z

Access Statistics for this article

Metrika: International Journal for Theoretical and Applied Statistics is currently edited by U. Kamps and Norbert Henze

More articles in Metrika: International Journal for Theoretical and Applied Statistics from Springer
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().