Consistency index: measuring the performances of scholar journal reviewers

Mingliang Yue, Hongbo Tang, Fan Liu () and Tingcan Ma ()
Additional contact information
Mingliang Yue: Wuhan Library of Chinese Academy of Science
Hongbo Tang: Wuhan Library of Chinese Academy of Science
Fan Liu: Wuhan University
Tingcan Ma: Wuhan Library of Chinese Academy of Science

Scientometrics, 2021, vol. 126, issue 8, No 35, 7183-7195

Abstract: Abstract Peer review is one of the most important processes of scientific evaluation. This paper proposes a consistency index to measure the performances of scholar journal reviewers on recognizing papers with higher scientific influences with respect to citations. The index is initially designed based on three consistencies, i.e., expert consistency, editor consistency and academic community consistency, to capture both the knowledge implied in the review data and the bibliometric data. An iterative computation process is then proposed to make the consistency flow among experts based on their historical performances. Further, a quantitative indicator is designed to characterize the ability of an expert of recognizing papers with higher quality during review process. Experiments are conducted to verify the convergence of the iterative computation, and to evaluate the effectiveness of the consistency index. The experimental results show that the performance of the top experts (with regard to consistency index) is always better than the tail experts, and the performance can be maintained over time. We believe that the proposed index can help journal editors get deeper understanding to their reviewers, and provide evidence in the process of reviewer selection.

Keywords: Scholar journal reviewers; Reviewer evaluation; Iterative computation; Quantitative analysis (search for similar items in EconPapers)
Date: 2021
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
http://link.springer.com/10.1007/s11192-021-04013-x Abstract (text/html)
Access to the full text of the articles in this series is restricted.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:scient:v:126:y:2021:i:8:d:10.1007_s11192-021-04013-x

Ordering information: This journal article can be ordered from
http://www.springer.com/economics/journal/11192

DOI: 10.1007/s11192-021-04013-x

Access Statistics for this article

Scientometrics is currently edited by Wolfgang Glänzel

More articles in Scientometrics from Springer, Akadémiai Kiadó
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().