An exploration of referees’ comments published in open peer review journals: The characteristics of review language and the association between review scrutiny and citations

Peer Review for Journals: Evidence on Quality Control, Fairness, and Innovation

Dietmar Wolfram, Peiling Wang and Fuad Abuzahra

Research Evaluation, 2021, vol. 30, issue 3, 314-322

Abstract: Journals that adopt open peer review (OPR), where review reports of published articles are publicly available, provide an opportunity to study both review content characteristics and quantitative aspects of the overall review process. This study investigates two areas relevant to the quality assessment of manuscript reviews. First, do journal policies for reviewers to identify themselves influence how reviewers evaluate the merits of a manuscript based on the relative frequency of hedging terms and research-related terms appearing in their reviews? Second, is there an association between the number of reviews/reviewers and the manuscript’s research impact once published as measured by citations? We selected reviews for articles published in 17 OPR journals from 2017 to 2018 to examine the incidence of reviewers’ uses of hedging terms and research-related terms. The results suggest that there was little difference in the relative use of hedging term usage regardless of whether reviewers were required to identify themselves or if this was optional, indicating that the use of hedging in review contents was not influenced by journal requirements for reviewers to identify themselves. There was a larger difference observed for research-related terminology. We compared the total number of reviews for a manuscript, rounds of revisions, and the number of reviewers with the number of Web of Science citations the article received since publication. The findings reveal that scrutiny by more reviewers or conducting more reviews or rounds of review do not result in more impactful papers for most of the journals studied. Implications for peer review practice are discussed.

Keywords: peer review; open peer review; scholarly communication; language use (search for similar items in EconPapers)
Date: 2021
References: Add references at CitEc
Citations: View citations in EconPapers (2)

Downloads: (external link)
http://hdl.handle.net/10.1093/reseval/rvab005 (application/pdf)
Access to full text is restricted to subscribers.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:oup:rseval:v:30:y:2021:i:3:p:314-322.

Access Statistics for this article

Research Evaluation is currently edited by Julia Melkers, Emanuela Reale and Thed van Leeuwen

More articles in Research Evaluation from Oxford University Press
Bibliographic data for series maintained by Oxford University Press ().