Please be polite to your peers: a multi-task model for assessing the tone and objectivity of critiques of peer review comments

Prabhat Kumar Bharti (), Mayank Agarwal () and Asif Ekbal ()
Additional contact information
Prabhat Kumar Bharti: Indian Institute of Technology
Mayank Agarwal: Indian Institute of Technology
Asif Ekbal: Indian Institute of Technology

Scientometrics, 2024, vol. 129, issue 3, No 9, 1377-1413

Abstract: Abstract The peer-review process plays a pivotal role in maintaining the quality and credibility of scientific publications. However, in recent times, there has been an increase in unhelpful and overly critical reviews, which can be detrimental to the process. This surge in unconstructive reviews can be attributed to a higher volume of paper submissions and the inclusion of inexperienced reviewers. Consequently, authors are left with limited valuable feedback, compromising the effectiveness of peer review. Peer review feedback must be not only objective but also delivered politely and constructively. Our study introduces a novel approach to assessing the constructiveness and tone of peer reviews. We propose a two-fold taxonomy that categorizes reviews into five labels for constructiveness and three labels for politeness. To facilitate this research, we have created a corpus of 2716 review sentences, which have been manually annotated with a high inter-annotation agreement of 88.27% for constructiveness and 83.49% for politeness, offering a valuable resource for the scientific community. Furthermore, we present a multi-task model named “Multi-Label Critique (MLC)”that leverages ToxicBERT representations and deep neural attention mechanisms. This model adeptly evaluates the constructiveness and politeness of review sentences, outperforming competitive baseline models with an impressive accuracy of 87.4%. Our paper includes an extensive analysis of the MLC model and its variations. Our research is a significant step towards contributing to the development of constructive peer-review reports.

Keywords: Peer review; Reviewers’ comments; Content analysis; Multi-task model; Dataset applications; Digital libraries (search for similar items in EconPapers)
Date: 2024
References: Add references at CitEc
Citations:

Downloads: (external link)
http://link.springer.com/10.1007/s11192-024-04938-z Abstract (text/html)
Access to the full text of the articles in this series is restricted.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:scient:v:129:y:2024:i:3:d:10.1007_s11192-024-04938-z

Ordering information: This journal article can be ordered from
http://www.springer.com/economics/journal/11192

DOI: 10.1007/s11192-024-04938-z

Access Statistics for this article

Scientometrics is currently edited by Wolfgang Glänzel

More articles in Scientometrics from Springer, Akadémiai Kiadó
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().