Textual features of peer review predict top-cited papers: An interpretable machine learning perspective

Zhuanlan Sun

Journal of Informetrics, 2024, vol. 18, issue 2

Abstract: Peer review is crucial in improving the quality and reliability of scientific research. However, the mechanisms through which peer review practices ensure papers become top-cited papers (TCPs) after publication are not well understood. In this study, by collecting a data set containing 13, 066 papers published between 2016 and 2020 from Nature communications with open peer review reports, we aim to examine how textual features embedded within the peer review reports of papers that reflect the reviewers’ emotions may predict the papers to be TCPs. We compiled a list of 15 textual features and classified them into three categories: peer review features, linguistic features, and sentiment features. We then chose the XGBoost machine learning model with the best performance in predicting TCPs, and utilized the explainable artificial intelligence techniques SHAP to interpret the role of feature importance on the prediction results. The distribution of feature importance ranking results demonstrates that sentiment features play a crucial role in determining papers’ potential to be highly cited. This conclusion still holds, even when the ranking of the feature importance changes in the subgroup analysis of dividing the samples into four disciplines (biological sciences, health sciences, physical sciences, and earth and environmental sciences), as well as two groups based on whether reviewers’ identities were revealed. This research emphasizes the textual features retrieved from peer review reports that play role in improving manuscript quality can predict the post-publication research impact.

Keywords: Peer review reports; Textual features; Machine learning; Explainable artificial intelligence; Research impact (search for similar items in EconPapers)
Date: 2024
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (2)

Downloads: (external link)
http://www.sciencedirect.com/science/article/pii/S1751157724000142
Full text for ScienceDirect subscribers only

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:eee:infome:v:18:y:2024:i:2:s1751157724000142

DOI: 10.1016/j.joi.2024.101501

Access Statistics for this article

Journal of Informetrics is currently edited by Leo Egghe

More articles in Journal of Informetrics from Elsevier
Bibliographic data for series maintained by Catherine Liu ().