How robust is journal rating in Humanities and Social Sciences? Evidence from a large-scale, multi-method exercise

Antonio Ferrara and Andrea Bonaccorsi

Research Evaluation, 2016, vol. 25, issue 3, 279-291

Abstract: This article reports on a large-scale exercise of classification of journals in the fields of Humanities and Social Sciences, carried out by the Italian Agency for the Evaluation of Universities and Research Institutes. After discussing at some length the controversies linked with journal classification and its impact, we endeavor to compare such a classification with the scores that individual articles published in the same journals were assigned by completely independent assessors in the same period of time. The data refer to an important subset of disciplines covering History, Philosophy, Geography, Anthropology, Education, and Library Sciences, allowing for comparisons between scientific fields of different sizes, outlooks, and methods. As the controversies surrounding the rating of journals focus on the difference between the container (the journal) and the content (the individual article), we addressed the following research questions: (1) Is journal rating, produced by an expert-based procedure, a good predictor of the quality of articles published in the journal? (2) To what extent different panel of experts evaluating the same journals produce consistent ratings? (3) To what extent the assessment of scientific societies on journal rating is a good predictor of the quality of articles published in journals? (4) Are there systematic biases in the peer review of articles and in the expert-based journal rating? We find that journal rating is a legitimate and robust assessment exercise, as long as it is managed carefully and in a cautious way and used to evaluate aggregates of researchers rather than individual researchers.

Date: 2016
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (16)

Downloads: (external link)
http://hdl.handle.net/10.1093/reseval/rvv048 (application/pdf)
Access to full text is restricted to subscribers.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:oup:rseval:v:25:y:2016:i:3:p:279-291.

Access Statistics for this article

Research Evaluation is currently edited by Julia Melkers, Emanuela Reale and Thed van Leeuwen

More articles in Research Evaluation from Oxford University Press
Bibliographic data for series maintained by Oxford University Press ().