Understanding the social and political dimensions of research(er) assessment: Interpretative flexibility and hidden criteria

Tony Ross-Hellauer, Noémie Aubert Bonn and Serge P. J. M. Horbach
Additional contact information
Noémie Aubert Bonn: Hasselt University
Serge P. J. M. Horbach: Radboud University

No e5uyv_v1, SocArXiv from Center for Open Science

Abstract: Debates about appropriate, fair and effective ways of assessing research and researchers have raged through the scientific community for decades, recently mostly concerned with discussing the merits and limitations of metric-based, quantitative assessments versus peer review-based, qualitative alternatives. Ample attention has been paid to formal assessment criteria, building to a consensus that less emphasis should be placed on quantification, while supporting open and diverse sets of criteria. Yet the theory and evidence upon which such policy reform depends is still surprisingly sparse. Based on qualitative free-text responses from 121 respondents gathered during an international survey of active researchers, this study examines how criteria are applied in practice and how those being assessed perceive informal criteria to determine the outcomes of assessments. While confirming the general critique on over-emphasising quantification, respondents particularly identify a mismatch between formal criteria and actual evaluation practices. Hidden criteria, including social, political, and demographic factors, are perceived important, especially in intransparent assessment procedures, opening up for assessors’ interpretative flexibility. This adds to ongoing discussions on the performativity of assessment criteria and lays bare a tension between the rigidity and flexibility of criteria and the extent to which these can be transparently communicated.

Date: 2023-03-18
References: Add references at CitEc
Citations:

Downloads: (external link)
https://osf.io/download/6412f338e256360dc22317f8/

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:osf:socarx:e5uyv_v1

DOI: 10.31219/osf.io/e5uyv_v1

Access Statistics for this paper

More papers in SocArXiv from Center for Open Science
Bibliographic data for series maintained by OSF ().