Overturning some assumptions about the effects of evaluation systems on publication performance

Carmen Osuna, Laura Cruz-Castro and Luis Sanz-Menéndez ()
Additional contact information
Carmen Osuna: CSIC Institute of Public Goods and Policies (IPP-CCHS), Consejo Superior de Investigaciones Científicas (CSIC)
Laura Cruz-Castro: CSIC Institute of Public Goods and Policies (IPP-CCHS), Consejo Superior de Investigaciones Científicas (CSIC)
Luis Sanz-Menéndez: CSIC Institute of Public Goods and Policies (IPP-CCHS), Consejo Superior de Investigaciones Científicas (CSIC)

Scientometrics, 2011, vol. 86, issue 3, No 3, 575-592

Abstract: Abstract In 1989 the Spanish Government established an individual retrospective research evaluation system (RES) for public researchers. Policy makers have associated the establishment of this evaluation system with the significant increase in the volume of scientific publications attributed to Spain over the last decades. In a similar vein to the analyses of other country cases, some scholars have also claimed that the growth of Spain’s international scientific publications is a result of the establishment of the new evaluation system. In this paper, we provide a methodological revision of the validity threats in previous research, including some interrupted time-series analyses and control groups to investigate the effects of this policy instrument on the number of papers produced by Spanish authors. In the years following the establishment of the evaluation system, the results indicate a considerable increase in the number of papers attributed to Spanish authors among those eligible for evaluation (the “treated” group), but also in the control groups. After testing various alternative explanations, we conclude that the growth in Spanish publications cannot be attributed indisputably to the effect of the establishment of the RES, but rather to the increase of expenditure and number of researchers in the Spanish R&D system along with some maturation effects. We take this case as an example of the need to improve and refine methodologies and to be more cautious when attributing effects to research evaluation mechanisms at the national level.

Keywords: Research evaluation systems; Scientific production; Effects of evaluations; Quasi-experimental design; Spain; CNEAI (search for similar items in EconPapers)
Date: 2011
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (15)

Downloads: (external link)
http://link.springer.com/10.1007/s11192-010-0312-7 Abstract (text/html)
Access to the full text of the articles in this series is restricted.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:scient:v:86:y:2011:i:3:d:10.1007_s11192-010-0312-7

Ordering information: This journal article can be ordered from
http://www.springer.com/economics/journal/11192

DOI: 10.1007/s11192-010-0312-7

Access Statistics for this article

Scientometrics is currently edited by Wolfgang Glänzel

More articles in Scientometrics from Springer, Akadémiai Kiadó
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().