Why summing up bibliometric indicators does not justify a composite indicator

Boris Forthmann (), Philipp Doebler and Rüdiger Mutz
Additional contact information
Boris Forthmann: University of Münster
Philipp Doebler: TU Dortmund University
Rüdiger Mutz: University of Zurich

Scientometrics, 2024, vol. 129, issue 12, No 2, 7475-7499

Abstract: Abstract Various bibliometric indicators have been used to assess the researchers’ impact, but composites of such indicators, namely a metric that combines various individual indicators to describe a complex construct, have received a strong critique thus far. We employ concepts from psychometrics to revisit a composite proposed by Ioannidis et al. (2020) that aimed to represent researcher impact. Based on a selected sample of highly cited researchers, our proof-of-concept study presents a psychometrically principled composite formation. Specifically, by relying on the congeneric measurement model (and related models) rooted in classical test theory, we found that one of the proposed indicators clearly violated the congeneric model’s fundamental assumption of unidimensionality, and two other indicators were excluded for redundancy. The resulting composite based on only three bibliometric indicators was found to display excellent reliability. Importantly, the reliability approached that of the composite based on five indicators, and it was clearly better than the original six-indicator composite. Further, we found rather homogeneous effective weights (i.e., relative contributions of each indicator to composite variance) for simple sum scores, and these weights were close to those calculated using an algorithm for equally effective weights. While the congeneric measurement model also showed strong measurement invariance across sexes, this model’s loadings and intercepts were not measurement invariant across scientific fields and academic age groups. Notably, we found that various derived composites correlate positively with academic age, hinting at a lack of fairness of the composites.

Keywords: Researcher assessment; Classical test theory; Confirmatory factor analysis; Measurement invariance; Composites; Weighting; Mathematics subject classification; 62P25; JEL classification; C18 (search for similar items in EconPapers)
Date: 2024
References: Add references at CitEc
Citations:

Downloads: (external link)
http://link.springer.com/10.1007/s11192-024-05194-x Abstract (text/html)
Access to the full text of the articles in this series is restricted.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:scient:v:129:y:2024:i:12:d:10.1007_s11192-024-05194-x

Ordering information: This journal article can be ordered from
http://www.springer.com/economics/journal/11192

DOI: 10.1007/s11192-024-05194-x

Access Statistics for this article

Scientometrics is currently edited by Wolfgang Glänzel

More articles in Scientometrics from Springer, Akadémiai Kiadó
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().