Economics at your fingertips  

Do we measure novelty when we analyze unusual combinations of cited references? A validation study of bibliometric novelty indicators based on F1000Prime data

Lutz Bornmann, Alexander Tekles, Helena H. Zhang and Fred Y. Ye

Journal of Informetrics, 2019, vol. 13, issue 4

Abstract: Lee et al. (2015) – based on Uzzi et al. (2013) – and Wang et al. (2017) proposed scores based on cited references (cited journals) data which can be used to measure the novelty of papers (named as novelty scores U and W in this study). Although previous research has used novelty scores in various empirical analyses, no study has been published up to now – to the best of our knowledge – which quantitatively tested the convergent validity of novelty scores: do these scores measure what they propose to measure? Using novelty assessments by faculty members (FMs) at F1000Prime for comparison, we tested the convergent validity of the two novelty scores (U and W). FMs’ assessments do not only refer to the quality of biomedical papers, but also to their characteristics (by assigning certain tags to the papers): for example, are the presented findings or formulated hypotheses novel (tags “new findings” and “hypothesis”)? We used these and other tags to investigate the convergent validity of both novelty scores. Our study reveals different results for the novelty scores: the results for novelty score U are mostly in agreement with previously formulated expectations. We found, for instance, that for a standard deviation (one unit) increase in novelty score U, the expected number of assignments of the “new finding” tag increase by 7.47%. The results for novelty score W, however, do not reflect convergent validity with the FMs’ assessments: only the results for some tags are in agreement with the expectations. Thus, we propose – based on our results – the use of novelty score U for measuring novelty quantitatively, but question the use of novelty score W.

Keywords: Bibliometrics; Novelty; Creativity; Cited references; F1000Prime (search for similar items in EconPapers)
Date: 2019
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (1) Track citations by RSS feed

Downloads: (external link)
Full text for ScienceDirect subscribers only

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link:

DOI: 10.1016/j.joi.2019.100979

Access Statistics for this article

Journal of Informetrics is currently edited by Leo Egghe

More articles in Journal of Informetrics from Elsevier
Bibliographic data for series maintained by Haili He ().

Page updated 2020-08-05
Handle: RePEc:eee:infome:v:13:y:2019:i:4:s1751157718304371