State of the Art of Novelty Indicators

Sotaro Shibayama, Zhao Wu, Deyun Yin, Kazuki Yokota and Pierre Pelletier
Additional contact information
Sotaro Shibayama: UTokyo - The University of Tokyo
Zhao Wu: HIT - Harbin Institute of Technology
Deyun Yin: NJU - Nanjing University
Kazuki Yokota: Hitotsubashi University
Pierre Pelletier: BETA - Bureau d'Économie Théorique et Appliquée - AgroParisTech - UNISTRA - Université de Strasbourg - Université de Haute-Alsace (UHA) - Université de Haute-Alsace (UHA) Mulhouse - Colmar - UL - Université de Lorraine - CNRS - Centre National de la Recherche Scientifique - INRAE - Institut National de Recherche pour l’Agriculture, l’Alimentation et l’Environnement

Working Papers from HAL

Abstract: Novelty is a core value in scientific research, and its measurement has been of wide scholarly and practical interest. Numerous bibliometric indicators for novelty have been proposed, some shared in open repositories, which have facilitated empirical investigation into scientific novelty. However, there remains a fundamental limitation. That is, we have insufficient knowledge as to what these indicators truly measure because of limited efforts for examining measurement validity. This study addresses this gap by evaluating a range of novelty indicators, employing various operationalisation strategies, against self-reported novelty assessments obtained from our originally designed questionnaires covering multiple novelty dimensions. Our analyses examining the correlation between the self-assessed scores and bibliometric indicators offer several insights. First, while most indicators detect some aspects of novelty, a single indicator may not sufficiently capture all forms of novelty. Second, a cross-disciplinary comparison reveals that indicators' performance varies across disciplines – some indicators demonstrate consistent correlations across disciplines while others show correlations only in limited disciplines. Third, some indicators capture both high and low levels of novelty along a continuum, others detect specifically high-level novelty, and yet others detect the presence of newness without differentiating its degree. Fourth, as employing language models in novelty evaluation has become common, we compare static and contextual language models, finding that indicators based on the latter outperforms those on the former. Fifth, we examine ex-post indicators, which require post-publication data (e.g., forward citation), and find that they offer no clear advantages over ex-ante indicators in detecting novelty. These findings highlight both the potential and limitations of existing indicators and offer implications for the future development and application of novelty indicators.

Keywords: Novelty; Originality; Validation; Language model; Recombination (search for similar items in EconPapers)
Date: 2026-10-12
References: Add references at CitEc
Citations:

There are no downloads for this item, see the EconPapers FAQ for hints about obtaining it.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:hal:wpaper:hal-05795718

DOI: 10.2139/ssrn.5379973

Access Statistics for this paper

More papers in Working Papers from HAL
Bibliographic data for series maintained by CCSD ().