Exploring the applicability of large language models to citation context analysis

Kai Nishikawa () and Hitoshi Koshiba
Additional contact information
Kai Nishikawa: University of Tsukuba
Hitoshi Koshiba: Ministry of Culture, Science and Sports (MEXT)

Scientometrics, 2024, vol. 129, issue 11, No 11, 6777 pages

Abstract: Abstract Unlike traditional citation analysis, which assumes that all citations in a paper are equivalent, citation context analysis considers the contextual information of individual citations. However, citation context analysis requires creating a large amount of data through annotation, which hinders its widespread use. This study explored the applicability of Large Language Models (LLM)—particularly Generative Pre-trained Transformer (GPT)—to citation context analysis by comparing LLM and human annotation results. The results showed that LLM annotation is as good as or better than human annotation in terms of consistency but poor in terms of its predictive performance. Thus, having LLM immediately replace human annotators in citation context analysis is inappropriate. However, the annotation results obtained by LLM can be used as reference information when narrowing the annotation results obtained by multiple human annotators down to one; alternatively, the LLM can be used as an annotator when it is difficult to prepare sufficient human annotators. This study provides basic findings important for the future development of citation context analysis.

Keywords: Scientometrics; Citation context analysis; Annotation; Large language models (LLM); Generative pre-trained transformer (GPT) (search for similar items in EconPapers)
Date: 2024
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
http://link.springer.com/10.1007/s11192-024-05142-9 Abstract (text/html)
Access to the full text of the articles in this series is restricted.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:scient:v:129:y:2024:i:11:d:10.1007_s11192-024-05142-9

Ordering information: This journal article can be ordered from
http://www.springer.com/economics/journal/11192

DOI: 10.1007/s11192-024-05142-9

Access Statistics for this article

Scientometrics is currently edited by Wolfgang Glänzel

More articles in Scientometrics from Springer, Akadémiai Kiadó
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().