Systematic Design and Evaluation of a Citation Function Classification Scheme in Indonesian Journals

Yaniasih Yaniasih and Indra Budi
Additional contact information
Yaniasih Yaniasih: Faculty of Computer Science, Universitas Indonesia, Depok 16424, Indonesia
Indra Budi: Faculty of Computer Science, Universitas Indonesia, Depok 16424, Indonesia

Publications, 2021, vol. 9, issue 3, 1-14

Abstract: Classifying citations according to function has many benefits when it comes to information retrieval tasks, scholarly communication studies, and ranking metric developments. Many citation function classification schemes have been proposed, but most of them have not been systematically designed for an extensive literature-based compilation process. Many schemes were also not evaluated properly before being used for classification experiments utilizing large datasets. This paper aimed to build and evaluate new citation function categories based upon sufficient scientific evidence. A total of 2153 citation sentences were collected from Indonesian journal articles for our dataset. To identify the new categories, a literature survey was conducted, analyses and groupings of category meanings were carried out, and then categories were selected based on the dataset’s characteristics and the purpose of the classification. The evaluation used five criteria: coherence, ease, utility, balance, and coverage. Fleiss’ kappa and automatic classification metrics using machine learning and deep learning algorithms were used to assess the criteria. These methods resulted in five citation function categories. The scheme’s coherence and ease of use were quite good, as indicated by an inter-annotator agreement value of 0.659 and a Long Short-Term Memory (LSTM) F1-score of 0.93. According to the balance and coverage criteria, the scheme still needs to be improved. This research data was limited to journals in food science published in Indonesia. Future research will involve classifying the citation function using a massive dataset collected from various scientific fields and published from some representative countries, as well as applying improved annotation schemes and deep learning methods.

Keywords: citation function; classification scheme; annotator agreement; machine learning; deep learning (search for similar items in EconPapers)
JEL-codes: A2 D83 L82 (search for similar items in EconPapers)
Date: 2021
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (1)

Downloads: (external link)
https://www.mdpi.com/2304-6775/9/3/27/pdf (application/pdf)
https://www.mdpi.com/2304-6775/9/3/27/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jpubli:v:9:y:2021:i:3:p:27-:d:584017

Access Statistics for this article

Publications is currently edited by Ms. Jennifer Zhang

More articles in Publications from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().