Evaluating Research Impact Based on Semantic Scholar Highly Influential Citations, Total Citations, and Altmetric Attention Scores: The Quest for Refined Measures Remains Illusive

Latefa Ali Dardas (), Malik Sallam, Amanda Woodward, Nadia Sweis, Narjes Sweis and Faleh A. Sawair
Additional contact information
Latefa Ali Dardas: Community Health Nursing Department, School of Nursing, The University of Jordan, Amman 11942, Jordan
Malik Sallam: Department of Pathology, Microbiology and Forensic Medicine, School of Medicine, The University of Jordan, Amman 11942, Jordan
Amanda Woodward: Lane Medical Library, Stanford University, Stanford, CA 94305, USA
Nadia Sweis: Department of Medicine, Division of Endocrinology, Diabetes, and Metabolism, The University of Illinois at Chicago, Chicago, IL 60612, USA
Narjes Sweis: School of Medicine, The University of Jordan, Amman 11942, Jordan
Faleh A. Sawair: School of Medicine, The University of Jordan, Amman 11942, Jordan

Publications, 2023, vol. 11, issue 1, 1-16

Abstract: Background : The evaluation of scholarly articles’ impact has been heavily based on the citation metrics despite the limitations of this approach. Therefore, the quest for meticulous and refined measures to evaluate publications’ impact is warranted. Semantic Scholar (SS) is an artificial intelligence-based database that allegedly identifies influential citations defined as “Highly Influential Citations” (HICs). Citations are considered highly influential according to SS when the cited publication has a significant impact on the citing publication (i.e., the citer uses or extends the cited work). Altmetrics are measures of online attention to research mined from activity in online tools and environments. Aims : The current study aimed to explore whether SS HICs provide an added value when it comes to measuring research impact compared to total citation counts and Altmetric Attention Score (AAS). Methods : Dimensions was used to generate the dataset for this study, which included COVID-19-related scholarly articles published by researchers affiliated to Jordanian institutions. Altmetric Explorer was selected as an altmetrics harvesting tool, while Semantic Scholar was used to extract details related to HICs. A total of 618 publications comprised the final dataset. Results : Only 4.57% (413/9029) of the total SS citations compiled in this study were classified as SS HICs. Based on SS categories of citations intent, 2626 were background citations (29.08%, providing historical context, justification of importance, and/or additional information related to the cited paper), 358 were result citations (3.97%, that extend on findings from research that was previously conducted), and 263 were method citations (2.91%, that use the previously established procedures or experiments to determine whether the results are consistent with findings in related studies). No correlation was found between HICs and AAS ( r = 0.094). Manual inspection of the results revealed substantial contradictions, flaws, and inconsistencies in the SS HICs tool. Conclusions : The use of SS HICs in gauging research impact is significantly limited due to the enigmatic method of its calculation and total dependence on artificial intelligence. Along with the already documented drawbacks of total citation counts and AASs, continuous evaluation of the existing tools and the conception of novel approaches are highly recommended to improve the reliability of publication impact assessment.

Keywords: altmetrics; influential citations; citation metrics; semantic scholar; research impact (search for similar items in EconPapers)
JEL-codes: A2 D83 L82 (search for similar items in EconPapers)
Date: 2023
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (1)

Downloads: (external link)
https://www.mdpi.com/2304-6775/11/1/5/pdf (application/pdf)
https://www.mdpi.com/2304-6775/11/1/5/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jpubli:v:11:y:2023:i:1:p:5-:d:1050427

Access Statistics for this article

Publications is currently edited by Ms. Jennifer Zhang

More articles in Publications from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().