Semantic Academic Profiler (SAP): a framework for researcher assessment based on semantic topic modeling

Felipe Viegas (), Antônio Pereira (), Pablo Cecílio (), Elisa Tuler (), Wagner Meira (), Marcos Gonçalves () and Leonardo Rocha ()
Additional contact information
Felipe Viegas: Federal University of Minas Gerais
Antônio Pereira: Federal University of São João Del Rey
Pablo Cecílio: Federal University of São João Del Rey
Elisa Tuler: Federal University of São João Del Rey
Wagner Meira: Federal University of Minas Gerais
Marcos Gonçalves: Federal University of Minas Gerais
Leonardo Rocha: Federal University of São João Del Rey

Scientometrics, 2022, vol. 127, issue 8, No 32, 5005-5026

Abstract: Abstract Recent efforts have focused on identifying multidisciplinary teams and detecting co-Authorship Networks based on exploring topic modeling to identify researchers’ expertise. Though promising, none of these efforts perform a real-life evaluation of the quality of the built topics. This paper proposes a Semantic Academic Profiler (SAP) framework that allows summarizing articles written by researchers to automatically build research profiles and perform online evaluations regarding these built profiles. SAP exploits and extends state-of-the-art Topic Modeling strategies based on Cluwords considering n-grams and introduces a new visual interface able to highlight the main topics related to articles, researchers and institutions. To evaluate SAP’s capability of summarizing the profile of such entities as well as its usefulness for supporting online assessments of the topics’ quality, we perform and contrast two types of evaluation, considering an extensive repository of Brazilian curricula vitae: (1) an offline evaluation, in which we exploit a traditional metric (NPMI) to measure the quality of several data representations strategies including (i) TFIDF, (ii) TFIDF with Bi-grams, (iii) Cluwords, and (iv) CluWords with Bi-grams; and (2) an online evaluation through an A/B test where researchers evaluate their own built profiles. We also perform an online assessment of SAP user interface through a usability test following the SUS methodology. Our experiments indicate that the CluWords with Bi-grams is the best solution and the SAP interface is very useful. We also observed essential differences in the online and offline assessments, indicating that using both together is very important for a comprehensive quality evaluation. Such type of study is scarce in the literature and our findings open space for new lines of investigation in the Topic Modeling area.

Keywords: Semantic Academic Profiler; Topic modeling; Word embeddings (search for similar items in EconPapers)
Date: 2022
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (2)

Downloads: (external link)
http://link.springer.com/10.1007/s11192-022-04449-9 Abstract (text/html)
Access to the full text of the articles in this series is restricted.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:scient:v:127:y:2022:i:8:d:10.1007_s11192-022-04449-9

Ordering information: This journal article can be ordered from
http://www.springer.com/economics/journal/11192

DOI: 10.1007/s11192-022-04449-9

Access Statistics for this article

Scientometrics is currently edited by Wolfgang Glänzel

More articles in Scientometrics from Springer, Akadémiai Kiadó
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().