Scaling research aim identification: Language models for classifying scientific and societal‐oriented studies

Mengjia Wu, Gunnar Sivertsen, Lin Zhang, Fan Qi and Yi Zhang

Journal of the Association for Information Science & Technology, 2025, vol. 76, issue 11, 1470-1487

Abstract: The classification of research according to its aims has been a longstanding focus in the fields of quantitative science studies and R&D statistics. Since 1963, the Organization for Economic Co‐operation and Development (OECD) has employed a classical distinction among basic, applied, and experimental research. Building on this framework, our previous work highlighted the utility of differentiating between scientific and societal progress as two primary research objectives. This distinction enabled the quantitative analysis of scientific publication abstracts and the development of an automated method for large‐scale classification. In the current study, we systematically evaluate text classification techniques, including traditional text mining models, classification tools, BERT‐based language models, and decoder‐only large language models (LLMs) such as ChatGPT. Our findings show that the fine‐tuned GPT‐4o‐mini model performs the best among single‐model approaches. However, traditional and BERT‐based models outperform in certain fine‐grained classification tasks. Leveraging majority voting strategies to incorporate their strengths yields performance comparable to closed‐source GPT models. A case study on 10 biomedical journals further validates the method, demonstrating strong alignment between journal scopes, model predictions, and outputs generated by the fine‐tuned GPT‐4o‐mini model. These results highlight the robustness and practical effectiveness of the proposed methodology for nuanced research aim classification.

Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
https://doi.org/10.1002/asi.70004

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:bla:jinfst:v:76:y:2025:i:11:p:1470-1487

Ordering information: This journal article can be ordered from
http://www.blackwell ... bs.asp?ref=2330-1635

Access Statistics for this article

More articles in Journal of the Association for Information Science & Technology from Association for Information Science & Technology
Bibliographic data for series maintained by Wiley Content Delivery ().