Replacing or enhancing the human coder? Multiclass classification of policy documents with large language models

Erkan Gunes () and Christoffer Koch Florczak
Additional contact information
Christoffer Koch Florczak: Aalborg University

Journal of Computational Social Science, 2025, vol. 8, issue 2, No 5, 20 pages

Abstract: Abstract Classifying policy documents into policy issue topics has been a long-time effort in political science and communication disciplines. In this work, we use the GPT 3.5, 4, 4o, Gemini 1.0 pro, 1.5 pro and 1.5 Flash models from OpenAI and Google respectively, which are pre-trained instruction-tuned Large Language Models (LLM), to classify congressional bills and hearings into the Comparative Agendas Project’s 21 major policy topics. We propose three use-case scenarios and estimate overall weighted F1 scores ranging from 0.44 to 0.82 depending on scenario and LLM models employed. The three scenarios aim at minimal, moderate, and major human interference, respectively. Our results point towards the insufficiency of complete reliance on instruction tuned LLMs, an increasing accuracy along with the human effort exerted, and a surprisingly high accuracy achieved in the most humanly demanding use-case. Our superior use-case, which combined GPT 4 and Gemini 1.5 Pro achieved 0.82 weighted F1 score on the 83% of the data in which the two models agreed. Benchmarking against Babel, a custom trained algorithm for this use case, Babel presents with a 13 and 16 percentage point higher accuracy. Future research, practical considerations, and implications of future LLM developments are discussed.

Keywords: Large language models; Text classification; Computational text analysis; Policy agendas (search for similar items in EconPapers)
Date: 2025
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
http://link.springer.com/10.1007/s42001-025-00362-2 Abstract (text/html)
Access to the full text of the articles in this series is restricted.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:jcsosc:v:8:y:2025:i:2:d:10.1007_s42001-025-00362-2

Ordering information: This journal article can be ordered from
http://www.springer. ... iences/journal/42001

DOI: 10.1007/s42001-025-00362-2

Access Statistics for this article

Journal of Computational Social Science is currently edited by Takashi Kamihigashi

More articles in Journal of Computational Social Science from Springer
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().