A Golden Era for Open-Ended Questions? Using LLMs for Text Classification Tasks

Ansgar Hudde () and Shannon Taflinger ()
Additional contact information
Ansgar Hudde: University of Cologne
Shannon Taflinger: University of Cologne

No 416, ECONtribute Discussion Papers Series from University of Bonn and University of Cologne, Germany

Abstract: Open-text questions in quantitative surveys can yield rich information from large samples, but analysing and coding these data using qualitative text analysis is resource-intensive. Large Language Models (LLMs) are a promising tool for scaling up such analyses, reducing time and financial costs. In this paper, we compare the coding accuracy of LLMs with that of student assistants, defining accuracy as agreement with a researcher-coded benchmark dataset. We assess performance on a semi-complex coding task: coding approximately 1,400 open-ended text responses from young US Americans about dating across party-political lines. A researcher-designed coding scheme, developed through thematic qualitative text analysis of the open-text responses, was applied by LLMs and student assistants. We evaluate models from OpenAI, Anthropic, and Mistral, with and without access to training data. The most advanced models outperform student assistants, and performance further increases with training data, highlighting LLMs’ capability to code open-text responses. Whereas previous research has mainly focused on social media texts, comparatively simple and surface-level coding tasks, and a technically oriented audience, we contribute to the literature by studying a particularly promising use case of open-ended survey responses and by providing practical recommendations to applied social scientists.

Keywords: Large language models; open-ended questions; text analysis (search for similar items in EconPapers)
JEL-codes: C45 C81 C83 (search for similar items in EconPapers)
Pages: 38 pages
Date: 2026-06
References: Add references at CitEc
Citations:

Downloads: (external link)
https://www.econtribute.de/RePEc/ajk/ajkdps/ECONtribute_416_2026.pdf First version, 2026 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:ajk:ajkdps:416

Access Statistics for this paper

More papers in ECONtribute Discussion Papers Series from University of Bonn and University of Cologne, Germany Niebuhrstrasse 5, 53113 Bonn, Germany.
Bibliographic data for series maintained by ECONtribute Office ().