Scaling Open-Ended Survey Coding: An LLM Pipeline Where Definitions Do the Heavy Lifting

Chris Soria

No gjvcf_v1, SocArXiv from Center for Open Science

Abstract: As large language model (LLM)–based text classification becomes routine in the social sciences, researchers confront dozens of competing models, inconsistent advice on prompting, and little standardized tooling with evidence‑based defaults. CatLLM, an open‑source Python and R package, addresses this gap with a three‑stage pipeline—exploration, extraction, classification—for coding open‑ended survey responses. The package offers a provider‑agnostic interface that supports multi‑model ensembles, batch processing, and fully local deployment via open‑weight models, and can be operated through a conversational interface by researchers with no programming experience. CatLLM’s defaults are calibrated by a systematic empirical study evaluating 21 LLMs across three capability tiers, six providers, and four survey questions, benchmarked against sociologist‑coded ground truth. This validation reveals a consistent problem: all models over‑classify, with precision lagging 40–50 percentage points behind sensitivity, implying that default LLM configurations may substantially overstate category prevalence. CatLLM encodes empirically grounded mitigations as defaults: verbose category definitions with explicit inclusion and exclusion criteria, unanimous multi‑model ensembling, and an automatic “Other” escape‑valve category, while disabling advanced prompting strategies that show no reliable benefit. Ensembles of inexpensive open‑weight models outperform the best individual cloud model, enabling fully local deployment without transmitting survey data to external servers. These findings replicate on two independent public datasets spanning political and emotional text, and an applied example linking tool‑coded “move reasons” to respondent demographics uncovers distinct life‑course patterns in residential mobility.

Date: 2026-03-20
New Economics Papers: this item is included in nep-cmp and nep-inv
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
https://osf.io/download/69bd7a620e33cbebb4a6f7d7/

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:osf:socarx:gjvcf_v1

DOI: 10.31219/osf.io/gjvcf_v1

Access Statistics for this paper

More papers in SocArXiv from Center for Open Science
Bibliographic data for series maintained by OSF ().