AI for Survey Design: Generating and Evaluating Survey Questions with Large Language Models

Anna Fuchs, Anna-Carolina Haensch and Wiebke Weber

No fzn7t_v1, SocArXiv from Center for Open Science

Abstract: Designing survey questions is easy; however designing good survey questions is a complex task. Large language models (LLMs) have the potential to support this task by automating parts of the item-generation process, but their suitability for survey research has not yet been systematically evaluated. Published research in this area remains sparse, and little is known about the quality and characteristics of survey items generated by LLMs or the factors influencing their performance. This work provides the first in-depth analysis of LLM-based survey item generation and systematically evaluates how different design choices affect item quality. Five LLMs, namely GPT-4o, GPT-4o-mini, GPT-oss-20B, LLaMA 3.1 8B, and LLaMA 3.1 70B, were used to generate survey items on four substantive domains: work, living conditions, national politics, and recent politics. We additionally evaluate three prompting strategies: zero-shot, role, and chain-of-thought prompting. To assess the quality of the generated survey items, we use the Survey Quality Predictor (SQP), a tool for estimating the quality of attitudinal survey items based on codings of their formal and linguistic characteristics. To code these characteristics, we used an LLM-assisted procedure. The findings show striking differences in survey item characteristics across the different models and prompting techniques. Both the choice of model and the prompting technique employed influence the quality of LLM-generated survey items. Closed-source GPT models generally produce more consistent items than open-source LLaMA models. Overall, chain-of-thought prompting achieved the best results. GPT-4o, GPT-4o-mini, and LLaMA 3.1 70B achieved similar item quality, while the LLaMA model showed greater variability.

Date: 2026-03-12
New Economics Papers: this item is included in nep-ain, nep-big, nep-cmp and nep-dcm
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://osf.io/download/69b19dfb595757277e98218e/

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:osf:socarx:fzn7t_v1

DOI: 10.31219/osf.io/fzn7t_v1

Access Statistics for this paper

More papers in SocArXiv from Center for Open Science
Bibliographic data for series maintained by OSF ().