Validating LLM-Generated Synthetic Survey Samples: A Use-Dependent Multi-Dimensional Framework for Social-Science Applications

Enrico Angelo Raffaele D'Ecclesiis, Fulvio Castellacci and Arash Ahmadi

Papers from arXiv.org

Abstract: Large language models (LLMs) make it increasingly feasible to generate synthetic survey respondents, but their empirical usefulness cannot be judged by a single notion of accuracy. A synthetic sample may perform weakly at the respondent level while still preserving distributional or substantive relationships relevant to social-science analysis. We develop a use-dependent, multi-dimensional validation framework distinguishing predictive, distributional, and substantive validity, and illustrate it using Eurobarometer data on attitudes toward artificial intelligence. Four off-the-shelf LLMs generate responses conditional on observed respondent profiles, while supervised machine learning provides a predictive reference for interpreting their performance. The results reveal substantial differences across validation dimensions: some LLM-generated samples preserve meaningful marginal heterogeneity, demographic ordering, cross-country variation, and predictor structure despite imperfect respondent-level correspondence. Performance varies across models and outcomes, but the broader picture is cautiously constructive. Synthetic samples should therefore be evaluated against the empirical properties required by their intended use.

Date: 2026-10
References: Add references at CitEc
Citations:

Downloads: (external link)
https://arxiv.org/pdf/2611.00256 Latest version (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:arx:papers:2611.00256

Access Statistics for this paper

More papers in Papers from arXiv.org
Bibliographic data for series maintained by arXiv administrators ().