Question form Matters: Examining Trust in Government Through Open and Closed Survey Items

Jana Bernhard-Harrer and Katharina Pfaff

Journal of Survey Statistics and Methodology, 2025, vol. 13, issue 4, 370-392

Abstract: How we ask questions in surveys significantly impacts the answers. While previous research has examined differences between open-ended and closed-ended survey responses, our understanding remains fragmented, especially considering the growing interest in open-ended questions spurred by advances in automated text analysis. We explore how computational text analysis methods––particularly large language models––can efficiently analyze open-ended responses with promising sentiment and actor identification results, though topic classification remains more challenging. Then, we systematically compare open-ended and closed-ended responses on trust in the Austrian coalition government to assess how different formats shape the expression of political trust. We find significant discrepancies between the trust levels indicated in closed-ended questions and the trust described in open-ended responses, with paired t-tests also revealing statistically significant differences. These findings highlight that question format affects how respondents express their views, raising important considerations about the usefulness and validity of trust measures. They highlight the unique value of open-ended questions in capturing nuanced perspectives often missed by closed-ended formats. This has broader implications for survey design: including open-ended questions could enrich data quality and provide deeper insights into public opinion.

Keywords: Large language models; Open- versus closed-ended questions; Trust in government (search for similar items in EconPapers)
Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
http://hdl.handle.net/10.1093/jssam/smaf014 (application/pdf)
Access to full text is restricted to subscribers.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:oup:jassam:v:13:y:2025:i:4:p:370-392.

Access Statistics for this article

Journal of Survey Statistics and Methodology is currently edited by Emily Berg and Brad Edwards

More articles in Journal of Survey Statistics and Methodology from American Association for Public Opinion Research and American Statistical Association
Bibliographic data for series maintained by Oxford University Press ().