An experimental test of the effectiveness of cognitive interviewing in pretesting questionnaires

Timo Lenzner (), Patricia Hadler () and Cornelia Neuert ()
Additional contact information
Timo Lenzner: GESIS – Leibniz Institute for the Social Sciences, Survey Design and Methodology
Patricia Hadler: GESIS – Leibniz Institute for the Social Sciences, Survey Design and Methodology
Cornelia Neuert: GESIS – Leibniz Institute for the Social Sciences, Survey Design and Methodology

Quality & Quantity: International Journal of Methodology, 2023, vol. 57, issue 4, No 13, 3199-3217

Abstract: Abstract Pretesting survey questions via cognitive interviewing is based on the assumptions that the problems identified by the method truly exist in a later survey and that question revisions based on cognitive interviewing findings produce higher-quality data than the original questions. In this study, we empirically tested these assumptions in a web survey experiment (n = 2,200). Respondents received one of two versions of a question on self-reported financial knowledge: either the original draft version, which was pretested in ten cognitive interviews, or a revised version, which was modified based on the results of the cognitive interviews. We examined whether the cognitive interviewing findings predicted problems encountered in the web survey and whether the revised question version was associated with higher content-related and criterion-related validity than the draft version. The results show that cognitive interviewing is effective in identifying real question problems, but not necessarily in fixing survey questions and improving data quality. Overall, our findings point to the importance of using iterative pretesting designs, that is, carrying out multiple rounds of cognitive interviews and also testing the revisions to ensure that they are indeed of higher quality than the draft questions.

Keywords: Cognitive interviewing; Criterion-related validity; Data quality; Experiment; Pretesting (search for similar items in EconPapers)
Date: 2023
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
http://link.springer.com/10.1007/s11135-022-01489-4 Abstract (text/html)
Access to the full text of the articles in this series is restricted.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:qualqt:v:57:y:2023:i:4:d:10.1007_s11135-022-01489-4

Ordering information: This journal article can be ordered from
http://www.springer.com/economics/journal/11135

DOI: 10.1007/s11135-022-01489-4

Access Statistics for this article

Quality & Quantity: International Journal of Methodology is currently edited by Vittorio Capecchi

More articles in Quality & Quantity: International Journal of Methodology from Springer
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().