Experimental Evidence on the Effect of Mixed-Mode Survey Designs on Item Measurement and Variable Correlations

Liam Wright, Georgia D Tomova, Matt Brown, Konstantinos G. Tsigaridis, Morag Henderson and Richard J. Silverwood
Additional contact information
Konstantinos G. Tsigaridis: University of Cambridge

No r97ht_v1, SocArXiv from Center for Open Science

Abstract: Background: Mixed-mode designs can generate bias in all types of analysis of survey data due to systematic differences in how survey participants may respond according to mode – a phenomenon referred to as a mode effect. There is widespread evidence of mode effects upon item measurement (e.g., item means and proportions), but less on associations between survey items, despite estimating associations being a primary use of survey data. Methods: We estimated average differences (means and proportions) in item responses according to mode using data from an experiment in which participants were randomized to respond via web questionnaire, video interview, or in-person interview. We then used these estimates to simulate (counterfactual) data from two web-then-in-person sequential mixed-mode surveys of young adults in the United Kingdom (Next Steps and the Millennium Cohort Study, MCS) as if they were collected using a single mode, web. We estimated associations between variables using this simulated data and compared these against associations obtained using the observed (i.e., factual) data; the difference between the two provides a measure of variation in variable correlations reflecting the mixed-mode survey design. Results: Mode effects were observed for an array of items comparing web with in-person and video modes; differences between video and in-person were more muted. Participants in web were more likely to report negatively valent values for sensitive items, in particular (e.g., poorer physical and mental health, lower life satisfaction, and greater financial difficulty), though effect sizes were generally small with few exceeding 0.2 SD and all below 0.3 SD. The impact of mode effects on variable associations in the mixed-mode surveys was also generally small: a 0.004 SD change in absolute effect sizes on average in both Next Steps and the MCS. Conclusion: Items related to sensitive characteristics showed small mode effects when comparing web and in-person or video interview modes. Mode effects did not appear to translate into substantial changes to associations between variables as assessed in a sequential web-then-in-person mixed-mode survey compared with a counterfactual web-only survey.

Date: 2026-07-31
References: Add references at CitEc
Citations:

Downloads: (external link)
https://osf.io/download/6a6c948a4ff4a241b10e4e34/

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:osf:socarx:r97ht_v1

DOI: 10.31219/osf.io/r97ht_v1

Access Statistics for this paper

More papers in SocArXiv from Center for Open Science
Bibliographic data for series maintained by OSF ().