Evaluating methods to prevent and detect inattentive respondents in web surveys

Lukas Olbrich, Joseph W. Sakshaug and Eric Lewandowski

No py9gz, SocArXiv from Center for Open Science

Abstract: Inattentive respondents pose a substantial threat to data quality in web surveys. To minimize this threat, we evaluate methods for preventing and detecting inattentive responding and investigate its impacts on substantive research. First, we test the effect of asking respondents to commit to providing high-quality responses at the beginning of the survey on various data quality measures. Second, we compare the proportion of flagged respondents for two versions of an attention check item instructing them to select a specific response vs. leaving the item blank. Third, we propose a timestamp-based cluster analysis approach that identifies clusters of respondents who exhibit different speeding behaviors. Lastly, we investigate the impact of inattentive respondents on univariate, regression, and experimental analyses. Our findings show that the commitment pledge had no effect on the data quality measures. Instructing respondents to leave the item blank instead of providing a specific response significantly increased the rate of flagged respondents (by 16.8 percentage points). The timestamp-based clustering approach efficiently identified clusters of likely inattentive respondents and outperformed a related method, while providing additional insights on speeding behavior throughout the questionnaire. Lastly, we show that inattentive respondents can have substantial impacts on substantive analyses.

Date: 2024-07-22
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://osf.io/download/669f30c8d0ce5200c180d582/

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:osf:socarx:py9gz

DOI: 10.31219/osf.io/py9gz

Access Statistics for this paper

More papers in SocArXiv from Center for Open Science
Bibliographic data for series maintained by OSF ().