Identifying Data Quality Challenges in Online Opt-In Panels Using Cognitive Interviews in English and Spanish

Trejo Yazmín García (), Meyers Mikelyn (), Martinez Mandi (), O’Brien Angela (), Goerman Patricia () and Class Betsarí Otero ()
Additional contact information
Trejo Yazmín García: U.S. Census Bureau, 4600 Silver Hill Road, Suitland-Silver Hill, MD 20746, U.S.A.
Meyers Mikelyn: U.S. Census Bureau, 4600 Silver Hill Road, Suitland-Silver Hill, MD 20746, U.S.A.
Martinez Mandi: U.S. Census Bureau, 4600 Silver Hill Road, Suitland-Silver Hill, MD 20746, U.S.A.
O’Brien Angela: U.S. Census Bureau, 4600 Silver Hill Road, Suitland-Silver Hill, MD 20746, U.S.A.
Goerman Patricia: U.S. Census Bureau, 4600 Silver Hill Road, Suitland-Silver Hill, MD 20746, U.S.A.
Class Betsarí Otero: U.S. Census Bureau, 4600 Silver Hill Road, Suitland-Silver Hill, MD 20746, U.S.A.

Journal of Official Statistics, 2022, vol. 38, issue 3, 793-822

Abstract: In this article, we evaluate how the analysis of open-ended probes in an online cognitive interview can serve as a metric to identify cases that should be excluded due to disingenuous responses by ineligible respondents. We analyze data collected in 2019 via an online opt-in panel in English and Spanish to pretest a public opinion questionnaire (n = 265 in English and 199 in Spanish). We find that analyzing open-ended probes allowed us to flag cases completed by respondents who demonstrated problematic behaviors (e.g., answering many probes with repetitive textual patterns, by typing random characters, etc.), as well as to identify cases completed by ineligible respondents posing as eligible respondents (i.e., non-Spanish-speakers posing as Spanish-speakers). These findings indicate that data collected for multilingual pretesting research using online opt-in panels likely require additional evaluations of data quality. We find that open-ended probes can help determine which cases should be replaced when conducting pretesting using opt-in panels. We argue that open-ended probes in online cognitive interviews, while more time consuming and expensive to analyze than close-ended questions, serve as a valuable method of verifying response quality and respondent eligibility, particularly for researchers conducting multilingual surveys with online opt-in panels.

Keywords: Data quality; cognitive interviews; open-ended probes; online opt-in panels (search for similar items in EconPapers)
Date: 2022
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://doi.org/10.2478/jos-2022-0035 (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:vrs:offsta:v:38:y:2022:i:3:p:793-822:n:6

DOI: 10.2478/jos-2022-0035

Access Statistics for this article

Journal of Official Statistics is currently edited by Annica Isaksson and Ingegerd Jansson

More articles in Journal of Official Statistics from Sciendo
Bibliographic data for series maintained by Peter Golla ().