Collecting samples from online services: How to use screeners to improve data quality

Aaron D. Arndt, John B. Ford, Barry J. Babin and Vinh Luong

International Journal of Research in Marketing, 2022, vol. 39, issue 1, 117-133

Abstract: Increasingly, marketing and consumer researchers rely on online data collection services. While actively-managed data collection services directly assist with the sampling process, minimally-managed data collection services, such as Amazon’s Mechanical Turk (MTurk), leave researchers solely responsible for recruiting, screening, cleaning, and evaluating responses. The research reported here proposes a 2 × 2 framework based on sampling goal and methodology for screening and evaluating the quality of online samples. By sampling goals, screeners can be categorized as selection, which involves matching the sample with the targeted population; or as accuracy, which involves ensuring that participants are appropriately attentive. By methodology, screeners can be categorized as direct, which screens individual responses; and as statistical, which provides quantitative signals of low quality. Multiple screeners for each of the four categories are compared across three MTurk samples, two actively-managed data collection samples (Qualtrics and Dynata), and a student sample. The results suggest the need for screening in every online sample, particularly for the MTurk samples, with the fewest supplier-provided filters. Recommendations are provided for researchers and journal reviewers that provide greater transparency with respect to sample practices.

Keywords: Online samples; Sampling; MTurk; Screening; Data quality; Integrity (search for similar items in EconPapers)
Date: 2022
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (11)

Downloads: (external link)
http://www.sciencedirect.com/science/article/pii/S0167811621000203
Full text for ScienceDirect subscribers only

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:eee:ijrema:v:39:y:2022:i:1:p:117-133

DOI: 10.1016/j.ijresmar.2021.05.001

Access Statistics for this article

International Journal of Research in Marketing is currently edited by Roland Rust

More articles in International Journal of Research in Marketing from Elsevier
Bibliographic data for series maintained by Catherine Liu ().