Intertemporal Differences Among MTurk Workers: Time-Based Sample Variations and Implications for Online Data Collection
Logan S. Casey,
Jesse Chandler,
Adam Seth Levine,
Andrew Proctor and
Dara Z. Strolovitch
SAGE Open, 2017, vol. 7, issue 2, 2158244017712774
Abstract:
The online labor market Amazon Mechanical Turk (MTurk) is an increasingly popular source of respondents for social science research. A growing body of research has examined the demographic composition of MTurk workers as compared with that of other populations. While these comparisons have revealed the ways in which MTurk workers are and are not representative of the general population, variations among samples drawn from MTurk have received less attention. This article focuses on whether MTurk sample composition varies as a function of time. Specifically, we examine whether demographic characteristics vary by (a) time of day, (b) day of week, and serial position (i.e., earlier or later in data collection), both (c) across the entire data collection and (d) within specific batches. We find that day of week differences are minimal, but that time of day and serial position are associated with small but important variations in demographic composition. This demonstrates that MTurk samples cannot be presumed identical across different studies, potentially affecting reliability, validity, and efforts to reproduce findings.
Keywords: political methodology; political science; social sciences; politics and social sciences; research methods; data collection; research methodology and design; reliability and validity; political behavior/psychology; psychology (search for similar items in EconPapers)
Date: 2017
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (7)
Downloads: (external link)
https://journals.sagepub.com/doi/10.1177/2158244017712774 (text/html)
Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.
Export reference: BibTeX
RIS (EndNote, ProCite, RefMan)
HTML/Text
Persistent link: https://EconPapers.repec.org/RePEc:sae:sagope:v:7:y:2017:i:2:p:2158244017712774
DOI: 10.1177/2158244017712774
Access Statistics for this article
More articles in SAGE Open
Bibliographic data for series maintained by SAGE Publications ().