MTurk ‘Unscrubbed’: Exploring the good, the ‘Super’, and the unreliable on Amazon’s Mechanical Turk

Jeanette Deetlefs (), Mathew Chylinski () and Andreas Ortmann
Additional contact information
Jeanette Deetlefs: School of Marketing, UNSW Business School, UNSW
Mathew Chylinski: School of Marketing, UNSW Business School, UNSW

No 2015-20, Discussion Papers from School of Economics, The University of New South Wales

Abstract: Widely accepted as a low-cost, fast-turnaround solution with acceptable validity, Amazon’s Mechanical Turk (MTurk) is increasingly being used to source participants for academic studies (Berinsky et al. 2012; Bohannon 2011; Chandler et al. 2014; Mason and Suri 2012). Yet two commonly raised concerns remain: the presence of quasi-professional respondents, or “Super-Turkers”, and the presence of “Spammers”, those that compromise quality while optimising their pay rate. We isolate the influence on research results of experienced subjects (Super-Turkers), and of unreliable subjects (Spammers), jointly and separately. Jointly including these subjects produces very similar results to jointly excluding them, yet effect sizes decrease disproportionately to their sample representation. Furthermore, separately including experienced subjects in research results is shown to be as problematic as inclusion of unreliable subjects, although the noise introduced by these subjects is divergent and measure dependent. Hence removing only one of these types of respondents can be even more damaging to the reliability of results, than including both.

Keywords: data collection; experimentation; field experiment; internet; Mechanical Turk (search for similar items in EconPapers)
JEL-codes: C91 C93 D80 (search for similar items in EconPapers)
Pages: 47 pages
Date: 2015-09
New Economics Papers: this item is included in nep-exp
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (1)

Downloads: (external link)
http://research.economics.unsw.edu.au/RePEc/papers/2015-20.pdf (application/pdf)
Our link check indicates that this URL is bad, the error code is: 503 Service Unavailable: Back-end server is at capacity

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:swe:wpaper:2015-20a

Access Statistics for this paper

More papers in Discussion Papers from School of Economics, The University of New South Wales Contact information at EDIRC.
Bibliographic data for series maintained by Hongyi Li ().