No Harm in Checking: Using Factual Manipulation Checks to Assess Attentiveness in Experiments

John V. Kane and Jason Barabas

American Journal of Political Science, 2019, vol. 63, issue 1, 234-249

Abstract: Manipulation checks are often advisable in experimental studies, yet they rarely appear in practice. This lack of usage may stem from fears of distorting treatment effects and uncertainty regarding which type to use (e.g., instructional manipulation checks [IMCs] or assessments of whether stimuli alter a latent independent variable of interest). Here, we first categorize the main variants and argue that factual manipulation checks (FMCs)—that is, objective questions about key elements of the experiment—can identify individual‐level attentiveness to experimental information and, as a consequence, better enable researchers to diagnose experimental findings. We then find, through four replication studies, little evidence that FMC placement affects treatment effects, and that placing FMCs immediately post‐outcome does not attenuate FMC passage rates. Additionally, FMC and IMC passage rates are only weakly related, suggesting that each technique identifies different sets of attentive subjects. Thus, unlike other methods, FMCs can confirm attentiveness to experimental protocols.

Date: 2019
References: View complete reference list from CitEc
Citations: View citations in EconPapers (21)

Downloads: (external link)
https://doi.org/10.1111/ajps.12396

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:wly:amposc:v:63:y:2019:i:1:p:234-249

Access Statistics for this article

More articles in American Journal of Political Science from John Wiley & Sons
Bibliographic data for series maintained by Wiley Content Delivery ().