How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It

Jacob M. Montgomery, Brendan Nyhan and Michelle Torres

American Journal of Political Science, 2018, vol. 62, issue 3, 760-775

Abstract: In principle, experiments offer a straightforward method for social scientists to accurately estimate causal effects. However, scholars often unwittingly distort treatment effect estimates by conditioning on variables that could be affected by their experimental manipulation. Typical examples include controlling for posttreatment variables in statistical models, eliminating observations based on posttreatment criteria, or subsetting the data based on posttreatment variables. Though these modeling choices are intended to address common problems encountered when conducting experiments, they can bias estimates of causal effects. Moreover, problems associated with conditioning on posttreatment variables remain largely unrecognized in the field, which we show frequently publishes experimental studies using these practices in our discipline's most prestigious journals. We demonstrate the severity of experimental posttreatment bias analytically and document the magnitude of the potential distortions it induces using visualizations and reanalyses of real‐world data. We conclude by providing applied researchers with recommendations for best practice.

Date: 2018
References: Add references at CitEc
Citations: View citations in EconPapers (67)

Downloads: (external link)
https://doi.org/10.1111/ajps.12357

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:wly:amposc:v:62:y:2018:i:3:p:760-775

Access Statistics for this article

More articles in American Journal of Political Science from John Wiley & Sons
Bibliographic data for series maintained by Wiley Content Delivery ().