New Evidence and Design Considerations for Repeated Measure Experiments in Survey Research

Diana Jordan, Andrew Trexler and Trent Ollerenshaw

No q6czp_v1, OSF Preprints from Center for Open Science

Abstract: An influential study in the American Political Science Review by Clifford, Sheagley, and Piston (2021) finds that including pre-treatment measures of outcome variables in survey experiments does not bias treatment effect estimates and greatly improves precision, prompting many researchers to adopt repeated measure designs. Though promising, the 2021 study remains the only empirical investigation to date to support this broad shift in experimental practice. In a large-scale partial replication, we experimentally manipulate the design of six classic experiments in political science and field all six experiments in three separate samples of U.S. adults (N = 13,163) to re-examine the central claim that repeated measure designs do not bias treatment effects. We also provide three extensions on additional design considerations regarding within-subject experimental designs, the relative separation of repeated measures within single surveys, and respondent characteristics in probability versus non-probability samples. In contrast to the original study, we find evidence of attenuation of treatment effects in repeated measure designs. However, this average attenuation bias is sufficiently small that we largely affirm the original authors' recommendation to prefer repeated measure designs in most research applications, because the large gains to statistical precision will typically produce a more accurate estimate ATE in expectation.

Date: 2025-04-02
References: Add references at CitEc
Citations:

Downloads: (external link)
https://osf.io/download/67ed45c0c6a3135e066de25c/

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:osf:osfxxx:q6czp_v1

DOI: 10.31219/osf.io/q6czp_v1

Access Statistics for this paper

More papers in OSF Preprints from Center for Open Science
Bibliographic data for series maintained by OSF ().