A new four-arm within-study comparison: Design, implementation, and data

Bryan Keller, Vivian C Wong, Sangbaek Park, Jingru Zhang, Patrick Sheehan and Peter M. Steiner
Additional contact information
Vivian C Wong: University of Virginia

No 2gur9, OSF Preprints from Center for Open Science

Abstract: Within-study comparisons (WSCs) use real, rather than simulated, data to compare estimates from observational studies against a benchmark randomized controlled trial (RCT). A primary goal of WSCs is to assess whether well-designed quasi-experimental designs (QEDs) can produce internally valid causal effect estimates comparable to those from RCTs. In this paper, we describe the design and implementation of a new type of WSC. Motivated by Shadish et al. (2008), we examine the impact of a mathematics training intervention and a vocabulary study session on posttest scores for mathematics and vocabulary, respectively. We extend the original design in three ways. First, before random assignment, we ask participants to express a preference for either the mathematics or vocabulary training session, after which they are randomly assigned regardless of preferences. This allows us to experimentally identify and estimate the overall average treatment effect (ATE) and two conditional ATEs: the average treatment effect on the treated (ATT) and the average treatment effect on the untreated (ATU). Second, participant recruitment and sample size (N = 2200) were determined through power analyses for comparing RCT and QED estimates, ensuring sufficient power for methodological comparisons. Finally, the study’s eligibility criteria, recruitment, treatment allocation, and analysis plan were preregistered on the Open Science Foundation platform, and the data are publicly accessible. We believe that this WSC design and the resulting data set will be valuable for researchers seeking to evaluate causal inference methods and test identification assumptions using real-world data.

Date: 2024-09-10
New Economics Papers: this item is included in nep-exp, nep-ipr and nep-mac
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://osf.io/download/66df54047f573a030f77af7b/

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:osf:osfxxx:2gur9

DOI: 10.31219/osf.io/2gur9

Access Statistics for this paper

More papers in OSF Preprints from Center for Open Science
Bibliographic data for series maintained by OSF ().