Is It Who You Are or Where You Are? Accounting for Compositional Differences in Cross-Site Treatment Effect Variation

Benjamin Lu, Eli Ben-Michael, Avi Feller and Luke Miratrix
Additional contact information
Benjamin Lu: University of California, Berkeley
Eli Ben-Michael: Carnegie Mellon University
Avi Feller: University of California, Berkeley
Luke Miratrix: Harvard University

Journal of Educational and Behavioral Statistics, 2023, vol. 48, issue 4, 420-453

Abstract: In multisite trials, learning about treatment effect variation across sites is critical for understanding where and for whom a program works. Unadjusted comparisons, however, capture “compositional†differences in the distributions of unit-level features as well as “contextual†differences in site-level features, including possible differences in program implementation. Our goal in this article is to adjust site-level estimates for differences in the distribution of observed unit-level features: If we can reweight (or “transport†) each site to have a common distribution of observed unit-level covariates, the remaining treatment effect variation captures contextual and unobserved compositional differences across sites. This allows us to make apples-to-apples comparisons across sites, parceling out the amount of cross-site effect variation explained by systematic differences in populations served. In this article, we develop a framework for transporting effects using approximate balancing weights, where the weights are chosen to directly optimize unit-level covariate balance between each site and the common target distribution. We first develop our approach for the general setting of transporting the effect of a single-site trial. We then extend our method to multisite trials, assess its performance via simulation, and use it to analyze a series of multisite trials of adult education and vocational training programs. In our application, we find that distributional differences are potentially masking cross-site variation. Our method is available in the balancer R package.

Keywords: multisite trials; generalizability; transportability; balancing weights; treatment effect variation (search for similar items in EconPapers)
Date: 2023
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://journals.sagepub.com/doi/10.3102/10769986231155427 (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:sae:jedbes:v:48:y:2023:i:4:p:420-453

DOI: 10.3102/10769986231155427

Access Statistics for this article

More articles in Journal of Educational and Behavioral Statistics
Bibliographic data for series maintained by SAGE Publications ().