How many is enough? Sample Size in Staggered Difference-in-Differences Designs

Benjamin Egerod and Florian M Hollenbach
Additional contact information
Florian M Hollenbach: Copenhagen Business School

No ac5ru_v1, OSF Preprints from Center for Open Science

Abstract: In difference-in-differences designs with staggered treatment timing and dynamic treatment effects, the two-way fixed effects estimator fails to recover an interpretable causal estimate. A large number of estimators have been proposed to remedy this issue. The flexibility of these estimators, however, increases their variance. This can lead to statistical tests with low statistical power. As a consequence, small effects are unlikely to be discovered. Additionally, under low power, if a statistically significant estimate is recovered, the estimate is often wrongly signed and/or greatly exaggerated. Using simulations on real-world data on US States, we show that effect sizes of 10 to 15% are necessary for the recently developed estimators for staggered difference-in-differences to produce statistical tests that achieve 80% power. Further, conditional on statistical significance, when the intervention generates weak effects, estimators recover the wrong sign in approximately 10% of the simulations and overestimate the true effect by several hundred percent on average. We use data on publicly traded firms to investigate which sample size is needed for a staggered difference-in-differences analysis to be informative. We find that depending on the dependent variable and effect size, even the most efficient estimators generally need more than 250 units to achieve reasonable power. We conclude with a discussion of how this type of ‘design analysis’ ought to be used by researchers before estimating staggered difference-in-differences models. We also discuss how power may under certain conditions be improved if a study is re-designed, e.g., by examining county-level outcomes with state-level interventions.

Date: 2024-06-18
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://osf.io/download/6672d409b5a03600dc9ff454/

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:osf:osfxxx:ac5ru_v1

DOI: 10.31219/osf.io/ac5ru_v1

Access Statistics for this paper

More papers in OSF Preprints from Center for Open Science
Bibliographic data for series maintained by OSF ().