How Much Should We Trust Modern Difference-in-Differences Estimates?

Amanda Weiss

No bqmws, OSF Preprints from Center for Open Science

Abstract: When do modern difference-in-differences (DID)-style methods work for empirical political science? Scholars exploit the staggered roll-out of policies like election regulation, civil service reform, and healthcare across places to estimate causal effects - often using the two-way fixed effects (TWFE) estimator. However, recent literature has highlighted the TWFE estimator's bias in the presence of heterogeneous treatment effects and tendency to make ``forbidden comparisons" between treated units. In response, scholars have increasingly turned to modern DID estimators that promise greater robustness to real-world data problems. This paper asks how well these modern methods work for for the empirical settings and sample sizes commonly used in political science, with the U.S. states as the running example. In particular, it provides a simulation study of the performance of seven DID methods under either constant or heterogeneous effects, in an N=50 setting that mimics the American federalism natural experiment. I find that many modern methods (1) produce confidence intervals that do not include the true average effect at the specified rate and (2) are underpowered. I show that many cases of coverage problems with modern DID estimators can be addressed using the block bootstrap to estimate standard errors. However, I also show that even where identification and estimation are straightforward, the fifty-state sample poses a power problem without large average effect sizes - at least 0.5 standard deviations. I illustrate the challenges of DID research with the fifty-state panel in the case of estimating the effects of strict voter identification laws on voter turnout.

Date: 2024-08-29
New Economics Papers: this item is included in nep-ecm
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (1)

Downloads: (external link)
https://osf.io/download/66ce13425d1af01bb8b0328e/

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:osf:osfxxx:bqmws

DOI: 10.31219/osf.io/bqmws

Access Statistics for this paper

More papers in OSF Preprints from Center for Open Science
Bibliographic data for series maintained by OSF ().