How Much Should We Trust Differences-in-Differences Estimates?

Marianne Bertrand, Esther Duflo and Sendhil Mullainathan ()

No 8841, NBER Working Papers from National Bureau of Economic Research, Inc

Abstract: Most Difference-in-Difference (DD) papers rely on many years of data and focus on serially correlated outcomes. Yet almost all these papers ignore the bias in the estimated standard errors that serial correlation introduce4s. This is especially troubling because the independent variable of interest in DD estimation (e.g., the passage of law) is itself very serially correlated, which will exacerbate the bias in standard errors. To illustrate the severity of this issue, we randomly generate placebo laws in state-level data on female wages from the Current Population Survey. For each law, we use OLS to compute the DD estimate of its 'effect' as well as the standard error for this estimate. The standard errors are severely biased: with about 20 years of data, DD estimation finds an 'effect' significant at the 5% level of up to 45% of the placebo laws. Two very simple techniques can solve this problem for large sample sizes. The first technique consists in collapsing the data and ignoring the time-series variation altogether; the second technique is to estimate standard errors while allowing for an arbitrary covariance structure between time periods. We also suggest a third technique, based on randomization inference testing methods, which works well irrespective of sample size. This technique uses the empirical distribution of estimated effects for placebo laws to form the test distribution.

JEL-codes: C10 C13 (search for similar items in EconPapers)
Date: 2002-03
Note: CH LS PE
View list of references View citations in EconPapers

Downloads: (external link)
http://www.nber.org/papers/w8841.pdf (application/pdf)
Access to the full text is generally limited to series subscribers, however if the top level domain of the client browser is in a developing country or transition economy free access is provided. More information about subscriptions and free access is available at http://www.nber.org/wwphelp.html.

Related works:
Journal Article: How Much Should We Trust Differences-in-Differences Estimates? (2004)
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: http://EconPapers.repec.org/RePEc:nbr:nberwo:8841

Ordering information: This working paper can be ordered from
http://www.nber.org/papers/w8841
The price is Paper copy available by mail.

Access Statistics for this paper

More papers in NBER Working Papers from National Bureau of Economic Research, Inc
Address: National Bureau of Economic Research, 1050 Massachusetts Avenue Cambridge, MA 02138, U.S.A.
Contact information at EDIRC.
Series data maintained by ().