Reusing Natural Experiments

Davidson Heath, Matthew Ringgenberg, Mehrdad Samadi (mehrdad.samadi@frb.gov) and Ingrid M. Werner
Additional contact information
Mehrdad Samadi: https://www.federalreserve.gov/econres/mehrdad-samadi.htm
Ingrid M. Werner: https://fisher.osu.edu/people/werner.47

No 1339, International Finance Discussion Papers from Board of Governors of the Federal Reserve System (U.S.)

Abstract: After a natural experiment is first used, other researchers often reuse the setting, examining different outcome variables. We use simulations based on real data to illustrate the multiple hypothesis testing problem that arises when researchers reuse natural experiments. We then provide guidance for future inference based on popular empirical settings including difference-in-differences regressions, instrumental variables regressions, and regression discontinuity designs. When we apply our guidance to two extensively studied natural experiments, business combination laws and the Regulation SHO pilot, we find that many results that were statistically significant using single hypothesis testing do not survive corrections for multiple hypothesis testing.

Keywords: False Positive; Multiple Hypothesis Testing; Natural Experiments (search for similar items in EconPapers)
JEL-codes: G10 (search for similar items in EconPapers)
Date: 2022-04-01
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (3)

Downloads: (external link)
https://www.federalreserve.gov/econres/ifdp/files/ifdp1339.pdf (application/pdf)

Related works:
Working Paper: Reusing Natural Experiments (2020)
Working Paper: Reusing Natural Experiments (2019)
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:fip:fedgif:1339

DOI: 10.17016/IFDP.2022.1339

Access Statistics for this paper

More papers in International Finance Discussion Papers from Board of Governors of the Federal Reserve System (U.S.) Contact information at EDIRC.
Bibliographic data for series maintained by Ryan Wolfslayer ; Keisha Fournillier (ryan.d.wolfslayer@frb.gov).