Highly Powered Analysis Plans

Michael Anderson and Jeremy Magruder

No 29843, NBER Working Papers from National Bureau of Economic Research, Inc

Abstract: Formal analysis plans limit false discoveries by registering and multiplicity adjusting statistical tests. As each registered test reduces power on other tests, researchers prune hypotheses based on prior knowledge, often by combining related indicators into evenly-weighted indices. We propose two improvements to maximize learning within these types of analysis plans. First, we develop data-driven optimized indices that can yield more powerful tests than evenly-weighted indices. Second, we discuss organizing the logical structure of an analysis plan into a gated tree that directs type I error towards these high-powered tests. In simulations we show that researchers may prefer these "optimus gates" across a wide range of data-generating processes. We then assess our strategy using the community-driven development (CDD) application from Casey et al. (2012) and the Oregon Health Insurance Experiment from Finkelstein et al. (2012). We find substantial power gains in both applications, meaningfully changing the conclusions of Casey et al. (2012).

JEL-codes: C12 C55 C81 C9 C93 O1 (search for similar items in EconPapers)
Date: 2022-03
New Economics Papers: this item is included in nep-exp and nep-hea
Note: DEV LS PE
References: Add references at CitEc
Citations:

Downloads: (external link)
http://www.nber.org/papers/w29843.pdf (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:nbr:nberwo:29843

Ordering information: This working paper can be ordered from
http://www.nber.org/papers/w29843

Access Statistics for this paper

More papers in NBER Working Papers from National Bureau of Economic Research, Inc National Bureau of Economic Research, 1050 Massachusetts Avenue Cambridge, MA 02138, U.S.A.. Contact information at EDIRC.
Bibliographic data for series maintained by ().