Multiple Hypothesis Testing in Experimental Economics

John List, Azeem Shaikh and Yang Xu

No 21875, NBER Working Papers from National Bureau of Economic Research, Inc

Abstract: Empiricism in the sciences allows us to test theories, formulate optimal policies, and learn how the world works. In this manner, it is critical that our empirical work provides accurate conclusions about underlying data patterns. False positives represent an especially important problem, as vast public and private resources can be misguided if we base decisions on false discovery. This study explores one especially pernicious influence on false positives—multiple hypothesis testing (MHT). While MHT potentially affects all types of empirical work, we consider three common scenarios where MHT influences inference within experimental economics: jointly identifying treatment effects for a set of outcomes, estimating heterogeneous treatment effects through subgroup analysis, and conducting hypothesis testing for multiple treatment conditions. Building upon the work of Romano and Wolf (2010), we present a correction procedure that incorporates the three scenarios, and illustrate the improvement in power by comparing our results with those obtained by the classic studies due to Bonferroni (1935) and Holm (1979). Importantly, under weak assumptions, our testing procedure asymptotically controls the familywise error rate – the probability of one false rejection – and is asymptotically balanced. We showcase our approach by revisiting the data reported in Karlan and List (2007), to deepen our understanding of why people give to charitable causes.

JEL-codes: C1 C9 C91 C92 C93 (search for similar items in EconPapers)
Date: 2016-01
New Economics Papers: this item is included in nep-exp
Note: EEE PE
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (193)

Published as John A. List & Azeem M. Shaikh & Yang Xu, 2019. "Multiple hypothesis testing in experimental economics," Experimental Economics, Springer;Economic Science Association, vol. 22(4), pages 773-793, December.
Published as John A. List & Azeem M. Shaikh & Yang Xu, 2019. "Multiple hypothesis testing in experimental economics," Experimental Economics, vol 22(4), pages 773-793.

Downloads: (external link)
http://www.nber.org/papers/w21875.pdf (application/pdf)

Related works:
Journal Article: Multiple hypothesis testing in experimental economics (2019)
Working Paper: Multiple Hypothesis Testing in Experimental Economics (2016)
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:nbr:nberwo:21875

Ordering information: This working paper can be ordered from
http://www.nber.org/papers/w21875

Access Statistics for this paper

More papers in NBER Working Papers from National Bureau of Economic Research, Inc National Bureau of Economic Research, 1050 Massachusetts Avenue Cambridge, MA 02138, U.S.A.. Contact information at EDIRC.
Bibliographic data for series maintained by ().