Multiple Hypothesis Testing in Experimental Economics

John List, Azeem Shaikh and Yang Xu

Artefactual Field Experiments from The Field Experiments Website

Abstract: Empiricism in the sciences allows us to test theories, formulate optimal policies, and learn how the world works. In this manner, it is critical that our empirical work provides accurate conclusions about underlying data patterns. False positives represent an especially important problem, as vast public and private resources can be misguided if we base decisions on false discovery. This study explores one especially pernicious influence on false positives-multiple hypothesis testing (MHT). While MHT potentially affects all types of empirical work, we consider three common scenarios where MHT influences inference within experimental economics: jointly identifying treatment effects for a set of outcomes, estimating heterogenous treatment effects through subgroup analysis, and conducting hypothesis testing for multiple treatment conditions. Building upon the work of Romano and Wolf (2010), we present a correction procedure that incorporates the three scenarios, and illustrate the improvement in power by comparing our results with those obtained by the classic studies due to Bonferroni (1935) and Holm (1979). Importantly, under weak assumptions, our testing procedure asymptotically controls the familywise error rate - the probability of one false rejection - and is asymptotically balanced. We showcase our approach by revisiting the data reported in Karlan and List (2007), to deepen our understanding of why people give to charitable causes.

Date: 2016
New Economics Papers: this item is included in nep-ecm and nep-exp
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (180)

Downloads: (external link)
http://s3.amazonaws.com/fieldexperiments-papers2/papers/00402.pdf

Related works:
Journal Article: Multiple hypothesis testing in experimental economics (2019)
Software Item: MHTEXP: Stata module to perform multiple hypothesis testing correction procedure (2016)
Software Item: Stata module to perform mutliple hypothesis testing correction procedure (2016)
Working Paper: Multiple Hypothesis Testing in Experimental Economics (2016)
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:feb:artefa:00402

Access Statistics for this paper

More papers in Artefactual Field Experiments from The Field Experiments Website
Bibliographic data for series maintained by Francesca Pagnotta ().