Inference on effect size after multiple hypothesis testing

Andreas Dzemski (), Ryo Okui () and Wenjie Wang ()
Additional contact information
Andreas Dzemski: Department of Economics, School of Business, Economics and Law, Göteborg University, Postal: P.O. Box 640, SE 40530 GÖTEBORG, Sweden, https://www.gu.se/en/school-business-economics-law/economics
Ryo Okui: Faculty of Economics, the University of Tokyo, Postal: Bunkyo-ku, Tokyo 113-0033, Japan
Wenjie Wang: Division of Economics, School of Social Sciences, Nanyang Technological University, Postal: HSS-04-65, 14, Nanyang Drive, Singapore 637332

No 852, Working Papers in Economics from University of Gothenburg, Department of Economics

Abstract: Significant treatment effects are often emphasized when interpreting and summarizing empirical findings in studies that estimate multiple, possibly many, treatment effects. Under this kind of selective reporting, conventional treatment effect estimates may be biased and their corresponding confidence intervals may undercover the true effect sizes. We propose new estimators and confidence intervals that provide valid inferences on the effect sizes of the significant effects after multiple hypothesis testing. Our methods are based on the principle of selective conditional inference and complement a wide range of tests, including step-up tests and bootstrap-based step-down tests. Our approach is scalable, allowing us to study an application with over 370 estimated effects. We justify our procedure for asymptotically normal treatment effect estimators. We provide two empirical examples that demonstrate bias correction and confidence interval adjustments for significant effects. The magnitude and direction of the bias correction depend on the correlation structure of the estimated effects and whether the interpretation of the significant effects depends on the (in)significance of other effects.

Keywords: Multiple hypothesis testing; post-selection inference; conditional inference; bias correction (search for similar items in EconPapers)
JEL-codes: C12 C52 (search for similar items in EconPapers)
Pages: 36 pages
Date: 2025-04-01
References: Add references at CitEc
Citations:

Downloads: (external link)
https://gupea.ub.gu.se/handle/2077/86067 Full text (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:hhs:gunwpe:0852

Ordering information: This working paper can be ordered from

Access Statistics for this paper

More papers in Working Papers in Economics from University of Gothenburg, Department of Economics Department of Economics, School of Business, Economics and Law, University of Gothenburg, Box 640, SE 405 30 GÖTEBORG, Sweden. Contact information at EDIRC.
Bibliographic data for series maintained by Jessica Oscarsson ().