Controlling the number of significant effects in multiple testing

Jacobo de Uña-Álvarez

Communications in Statistics - Theory and Methods, 2025, vol. 54, issue 20, 6493-6511

Abstract: In multiple testing, several criteria to control for type I errors exist. The false discovery rate, which evaluates the expected proportion of false discoveries among the rejected null hypotheses, has become the standard approach in this setting. However, false discovery rate control may be too conservative when the effects are weak, that is, when the true alternative hypotheses are close to their corresponding nulls. In this article, we alternatively propose to focus on the number of significant effects, where ’significant’ refers to a pre-specified threshold γ. In particular, a (1−α)-lower confidence bound N for the number of non true null hypotheses with p-value below γ is provided. When one rejects the nulls corresponding to the N smallest p-values, the probability that the number of false positives exceeds the number of false negatives among the significant effects is bounded by α. The method aims to improve the statistical power in the multiple testing setup while avoiding an unjustifiably large amount of rejected nulls. Relative merits of the proposed criterion are discussed. Procedures to control for the number of significant effects in practice are introduced and investigated both theoretically and through simulations. Illustrative real data applications are given.

Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
http://hdl.handle.net/10.1080/03610926.2025.2458188 (text/html)
Access to full text is restricted to subscribers.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:taf:lstaxx:v:54:y:2025:i:20:p:6493-6511

Ordering information: This journal article can be ordered from
http://www.tandfonline.com/pricing/journal/lsta20

DOI: 10.1080/03610926.2025.2458188

Access Statistics for this article

Communications in Statistics - Theory and Methods is currently edited by Debbie Iscoe

More articles in Communications in Statistics - Theory and Methods from Taylor & Francis Journals
Bibliographic data for series maintained by Chris Longhurst ().