On the Number of Replications in Resampling Tests and Monte Carlo Simulation Studies

Daniel Gaigall and Julian Gerstenberg

The American Statistician, 2026, vol. 80, issue 3, 385-394

Abstract: We investigate rejection probabilities of statistical tests based on resampling procedures. Our general framework under consideration covers, in particular, bootstrap and permutation techniques. It turns out that specific properties of the p-value distribution play a key role, namely convexity or concavity, the Bernstein property and those of beta mixture models. We provide a detailed analysis and clarify how these properties relate to each other. We derive new bounds for the rejection probability. The results link the number of replications with size and power of the test. Numerical considerations demonstrate the quality of the bounds. An important application is the nested simulation estimator in Monte Carlo simulation studies. Our findings indicate that a moderate or even rather small number of replications is sufficient to obtain useful simulation results. This enables a substantial reduction of the computational effort in Monte Carlo simulation studies.

Date: 2026
References: Add references at CitEc
Citations:

Downloads: (external link)
http://hdl.handle.net/10.1080/00031305.2025.2612197 (text/html)
Access to full text is restricted to subscribers.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:taf:amstat:v:80:y:2026:i:3:p:385-394

Ordering information: This journal article can be ordered from
http://www.tandfonline.com/pricing/journal/UTAS20

DOI: 10.1080/00031305.2025.2612197

Access Statistics for this article

The American Statistician is currently edited by Eric Sampson

More articles in The American Statistician from Taylor & Francis Journals
Bibliographic data for series maintained by Chris Longhurst ().