Limitations of Randomization Tests in Finite Samples

Deniz Dutz and Xinyi Zhang

Papers from arXiv.org

Abstract: Randomization tests yield exact finite-sample Type 1 error control when the null satisfies the randomization hypothesis. However, achieving these guarantees in practice often requires stronger conditions than the null hypothesis of primary interest. For instance, sign-change tests for mean zero require symmetry and fail to control finite-sample error for non-symmetric mean-zero distributions. We investigate whether such limitations stem from specific test choices or reflect a fundamental inability to construct valid randomization tests for certain hypotheses. We develop a framework providing a simple necessary and sufficient condition for when null hypotheses admit randomization tests. Applying this framework to one-sample tests, we provide characterizations of which nulls satisfy this condition for both finite and continuous supports. In doing so, we prove that certain null hypotheses -- including mean zero -- do not admit randomization tests. We further show that nulls that admit randomization tests based on linear group actions correspond only to subsets of symmetric or normal distributions. Overall, our findings affirm that practitioners are not inadvertently incurring additional Type 1 error when using existing tests and further motivate focusing on the asymptotic validity of randomization tests.

Date: 2025-12
References: Add references at CitEc
Citations:

Downloads: (external link)
http://arxiv.org/pdf/2512.07099 Latest version (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:arx:papers:2512.07099

Access Statistics for this paper

More papers in Papers from arXiv.org
Bibliographic data for series maintained by arXiv administrators ().