Limitations of Randomization Tests in Finite Samples

Deniz Dutz and Xinyi Zhang

Papers from arXiv.org

Abstract: Randomization tests deliver exact finite-sample Type 1 error control when the null satisfies the randomization hypothesis. In practice, achieving these guarantees often requires stronger conditions than the null hypothesis of primary interest. For example, sign-change tests of mean zero require symmetry and need not control finite-sample size for non-symmetric mean-zero distributions. We investigate whether the mismatch between the null and the invariance conditions required for exactness reflects the use of particular transformations or a more fundamental limitation. We provide a simple necessary and sufficient condition for a null hypothesis to admit a randomization test. Applying this framework to one-sample problems, we characterize the nulls that admit randomization tests on finite supports and derive impossibility results on continuous supports. In particular, we show that several common nulls, including mean zero, do not admit randomization tests. We further show that, among one-sample tests using linear group actions, the admissible nulls are limited to subsets of symmetric or Gaussian distributions. These results confirm that the absence of exact finite-sample validity is inherent for many commonly studied nulls and that practitioners using existing tests are not foregoing feasible exact alternatives.

Date: 2025-12, Revised 2026-04
New Economics Papers: this item is included in nep-ecm
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
http://arxiv.org/pdf/2512.07099 Latest version (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:arx:papers:2512.07099

Access Statistics for this paper

More papers in Papers from arXiv.org
Bibliographic data for series maintained by arXiv administrators ().