A primer on power and sample size calculations for randomisation inference with experimental data

Brandon Hauser and Mauricio Olivares

Fiscal Studies, 2025, vol. 46, issue 3, 349-371

Abstract: This paper revisits the problem of power analysis and sample size calculations in randomised experiments, with a focus on settings where inference on average treatment effects is conducted using randomisation tests. While standard formulas based on the two‐sample t$t$‐test are widely used in practice, we show that these calculations may yield misleading results when directly applied to randomisation‐based inference – unless certain assumptions are met. We demonstrate that differences in potential outcome variances or unequal group sizes can distort the behaviour of the randomisation test, leading to incorrect power and flawed sample size calculations. However, a simple adjustment – studentising the test statistic – restores the validity of the randomisation test in large samples. This adjustment allows researchers to safely apply standard power and sample size formulas, even when using randomisation inference. We extend these results to a range of experimental designs commonly used in applied economics, including stratified randomisation, matched pairs and cluster‐randomised trials. Throughout, we provide practical guidance to help researchers ensure that their design‐stage calculations remain valid under the inferential methods they plan to use.

Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
https://doi.org/10.1111/1475-5890.70004

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:wly:fistud:v:46:y:2025:i:3:p:349-371

Access Statistics for this article

More articles in Fiscal Studies from John Wiley & Sons
Bibliographic data for series maintained by Wiley Content Delivery ().