Another Look at the Lady Tasting Tea and Differences Between Permutation Tests and Randomisation Tests

Jesse Hemerik and Jelle J. Goeman

International Statistical Review, 2021, vol. 89, issue 2, 367-381

Abstract: The statistical literature is known to be inconsistent in the use of the terms ‘permutation test’ and ‘randomisation test’. Several authors successfully argue that these terms should be used to refer to two distinct classes of tests and that there are major conceptual differences between these classes. The present paper explains an important difference in mathematical reasoning between these classes: a permutation test fundamentally requires that the set of permutations has a group structure, in the algebraic sense; the reasoning behind a randomisation test is not based on such a group structure, and it is possible to use an experimental design that does not correspond to a group. In particular, we can use a randomisation scheme where the number of possible treatment patterns is larger than in standard experimental designs. This leads to exact p values of improved resolution, providing increased power for very small significance levels, at the cost of decreased power for larger significance levels. We discuss applications in randomised trials and elsewhere. Further, we explain that Fisher's famous Lady Tasting Tea experiment, which is commonly referred to as the first permutation test, is in fact a randomisation test. This distinction is important to avoid confusion and invalid tests.

Date: 2021
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (5)

Downloads: (external link)
https://doi.org/10.1111/insr.12431

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:bla:istatr:v:89:y:2021:i:2:p:367-381

Ordering information: This journal article can be ordered from
http://www.blackwell ... bs.asp?ref=0306-7734

Access Statistics for this article

International Statistical Review is currently edited by Eugene Seneta and Kees Zeelenberg

More articles in International Statistical Review from International Statistical Institute Contact information at EDIRC.
Bibliographic data for series maintained by Wiley Content Delivery ().