Implementing Monte Carlo tests with p‐value buckets

Axel Gandy, Georg Hahn and Dong Ding

Scandinavian Journal of Statistics, 2020, vol. 47, issue 3, 950-967

Abstract: Software packages usually report the results of statistical tests using p‐values. Users often interpret these values by comparing them with standard thresholds, for example, 0.1, 1, and 5%, which is sometimes reinforced by a star rating (***, **, and *, respectively). We consider an arbitrary statistical test whose p‐value p is not available explicitly, but can be approximated by Monte Carlo samples, for example, by bootstrap or permutation tests. The standard implementation of such tests usually draws a fixed number of samples to approximate p. However, the probability that the exact and the approximated p‐value lie on different sides of a threshold (the resampling risk) can be high, particularly for p‐values close to a threshold. We present a method to overcome this. We consider a finite set of user‐specified intervals that cover [0, 1] and that can be overlapping. We call these p‐value buckets. We present algorithms that, with arbitrarily high probability, return a p‐value bucket containing p. We prove that for both a bounded resampling risk and a finite runtime, overlapping buckets need to be employed, and that our methods both bound the resampling risk and guarantee a finite runtime for such overlapping buckets. To interpret decisions with overlapping buckets, we propose an extension of the star rating system. We demonstrate that our methods are suitable for use in standard software, including for low p‐value thresholds occurring in multiple testing settings, and that they can be computationally more efficient than standard implementations.

Date: 2020
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://doi.org/10.1111/sjos.12434

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:bla:scjsta:v:47:y:2020:i:3:p:950-967

Ordering information: This journal article can be ordered from
http://www.blackwell ... bs.asp?ref=0303-6898

Access Statistics for this article

Scandinavian Journal of Statistics is currently edited by ÿrnulf Borgan and Bo Lindqvist

More articles in Scandinavian Journal of Statistics from Danish Society for Theoretical Statistics, Finnish Statistical Society, Norwegian Statistical Association, Swedish Statistical Association
Bibliographic data for series maintained by Wiley Content Delivery ().