Hypothesis testing in adaptively sampled data: ART to maximize power beyond iid sampling

Dae Woong Ham () and Jiaze Qiu ()
Additional contact information
Dae Woong Ham: Harvard University
Jiaze Qiu: Harvard University

TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, 2023, vol. 32, issue 3, No 7, 998-1037

Abstract: Abstract Testing whether a variable of interest affects the outcome is one of the most fundamental problems in statistics and is often the main scientific question of interest. To tackle this problem, the conditional randomization test (CRT) is widely used to test the independence of variable(s) of interest (X) with an outcome (Y) holding other variable(s) (Z) fixed. The CRT uses “Model-X” inference framework that relies solely on the iid sampling of (X, Z) to produce exact finite-sample p values that are constructed using any test statistic. We propose a new method, the adaptive randomization test (ART), that tackles the same independence problem while allowing the data to be adaptively sampled. Like the CRT, the ART relies solely on knowing the (adaptive) sampling distribution of (X, Z). Although the ART allows practitioners to flexibly design and analyze adaptive experiments, the method itself does not guarantee a powerful adaptive sampling procedure. For this reason, we show substantial power gains obtained from adaptively sampling compared to the typical iid sampling procedure in a multi-arm bandit setting and an application in conjoint analysis. We believe that the proposed adaptive procedure is successful because it takes arms that may initially look like “fake” signals due to random chance and stabilizes them closer to “null” signals and samples more/less from signal/null arms.

Keywords: Conditional independence testing; Randomization inference; Adaptive sampling; Model-X; Nonparametric testing; 62G10; 62G09 (search for similar items in EconPapers)
Date: 2023
References: Add references at CitEc
Citations:

Downloads: (external link)
http://link.springer.com/10.1007/s11749-023-00861-2 Abstract (text/html)
Access to the full text of the articles in this series is restricted.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:testjl:v:32:y:2023:i:3:d:10.1007_s11749-023-00861-2

Ordering information: This journal article can be ordered from
http://www.springer. ... cs/journal/11749/PS2

DOI: 10.1007/s11749-023-00861-2

Access Statistics for this article

TEST: An Official Journal of the Spanish Society of Statistics and Operations Research is currently edited by Alfonso Gordaliza and Ana F. Militino

More articles in TEST: An Official Journal of the Spanish Society of Statistics and Operations Research from Springer, Sociedad de Estadística e Investigación Operativa
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().