Near-Optimal A-B Testing

Nikhil Bhat (), Vivek F. Farias (), Ciamac C. Moallemi () and Deeksha Sinha ()
Additional contact information
Nikhil Bhat: Graduate School of Business, Columbia University, New York, New York 10028;
Vivek F. Farias: Sloan School of Management, Massachusetts Institute of Technology, Cambridge, Massachusetts 02139;
Ciamac C. Moallemi: Graduate School of Business, Columbia University, New York, New York 10028;
Deeksha Sinha: Operations Research Center, Massachusetts Institute of Technology, Cambridge, Massachusetts 02139

Management Science, 2020, vol. 66, issue 10, 4477-4495

Abstract: We consider the problem of A-B testing when the impact of the treatment is marred by a large number of covariates. Randomization can be highly inefficient in such settings, and thus we consider the problem of optimally allocating test subjects to either treatment with a view to maximizing the precision of our estimate of the treatment effect. Our main contribution is a tractable algorithm for this problem in the online setting, where subjects arrive, and must be assigned, sequentially, with covariates drawn from an elliptical distribution with finite second moment. We further characterize the gain in precision afforded by optimized allocations relative to randomized allocations, and show that this gain grows large as the number of covariates grows. Our dynamic optimization framework admits several generalizations that incorporate important operational constraints such as the consideration of selection bias, budgets on allocations, and endogenous stopping times. In a set of numerical experiments, we demonstrate that our method simultaneously offers better statistical efficiency and less selection bias than state-of-the-art competing biased coin designs.

Keywords: decision analysis: inference; dynamic programming, Markov, infinite state; statistics: design of experiments (search for similar items in EconPapers)
Date: 2020
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (5)

Downloads: (external link)
https://doi.org/10.1287/mnsc.2019.3424 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:inm:ormnsc:v:66:y:2020:i:10:p:4477-4495

Access Statistics for this article

More articles in Management Science from INFORMS Contact information at EDIRC.
Bibliographic data for series maintained by Chris Asher ().