Always Valid Inference: Continuous Monitoring of A/B Tests

Ramesh Johari (), Pete Koomen (), Leonid Pekelis () and David Walsh ()
Additional contact information
Ramesh Johari: Department of Management Science and Engineering, Stanford University, Stanford, California 94305
Pete Koomen: Optimizely, Inc., San Francisco, California 94105
Leonid Pekelis: CloudTrucks, Inc., San Francisco, California 94103
David Walsh: Unlearn.AI, San Francisco, California 94105

Operations Research, 2022, vol. 70, issue 3, 1806-1821

Abstract: A/B tests are typically analyzed via frequentist p -values and confidence intervals, but these inferences are wholly unreliable if users endogenously choose samples sizes by continuously monitoring their tests. We define always valid p -values and confidence intervals that let users try to take advantage of data as fast as it becomes available, providing valid statistical inference whenever they make their decision. Always valid inference can be interpreted as a natural interface for a sequential hypothesis test, which empowers users to implement a modified test tailored to them. In particular, we show in an appropriate sense that the measures we develop trade off sample size and power efficiently, despite a lack of prior knowledge of the user’s relative preference between these two goals. We also use always valid p -values to obtain multiple hypothesis testing control in the sequential context. Our methodology has been implemented in a large-scale commercial A/B testing platform to analyze hundreds of thousands of experiments to date.

Keywords: Stochastic Models; A/B testing; p -values; confidence intervals; sequential hypothesis testing; multiple hypothesis testing (search for similar items in EconPapers)
Date: 2022
References: Add references at CitEc
Citations: View citations in EconPapers (1)

Downloads: (external link)
http://dx.doi.org/10.1287/opre.2021.2135 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:inm:oropre:v:70:y:2022:i:3:p:1806-1821

Access Statistics for this article

More articles in Operations Research from INFORMS Contact information at EDIRC.
Bibliographic data for series maintained by Chris Asher ().