Nonstationary A/B Tests: Optimal Variance Reduction, Bias Correction, and Valid Inference

Yuhang Wu (), Zeyu Zheng (), Guangyu Zhang (), Zuohua Zhang () and Chu Wang ()
Additional contact information
Yuhang Wu: Department of Industrial Engineering and Operations Research, University of California, Berkeley, California 94720
Zeyu Zheng: Department of Industrial Engineering and Operations Research, University of California, Berkeley, California 94720
Guangyu Zhang: Amazon.com Inc, Seattle, Washington 98109
Zuohua Zhang: Amazon.com Inc, Seattle, Washington 98109
Chu Wang: Amazon.com Inc, Seattle, Washington 98109

Management Science, 2025, vol. 71, issue 6, 4707-4727

Abstract: We develop an analytical framework to appropriately model and adequately analyze A/B tests in presence of nonparametric nonstationarities in the targeted business metrics. A/B tests, also known as online randomized controlled experiments, have been used at scale by data-driven enterprises to guide decisions and test innovative ideas to improve core business metrics. Meanwhile, nonstationarities, such as the time-of-day effect and the day-of-week effect, can often arise nonparametrically in key business metrics involving purchases, revenue, conversions, customer experiences, and so on. First, we develop a generic nonparametric stochastic model to capture nonstationarities in A/B test experiments, where each sample represents a visit or action associated with a time label. We build a practically relevant limiting regime to facilitate analyzing large-sample estimator performances under nonparametric nonstationarities. Second, we show that ignoring or inadequately addressing nonstationarities can cause standard A/B test estimators to have suboptimal variance and nonvanishing bias, therefore leading to loss of statistical efficiency and accuracy. We provide a new estimator that views time as a continuous strata and performs poststratification with a data-dependent number of stratification levels. Without making parametric assumptions, we prove a central limit theorem for the proposed estimator and show that the estimator attains the best achievable asymptotic variance and is asymptotically unbiased. Third, we propose a time-grouped randomization that is designed to balance treatment and control assignments at granular time scales. We show that when the time-grouped randomization is integrated to standard experimental designs to generate experiment data, simple A/B test estimators can achieve asymptotically optimal variance. A brief account of numerical experiments are conducted to illustrate the analysis.

Keywords: A/B tests; nonstationarity; central limit theorem; optimal asymptotic variance; bias; inference (search for similar items in EconPapers)
Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
http://dx.doi.org/10.1287/mnsc.2022.01205 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:inm:ormnsc:v:71:y:2025:i:6:p:4707-4727

Access Statistics for this article

More articles in Management Science from INFORMS Contact information at EDIRC.
Bibliographic data for series maintained by Chris Asher ().