An online sequential test for qualitative treatment effects

Chengchun Shi, Shikai Luo, Hongtu Zhu and Rui Song

LSE Research Online Documents on Economics from London School of Economics and Political Science, LSE Library

Abstract: Tech companies (e.g., Google or Facebook) often use randomized online experiments and/or A/B testing primarily based on the average treatment effects to compare their new product with an old one. However, it is also critically important to detect qualitative treatment effects such that the new one may significantly outperform the existing one only under some specific circumstances. The aim of this paper is to develop a powerful testing procedure to efficiently detect such qualitative treatment effects. We propose a scalable online updating algorithm to implement our test procedure. It has three novelties including adaptive randomization, sequential monitoring, and online updating with guaranteed type-I error control. We also thoroughly examine the theoretical properties of our testing procedure including the limiting distribution of test statistics and the justification of an efficient bootstrap method. Extensive empirical studies are conducted to examine the finite sample performance of our test procedure.

JEL-codes: C1 (search for similar items in EconPapers)
Date: 2021-12-01
New Economics Papers: this item is included in nep-ecm, nep-exp and nep-pay
References: View references in EconPapers View complete reference list from CitEc
Citations:

Published in Journal of Machine Learning Research, 1, December, 2021, 22. ISSN: 1532-4435

Downloads: (external link)
http://eprints.lse.ac.uk/112521/ Open access version. (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:ehl:lserod:112521

Access Statistics for this paper

More papers in LSE Research Online Documents on Economics from London School of Economics and Political Science, LSE Library LSE Library Portugal Street London, WC2A 2HD, U.K.. Contact information at EDIRC.
Bibliographic data for series maintained by LSERO Manager ().