 The A/B testing problem with Gaussian priorsEduardo M. Azevedo, David Mao, José Luis Montiel Olea and Amilcar Velez Journal of Economic Theory, 2023, vol. 210, issue C Abstract: A risk-neutral firm can perform a randomized experiment (A/B test) to learn about the effects of implementing an idea of unknown quality. The firm's goal is to decide the experiment's sample size and whether or not the idea should be implemented after observing the experiment's outcome. We show that when the distribution for idea quality is Gaussian and there are linear costs of experimentation, there are exact formulae for the firm's optimal implementation decisions, the value of obtaining more data, and optimal experiment sizes. Our formulae—which assume that companies use randomized experiments to help them maximize expected profits—provide a simple alternative to i) the standard rules-of-thumb of power calculations for determining the sample size of an experiment, and also to ii) ad hoc thresholds based on statistical significance to interpret the outcome of an experiment. Keywords: Statistical Decision Theory; Optimal learning; Experiment design; A/B testing (search for similar items in EconPapers) Downloads: (external link) Related works: Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text Persistent link: https://EconPapers.repec.org/RePEc:eee:jetheo:v:210:y:2023:i:c:s002205312300042x DOI: 10.1016/j.jet.2023.105646 Access Statistics for this article Journal of Economic Theory is currently edited by A. Lizzeri and K. Shell More articles in Journal of Economic Theory from Elsevier |
Is your work missing from RePEc? Here is how to contribute.
Questions or problems? Check the EconPapers FAQ or send mail to .
EconPapers is hosted by the
School of Business at Örebro University.