A/B Testing with Fat Tails

Eduardo Azevedo, Alex Deng, José Luis Montiel Olea, Justin Rao and E. Glen Weyl

Journal of Political Economy, 2020, vol. 128, issue 12, 4614 - 000

Abstract: We propose a new framework for optimal experimentation, which we term the “A/B testing problem.” Our model departs from the existing literature by allowing for fat tails. Our key insight is that the optimal strategy depends on whether most gains accrue from typical innovations or from rare, unpredictable large successes. If the tails of the unobserved distribution of innovation quality are not too fat, the standard approach of using a few high-powered “big” experiments is optimal. However, if the distribution is very fat tailed, a “lean” strategy of trying more ideas, each with possibly smaller sample sizes, is preferred. Our theoretical results, along with an empirical analysis of Microsoft Bing’s EXP platform, suggest that simple changes to business practices could increase innovation productivity.

Date: 2020
References: Add references at CitEc
Citations: View citations in EconPapers (17)

Downloads: (external link)
http://dx.doi.org/10.1086/710607 (application/pdf)
http://dx.doi.org/10.1086/710607 (text/html)
Access to the online full text or PDF requires a subscription.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:ucp:jpolec:doi:10.1086/710607

Access Statistics for this article

More articles in Journal of Political Economy from University of Chicago Press
Bibliographic data for series maintained by Journals Division ().