On the number of trials needed to distinguish similar alternatives

Flavio Chierichetti, Ravi Kumar and Andrew Tomkins
Additional contact information
Flavio Chierichetti: a Dipartimento di Informatica, Sapienza University of Rome, Rome, Italy 00185; and;
Ravi Kumar: b Google, Mountain View, CA 94043
Andrew Tomkins: b Google, Mountain View, CA 94043

Proceedings of the National Academy of Sciences, 2022, vol. 119, issue 31, e2202116119

Abstract: Consider the process of testing two vintages of wine, two TV manufacturing processes, or two recommendation algorithms to determine whether one is preferred. Under the standard model of discrete choice, we study a wide range of A/B testing approaches to determine how many samples are required to pick a winner. We observe that, as quality (and level of investment) increases, the distinctions between alternatives become increasingly fine grained. We analyze the setting where the degree of difference between alternatives shrinks toward zero, and compute closed-form expressions for the asymptotically exact sample complexity of each test type. From this characterization, we are able to make specific recommendations for testing methodology at all target levels of error.

Keywords: discrete choice; statistical testing; sample complexity (search for similar items in EconPapers)
Date: 2022
References: Add references at CitEc
Citations:

Downloads: (external link)
http://www.pnas.org/content/119/31/e2202116119.full (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:nas:journl:v:119:y:2022:p:e2202116119

Access Statistics for this article

More articles in Proceedings of the National Academy of Sciences from Proceedings of the National Academy of Sciences
Bibliographic data for series maintained by PNAS Product Team ().