Are Multi-Armed Bandits Susceptible to Peeking?

Markus Loecher
Additional contact information
Markus Loecher: Berlin School of Economics and Law, Berlin, Germany

Zagreb International Review of Economics and Business, 2018, vol. 21, issue 1, 95-104

Abstract: A standard method to evaluate new features and changes to e.g. Web sites is A/B testing. A common pitfall in performing A/B testing is the habit of looking at a test while it's running, then stopping early. Due to the implicit multiple testing, the p-value is no longer trustworthy and usually too small. We investigate the claim that Bayesian methods, unlike frequentist tests, are immune to this "peeking" problem. We demonstrate that two regularly used measures, namely posterior probability and value remaining are severely affected by repeated testing. We further show a strong dependence on the prior probability of the parameters of interest. JEL Classification: C4, C6

Keywords: multiple comparisons; A/B testing; Bayesian decision theory (search for similar items in EconPapers)
Date: 2018
References: Add references at CitEc
Citations:

Downloads: (external link)
https://hrcak.srce.hr/index.php?show=clanak&id_clanak_jezik=295781 (text/html)
Abstract only available on-line

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:zag:zirebs:v:21:y:2018:i:1:p:95-104

Ordering information: This journal article can be ordered from
Zagreb International Review of Economics and Business, Faculty of Economics and Business, Trg J. F. Kennedy 6, 10000 Zagreb, Croatia.
http://www.efzg.hr/default.aspx?id=6045

DOI: 10.2478/zireb-2018-0004

Access Statistics for this article

Zagreb International Review of Economics and Business is currently edited by Soumitra Sharma

More articles in Zagreb International Review of Economics and Business from Faculty of Economics and Business, University of Zagreb Contact information at EDIRC.
Bibliographic data for series maintained by Jurica Šimurina ( this e-mail address is bad, please contact ).