Efficient experimentation: A review of four methods to reduce the costs of A/B testing

Malte Bleeker and Philipp Kaufmann
Additional contact information
Malte Bleeker: University of St. Gallen, Switzerland
Philipp Kaufmann: University of St. Gallen, Switzerland

Applied Marketing Analytics: The Peer-Reviewed Journal, 2025, vol. 11, issue 1, 61-71

Abstract: Thanks to its ability to detect causality, A/B testing has become the standard method for identifying superior variants of advertisements or product features. At the same time, however, it has also become an obstacle to implementation, due to its reliance on large sample sizes and test durations to identify significant effects. Approaches such as stratification, CUPED (Controlled-Experiment using Pre-Experiment Data), p-value adjustment and multi-armed bandits have been developed to mitigate these issues, but managers still struggle to understand them and to select the best methods for established experimentation platforms. This paper addresses this problem by providing a comprehensive overview of selection criteria for these methods and a decision framework to assist managers in selecting the most appropriate approach for their specific situation to reduce time and opportunity costs. While stratification proves advantageous for reducing sample and time requirements in settings affecting primarily new users, CUPED benefits most in time and sample size constrained settings, affecting existing users. If time and sample size constraints persist despite the regular utilisation of these methods, p-value threshold adjustments can be contemplated at the organisational level. Multiarmed bandits have proven most suitable when immediate optimisation is imperative and relevance would have already passed, before a superior version could have been reliably identified using other methods. In conclusion, it is important to consider the organisational requirements as well as the specific task at hand when choosing an experimental method. This article is also included in The Business & Management Collection which can be accessed at https://hstalks.com/business/.

Keywords: experimentation; A/B testing; data-driven marketing; marketing performance measurement; marketing efficiency; p-value (search for similar items in EconPapers)
JEL-codes: M3 (search for similar items in EconPapers)
Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
https://hstalks.com/article/9570/download/ (application/pdf)
https://hstalks.com/article/9570/ (text/html)
Requires a paid subscription for full access.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:aza:ama000:y:2025:v:11:i:1:p:61-71

Access Statistics for this article

More articles in Applied Marketing Analytics: The Peer-Reviewed Journal from Henry Stewart Publications
Bibliographic data for series maintained by Henry Stewart Talks ().