Assessing the statistical significance of repeated A/B tests with meta-analysis

David Harman
Additional contact information
David Harman: Assistant Professor, University of St. Thomas, USA

Applied Marketing Analytics: The Peer-Reviewed Journal, 2022, vol. 7, issue 4, 374-385

Abstract: This paper presents the steps to conduct a meta-analysis of a set of repeated A/B tests. Repeated A/B testing is common in professional marketing practice. Each test, however, can only be individually evaluated for statistical significance when using basic statistical analysis. A meta-analysis of the same A/B tests provides three useful analytics outputs: an overall treatment effect across all tests, a confidence interval for that effect to assess statistical significance, and a measure of heterogeneity between tests to assess the context variation between campaigns. Meta-analysis is a useful addition to marketing analytics practice.

Keywords: meta-analysis; A/B testing; significance testing; marketing campaign analytics (search for similar items in EconPapers)
JEL-codes: M3 (search for similar items in EconPapers)
Date: 2022
References: Add references at CitEc
Citations:

Downloads: (external link)
https://hstalks.com/article/6954/download/ (application/pdf)
https://hstalks.com/article/6954/ (text/html)
Requires a paid subscription for full access.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:aza:ama000:y:2022:v:7:i:4:p:374-385

Access Statistics for this article

More articles in Applied Marketing Analytics: The Peer-Reviewed Journal from Henry Stewart Publications
Bibliographic data for series maintained by Henry Stewart Talks ().