A/B testing: The importance of significance and test duration

Sabine Langmann

Applied Marketing Analytics: The Peer-Reviewed Journal, 2018, vol. 4, issue 2, 149-156

Abstract: This paper demonstrates that various psychological patterns and a lack of knowledge of basic statistical principles can lead to erroneous interpretations of A/B tests. It explains the statistics behind testing with emphasis on the pitfalls to be avoided. Readers will gain an understanding of the importance of significance in A/B testing, but also learn that a significant test is no guarantee of a universally valid outcome. For reliable results, experimenters must also select a representative time frame. Finally, the paper discusses methods to create a healthy testing culture.

Keywords: A/B testing; conversion rate optimisation; applied statistics; significance; confidence level; effect measure (search for similar items in EconPapers)
JEL-codes: M3 (search for similar items in EconPapers)
Date: 2018
References: Add references at CitEc
Citations:

Downloads: (external link)
https://hstalks.com/article/3911/download/ (application/pdf)
https://hstalks.com/article/3911/ (text/html)
Requires a paid subscription for full access.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:aza:ama000:y:2018:v:4:i:2:p:149-156

Access Statistics for this article

More articles in Applied Marketing Analytics: The Peer-Reviewed Journal from Henry Stewart Publications
Bibliographic data for series maintained by Henry Stewart Talks ().