Seesaw Experimentation: A/B Tests with Spillovers

Jin Li, Ye Luo and Xiaowei Zhang

Papers from arXiv.org

Abstract: This paper examines how spillover effects in A/B testing can impede organizational progress and develops strategies for mitigating these challenges. We identify a phenomenon termed ``seesaw experimentation'', where a firm's overall performance paradoxically deteriorates despite achieving continuous improvements in measured A/B testing metrics. Seesaw experimentation arises when successful innovations in primary metrics generate negative externalities in secondary, unmeasured dimensions. To address this problem, we propose implementing a positive hurdle rate for A/B test approval. We derive the optimal hurdle rate, offering a simple solution that preserves decentralized experimentation while mitigating negative spillovers.

Date: 2024-11, Revised 2025-01
References: Add references at CitEc
Citations:

Downloads: (external link)
http://arxiv.org/pdf/2411.02085 Latest version (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:arx:papers:2411.02085

Access Statistics for this paper

More papers in Papers from arXiv.org
Bibliographic data for series maintained by arXiv administrators ().