A Bias Correction Approach for Interference in Ranking Experiments

Ali Goli (), Anja Lambrecht () and Hema Yoganarasimhan ()
Additional contact information
Ali Goli: University of Washington, Seattle, Washington 98195
Anja Lambrecht: London Business School, London NW1 4SA, United Kingdom
Hema Yoganarasimhan: University of Washington, Seattle, Washington 98195

Marketing Science, 2024, vol. 43, issue 3, 590-614

Abstract: Online marketplaces use ranking algorithms to determine the rank-ordering of items sold on their websites. The standard practice is to determine the optimal algorithm using A/B tests. We present a theoretical framework to characterize the total average treatment effect (TATE) of a ranking algorithm in an A/B test and show that naive TATE estimates can be biased because of interference. We propose a bias-correction approach that can recover the TATE of a ranking algorithm based on past A/B tests even if those tests suffer from a combination of interference issues. Our solution leverages data across multiple experiments and identifies observations in partial equilibrium in each experiment, that is, items close to their positions under the true counterfactual equilibrium of interest. We apply our framework to data from a travel website and present comprehensive evidence for interference bias in this setting. Next, we use our solution concept to build a customized deep learning model to predict the true TATE of the main algorithm of interest in our data. Counterfactual estimates from our model show that naive TATE estimates of click and booking rates can be biased by as much as 15% and 29%, respectively.

Keywords: experiments; A/B tests; treatment effects; digital platforms; interference; machine learning; bias correction (search for similar items in EconPapers)
Date: 2024
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (1)

Downloads: (external link)
http://dx.doi.org/10.1287/mksc.2022.0046 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:inm:ormksc:v:43:y:2024:i:3:p:590-614

Access Statistics for this article

More articles in Marketing Science from INFORMS Contact information at EDIRC.
Bibliographic data for series maintained by Chris Asher ().