Reducing Interference Bias in Online Marketplace Experiments Using Cluster Randomization: Evidence from a Pricing Meta-experiment on Airbnb

David Holtz (), Felipe Lobel (), Ruben Lobel (), Inessa Liskovich () and Sinan Aral ()
Additional contact information
David Holtz: Management of Organizations and Entrepreneurship and Innovation, Haas School of Business, University of California, Berkeley, California 94720; MIT Initiative on the Digital Economy, MIT Sloan School of Management, Cambridge, Massachusetts 02142
Felipe Lobel: Department of Economics, University of California, Berkeley, California 94720
Ruben Lobel: Airbnb, San Francisco, California 94103
Inessa Liskovich: Airbnb, San Francisco, California 94103
Sinan Aral: MIT Initiative on the Digital Economy, MIT Sloan School of Management, Cambridge, Massachusetts 02142; Sloan School of Management, Massachusetts Institute of Technology, Cambridge, Massachusetts 02142

Management Science, 2025, vol. 71, issue 1, 390-406

Abstract: Online marketplace designers frequently run randomized experiments to measure the impact of proposed product changes. However, given that marketplaces are inherently connected, total average treatment effect (TATE) estimates obtained through individual-level randomized experiments may be biased because of violations of the stable unit treatment value assumption, a phenomenon we refer to as “interference bias.” Cluster randomization (i.e., the practice of randomizing treatment assignment at the level of “clusters” of similar individuals) is an established experiment design technique for countering interference bias in social networks, but it is unclear ex ante if it will be effective in marketplace settings. In this paper, we use a meta-experiment or “experiment over experiments” conducted on Airbnb to both provide empirical evidence of interference bias in online marketplace settings and assess the viability of cluster randomization as a tool for reducing interference bias in marketplace TATE estimates. Results from our meta-experiment indicate that at least 20% of the TATE estimate produced by an individual-level randomized evaluation of the platform fee increase we study is attributable to interference bias and eliminated through the use of cluster randomization. We also find suggestive, nonstatistically significant evidence that interference bias in seller-side experiments is more severe in demand-constrained geographies and that the efficacy of cluster randomization at reducing interference bias increases with cluster quality.

Keywords: design of experiments; electronic markets and auctions; interference; cluster randomization; Airbnb (search for similar items in EconPapers)
Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
http://dx.doi.org/10.1287/mnsc.2020.01157 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:inm:ormnsc:v:71:y:2025:i:1:p:390-406

Access Statistics for this article

More articles in Management Science from INFORMS Contact information at EDIRC.
Bibliographic data for series maintained by Chris Asher ().