Experimental Design in Two-Sided Platforms: An Analysis of Bias

Ramesh Johari (), Hannah Li (), Inessa Liskovich () and Gabriel Y. Weintraub ()
Additional contact information
Ramesh Johari: Management Science and Engineering, Stanford University, Stanford, California 94305
Hannah Li: Management Science and Engineering, Stanford University, Stanford, California 94305
Inessa Liskovich: Airbnb Inc., San Francisco, California 94117
Gabriel Y. Weintraub: Graduate School of Business, Stanford University, Stanford, California 94305

Management Science, 2022, vol. 68, issue 10, 7069-7089

Abstract: We develop an analytical framework to study experimental design in two-sided marketplaces. Many of these experiments exhibit interference , where an intervention applied to one market participant influences the behavior of another participant. This interference leads to biased estimates of the treatment effect of the intervention. We develop a stochastic market model and associated mean field limit to capture dynamics in such experiments and use our model to investigate how the performance of different designs and estimators is affected by marketplace interference effects. Platforms typically use two common experimental designs: demand-side “customer” randomization ( CR ) and supply-side “listing” randomization ( LR ), along with their associated estimators. We show that good experimental design depends on market balance; in highly demand-constrained markets, CR is unbiased, whereas LR is biased; conversely, in highly supply-constrained markets, LR is unbiased, whereas CR is biased. We also introduce and study a novel experimental design based on two-sided randomization ( TSR ) where both customers and listings are randomized to treatment and control. We show that appropriate choices of TSR designs can be unbiased in both extremes of market balance while yielding relatively low bias in intermediate regimes of market balance.

Keywords: statistics: design of experiments; probability: stochastic model applications; two-sided markets (search for similar items in EconPapers)
Date: 2022
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (19)

Downloads: (external link)
http://dx.doi.org/10.1287/mnsc.2021.4247 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:inm:ormnsc:v:68:y:2022:i:10:p:7069-7089

Access Statistics for this article

More articles in Management Science from INFORMS Contact information at EDIRC.
Bibliographic data for series maintained by Chris Asher ().