Evaluating dynamic conditional quantile treatment effects with applications in ridesharing

Ting Li, Chengchun Shi, Zhaohua Lu, Yi Li and Hongtu Zhu

LSE Research Online Documents on Economics from London School of Economics and Political Science, LSE Library

Abstract: Many modern tech companies, such as Google, Uber, and Didi, use online experiments (also known as A/B testing) to evaluate new policies against existing ones. While most studies concentrate on average treatment effects, situations with skewed and heavy-tailed outcome distributions may benefit from alternative criteria, such as quantiles. However, assessing dynamic quantile treatment effects (QTE) remains a challenge, particularly when dealing with data from ride-sourcing platforms that involve sequential decision-making across time and space. In this article, we establish a formal framework to calculate QTE conditional on characteristics independent of the treatment. Under specific model assumptions, we demonstrate that the dynamic conditional QTE (CQTE) equals the sum of individual CQTEs across time, even though the conditional quantile of cumulative rewards may not necessarily equate to the sum of conditional quantiles of individual rewards. This crucial insight significantly streamlines the estimation and inference processes for our target causal estimand. We then introduce two varying coefficient decision process (VCDP) models and devise an innovative method to test the dynamic CQTE. Moreover, we expand our approach to accommodate data from spatiotemporal dependent experiments and examine both conditional quantile direct and indirect effects. To showcase the practical utility of our method, we apply it to three real-world datasets from a ride-sourcing platform. Theoretical findings and comprehensive simulation studies further substantiate our proposal. Supplementary materials for this article are available online Code implementing the proposed method is also available at: https://github.com/BIG-S2/CQSTVCM.

Keywords: varying coefficient models; A/B testing; policy evaluation; quantile treatment effect; ridesourcing platform; spatialtemporal experiments; Li’s research is partially supported by the Nation12101388; CCF-DiDi GAIA Collaborative Research Funds for Young Scholars and Program for Innovative Research Team of Shanghai University of Finance and Economics; EP/W014971/1 (search for similar items in EconPapers)
JEL-codes: C1 (search for similar items in EconPapers)
Pages: 15 pages
Date: 2024-08-31
New Economics Papers: this item is included in nep-ecm
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (2)

Published in Journal of the American Statistical Association, 31, August, 2024, 119(547), pp. 1736 - 1750. ISSN: 0162-1459

Downloads: (external link)
http://eprints.lse.ac.uk/122488/ Open access version. (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:ehl:lserod:122488

Access Statistics for this paper

More papers in LSE Research Online Documents on Economics from London School of Economics and Political Science, LSE Library LSE Library Portugal Street London, WC2A 2HD, U.K.. Contact information at EDIRC.
Bibliographic data for series maintained by LSERO Manager ().