Off-policy confidence interval estimation with confounded Markov decision process

Chengchun Shi, Jin Zhu, Ye Shen, Shikai Luo, Hongtu Zhu and Rui Song

LSE Research Online Documents on Economics from London School of Economics and Political Science, LSE Library

Abstract: This article is concerned with constructing a confidence interval for a target policy’s value offline based on a pre-collected observational data in infinite horizon settings. Most of the existing works assume no unmeasured variables exist that confound the observed actions. This assumption, however, is likely to be violated in real applications such as healthcare and technological industries. In this article, we show that with some auxiliary variables that mediate the effect of actions on the system dynamics, the target policy’s value is identifiable in a confounded Markov decision process. Based on this result, we develop an efficient off-policy value estimator that is robust to potential model misspecification and provide rigorous uncertainty quantification. Our method is justified by theoretical results, simulated and real datasets obtained from ridesharing companies. A Python implementation of the proposed procedure is available at https://github.com/Mamba413/cope.

Keywords: reinforcement learning; off-policy evaluation; statistical inference; unmeasured confounders; infinite horizons; ridesourcing platforms (search for similar items in EconPapers)
JEL-codes: C1 (search for similar items in EconPapers)
Pages: 12 pages
Date: 2022-10-05
New Economics Papers: this item is included in nep-ecm
References: Add references at CitEc
Citations: View citations in EconPapers (1)

Published in Journal of the American Statistical Association, 5, October, 2022. ISSN: 0162-1459

Downloads: (external link)
http://eprints.lse.ac.uk/115774/ Open access version. (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:ehl:lserod:115774

Access Statistics for this paper

More papers in LSE Research Online Documents on Economics from London School of Economics and Political Science, LSE Library LSE Library Portugal Street London, WC2A 2HD, U.K.. Contact information at EDIRC.
Bibliographic data for series maintained by LSERO Manager ().