Off-policy evaluation in doubly inhomogeneous environments

Zeyu Bian, Chengchun Shi, Zhengling Qi and Lan Wang

LSE Research Online Documents on Economics from London School of Economics and Political Science, LSE Library

Abstract: Abstract–: This work aims to study off-policy evaluation (OPE) under scenarios where two key reinforcement learning (RL) assumptions—temporal stationarity and individual homogeneity are both violated. To handle the “double inhomogeneities”, we propose a class of latent factor models for the reward and transition functions, under which we develop a general OPE framework that consists of both model-based and model-free approaches. To our knowledge, this is the first article that develops statistically sound OPE methods in offline RL with double inhomogeneities. It contributes to a deeper understanding of OPE in environments, where standard RL assumptions are not met, and provides several practical approaches in these settings. We establish the theoretical properties of the proposed value estimators and empirically show that our approach outperforms state-of-the-art methods. Finally, we illustrate our method on a dataset from the Medical Information Mart for Intensive Care. An R implementation of the proposed procedure is available athttps://github.com/ZeyuBian/2FEOPE. Supplementary materials for this article are available online, including a standardized description of the materials available for reproducing the work.

Keywords: double inhomogeneities; off-policy evaluation; reinforcement learning; two-way fixed model (search for similar items in EconPapers)
JEL-codes: C1 (search for similar items in EconPapers)
Pages: 13 pages
Date: 2024-10-11
New Economics Papers: this item is included in nep-ecm
References: Add references at CitEc
Citations:

Published in Journal of the American Statistical Association, 11, October, 2024. ISSN: 0162-1459

Downloads: (external link)
http://eprints.lse.ac.uk/124630/ Open access version. (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:ehl:lserod:124630

Access Statistics for this paper

More papers in LSE Research Online Documents on Economics from London School of Economics and Political Science, LSE Library LSE Library Portugal Street London, WC2A 2HD, U.K.. Contact information at EDIRC.
Bibliographic data for series maintained by LSERO Manager ().