Statistically efficient advantage learning for offline reinforcement learning in infinite horizons

Chengchun Shi, Shikai Luo, Yuan Le, Hongtu Zhu and Rui Song

LSE Research Online Documents on Economics from London School of Economics and Political Science, LSE Library

Abstract: We consider reinforcement learning (RL) methods in offline domains without additional online data collection, such as mobile health applications. Most of existing policy optimization algorithms in the computer science literature are developed in online settings where data are easy to collect or simulate. Their generalizations to mobile health applications with a pre-collected offline dataset remain unknown. The aim of this paper is to develop a novel advantage learning framework in order to efficiently use pre-collected data for policy optimization. The proposed method takes an optimal Q-estimator computed by any existing state-of-the-art RL algorithms as input, and outputs a new policy whose value is guaranteed to converge at a faster rate than the policy derived based on the initial Q-estimator. Extensive numerical experiments are conducted to back up our theoretical findings. A Python implementation of our proposed method is available at https://github.com/leyuanheart/SEAL

Keywords: reinforcement learning; advantage learning; infinite horizons; rate of convergence; mobile health applications (search for similar items in EconPapers)
JEL-codes: C1 (search for similar items in EconPapers)
Date: 2022-09-27
New Economics Papers: this item is included in nep-cmp and nep-ecm
References: View references in EconPapers View complete reference list from CitEc
Citations:

Published in Journal of the American Statistical Association, 27, September, 2022. ISSN: 0162-1459

Downloads: (external link)
http://eprints.lse.ac.uk/115598/ Open access version. (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:ehl:lserod:115598

Access Statistics for this paper

More papers in LSE Research Online Documents on Economics from London School of Economics and Political Science, LSE Library LSE Library Portugal Street London, WC2A 2HD, U.K.. Contact information at EDIRC.
Bibliographic data for series maintained by LSERO Manager ().