Statistical inference of the value function for reinforcement learning in infinite-horizon settings

Chengchun Shi, Shengxing Zhang, Wenbin Lu and Rui Song

LSE Research Online Documents on Economics from London School of Economics and Political Science, LSE Library

Abstract: Reinforcement learning is a general technique that allows an agent to learn an optimal policy and interact with an environment in sequential decision-making problems. The goodness of a policy is measured by its value function starting from some initial state. The focus of this paper was to construct confidence intervals (CIs) for a policy’s value in infinite horizon settings where the number of decision points diverges to infinity. We propose to model the action-value state function (Q-function) associated with a policy based on series/sieve method to derive its confidence interval. When the target policy depends on the observed data as well, we propose a SequentiAl Value Evaluation (SAVE) method to recursively update the estimated policy and its value estimator. As long as either the number of trajectories or the number of decision points diverges to infinity, we show that the proposed CI achieves nominal coverage even in cases where the optimal policy is not unique. Simulation studies are conducted to back up our theoretical findings. We apply the proposed method to a dataset from mobile health studies and find that reinforcement learning algorithms could help improve patient’s health status. A Python implementation of the proposed procedure is available at https://github.com/shengzhang37/SAVE.

Keywords: bidirectional asymptotics; confidence interval; infinite horizons; reinforcement learning; value function; New Research Support Fund; DMS-1555244; DMS-2113637 (search for similar items in EconPapers)
JEL-codes: C1 (search for similar items in EconPapers)
Pages: 29 pages
Date: 2022-07-01
New Economics Papers: this item is included in nep-dge and nep-ecm
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (3)

Published in Journal of the Royal Statistical Society. Series B: Statistical Methodology, 1, July, 2022, 84(3), pp. 765 - 793. ISSN: 1369-7412

Downloads: (external link)
http://eprints.lse.ac.uk/110882/ Open access version. (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:ehl:lserod:110882

Access Statistics for this paper

More papers in LSE Research Online Documents on Economics from London School of Economics and Political Science, LSE Library LSE Library Portugal Street London, WC2A 2HD, U.K.. Contact information at EDIRC.
Bibliographic data for series maintained by LSERO Manager ().