Uncertainty Quantification and Exploration for Reinforcement Learning

Yi Zhu (), Jing Dong () and Henry Lam ()
Additional contact information
Yi Zhu: Department of Industrial Engineering and Management Sciences, Northwestern University, Evanston, Illinois 60208
Jing Dong: Division, Risk and Operations Division, Columbia Business School, New York, New York 10027
Henry Lam: Department of Industrial Engineering and Operations Research, Columbia University, New York, New York 10027

Operations Research, 2024, vol. 72, issue 4, 1689-1709

Abstract: We investigate statistical uncertainty quantification for reinforcement learning (RL) and its implications in exploration policy. Despite ever-growing literature on RL applications, fundamental questions about inference and error quantification, such as large-sample behaviors, appear to remain quite open. In this paper, we fill in the literature gap by studying the central limit theorem behaviors of estimated Q-values and value functions under various RL settings. In particular, we explicitly identify closed-form expressions of the asymptotic variances, which allow us to efficiently construct asymptotically valid confidence regions for key RL quantities. Furthermore, we utilize these asymptotic expressions to design an effective exploration strategy, which we call Q-value-based Optimal Computing Budget Allocation (Q-OCBA). The policy relies on maximizing the relative discrepancies among the Q-value estimates. Numerical experiments show superior performances of our exploration strategy than other benchmark policies.

Keywords: Simulation; reinforcement learning; statistical inference; exploration policy; OCBA (search for similar items in EconPapers)
Date: 2024
References: Add references at CitEc
Citations:

Downloads: (external link)
http://dx.doi.org/10.1287/opre.2023.2436 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:inm:oropre:v:72:y:2024:i:4:p:1689-1709

Access Statistics for this article

More articles in Operations Research from INFORMS Contact information at EDIRC.
Bibliographic data for series maintained by Chris Asher ().