Testing stationarity and change point detection in reinforcement learning

Mengbing Li, Chengchun Shi, Zhenke Wu and Piotr Fryzlewicz

LSE Research Online Documents on Economics from London School of Economics and Political Science, LSE Library

Abstract: We consider reinforcement learning (RL) in possibly nonstationary environments. Many existing RL algorithms in the literature rely on the stationarity assumption that requires the state transition and reward functions to be constant over time. However, this assumption is restrictive in practice and is likely to be violated in a number of applications, including traffic signal control, robotics and mobile health. In this paper, we develop a model-free test to assess the stationarity of the optimal Q-function based on pre-collected historical data, without additional online data collection. Based on the proposed test, we further develop a change point detection method that can be naturally coupled with existing state-of-the-art RL methods designed in stationary environments for online policy optimization in nonstationary environments. The usefulness of our method is illustrated by theoretical results, simulation studies, and a real data example from the 2018 Intern Health Study. A Python implementation of the proposed procedure is publicly available at https://github.com/limengbinggz/CUSUM-RL.

Keywords: change point detection; hypothesis testing; nonstationarity; reinforcement learning (search for similar items in EconPapers)
JEL-codes: C1 (search for similar items in EconPapers)
Pages: 27 pages
Date: 2025-06-30
New Economics Papers: this item is included in nep-ets
References: Add references at CitEc
Citations:

Published in Annals of Statistics, 30, June, 2025, 53(3), pp. 1230 - 1256. ISSN: 0090-5364

Downloads: (external link)
http://eprints.lse.ac.uk/127507/ Open access version. (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:ehl:lserod:127507

Access Statistics for this paper

More papers in LSE Research Online Documents on Economics from London School of Economics and Political Science, LSE Library LSE Library Portugal Street London, WC2A 2HD, U.K.. Contact information at EDIRC.
Bibliographic data for series maintained by LSERO Manager ().