Markov Decision Processes with Observation Costs: Framework and Computation with a Penalty Scheme

Christoph Reisinger () and Jonathan Tam ()
Additional contact information
Christoph Reisinger: Mathematical Institute, University of Oxford, Oxford OX2 6GG, United Kingdom
Jonathan Tam: Mathematical Institute, University of Oxford, Oxford OX2 6GG, United Kingdom

Mathematics of Operations Research, 2025, vol. 50, issue 2, 1305-1332

Abstract: We consider Markov decision processes where the state of the chain is only given at chosen observation times and of a cost. Optimal strategies involve the optimization of observation times as well as the subsequent action values. We consider the finite horizon and discounted infinite horizon problems as well as an extension with parameter uncertainty. By including the time elapsed from observations as part of the augmented Markov system, the value function satisfies a system of quasivariational inequalities (QVIs). Such a class of QVIs can be seen as an extension to the interconnected obstacle problem. We prove a comparison principle for this class of QVIs, which implies the uniqueness of solutions to our proposed problem. Penalty methods are then utilized to obtain arbitrarily accurate solutions. Finally, we perform numerical experiments on three applications that illustrate our framework.

Keywords: Primary: 49N30; 93C41; 49L20; secondary: 65K15; stochastic control; partial information; observation cost; quasivariational inequality; penalty scheme (search for similar items in EconPapers)
Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
http://dx.doi.org/10.1287/moor.2023.0172 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:inm:ormoor:v:50:y:2025:i:2:p:1305-1332

Access Statistics for this article

More articles in Mathematics of Operations Research from INFORMS Contact information at EDIRC.
Bibliographic data for series maintained by Chris Asher ().