Distributionally Robust Markov Decision Processes and Their Connection to Risk Measures

Nicole Bäuerle () and Alexander Glauner ()
Additional contact information
Nicole Bäuerle: Department of Mathematics, Karlsruhe Institute of Technology, 76128 Karlsruhe, Germany
Alexander Glauner: Department of Mathematics, Karlsruhe Institute of Technology, 76128 Karlsruhe, Germany

Mathematics of Operations Research, 2022, vol. 47, issue 3, 1757-1780

Abstract: We consider robust Markov decision processes with Borel state and action spaces, unbounded cost, and finite time horizon. Our formulation leads to a Stackelberg game against nature. Under integrability, continuity, and compactness assumptions, we derive a robust cost iteration for a fixed policy of the decision maker and a value iteration for the robust optimization problem. Moreover, we show the existence of deterministic optimal policies for both players. This is in contrast to classical zero-sum games. In case the state space is the real line, we show under some convexity assumptions that the interchange of supremum and infimum is possible with the help of Sion’s minimax theorem. Further, we consider the problem with special ambiguity sets. In particular, we are able to derive some cases where the robust optimization problem coincides with the minimization of a coherent risk measure. In the final section, we discuss two applications: a robust linear-quadratic problem and a robust problem for managing regenerative energy.

Keywords: Primary: 90C40; secondary: 90C17; 91G70; robust Markov decision process; dynamic games; minimax theorem; risk measures (search for similar items in EconPapers)
Date: 2022
References: Add references at CitEc
Citations:

Downloads: (external link)
http://dx.doi.org/10.1287/moor.2021.1187 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:inm:ormoor:v:47:y:2022:i:3:p:1757-1780

Access Statistics for this article

More articles in Mathematics of Operations Research from INFORMS Contact information at EDIRC.
Bibliographic data for series maintained by Chris Asher ().