Easy Affine Markov Decision Processes

Jie Ning () and Matthew J. Sobel ()
Additional contact information
Jie Ning: Department of Operations, Weatherhead School of Management, Case Western Reserve University, Cleveland, Ohio 44106
Matthew J. Sobel: Department of Operations, Weatherhead School of Management, Case Western Reserve University, Cleveland, Ohio 44106

Operations Research, 2019, vol. 67, issue 6, 1719-1737

Abstract: This paper characterizes the class of decomposable affine Markov decision processes (MDPs), which have continuous multidimensional endogenous states and actions, and Markov-modulated exogenous states. This class of MDPs has affine dynamics and single-period rewards, sets of feasible actions that decompose into bounded polytopes, and endogenous state variables that are nonnegative or nonpositive. It is shown that decomposable affine MDPs with discounted criteria have an affine value function and an affine optimal policy. The affine coefficients of the value function and optimal policy are determined by the solution of auxiliary equations, which themselves resemble the dynamic program of a finite MDP. This result exorcizes the curse of dimensionality for decomposable affine MDPs, which otherwise could be solved only approximately with discrete approximations. Additionally, the paper characterizes partially decomposable affine MDPs that meet only some of the assumptions for decomposable affine MDPs. It shows that they are composites of two smaller MDPs, one of which is a decomposable affine MDP. The applicability of the classes of MDPs in the paper is exemplified with models of fishery management, dynamic capacity portfolio management, and commodity procurement.

Keywords: Markov decision processes; curse of dimensionality; decomposable affine MDP; tractable, dynamic program (search for similar items in EconPapers)
Date: 2019
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (2)

Downloads: (external link)
https://doi.org/10.1287/opre.2018.1836 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:inm:oropre:v:67:y:2019:i:6:p:1719-1737

Access Statistics for this article

More articles in Operations Research from INFORMS Contact information at EDIRC.
Bibliographic data for series maintained by Chris Asher ().