Solution for Infinite Horizon Double-Factored Markov Decision Processes with Application

Chengjun Hou ()
Additional contact information
Chengjun Hou: Software Research and Data Science, Amazon Robotics

SN Operations Research Forum, 2025, vol. 6, issue 4, 1-25

Abstract: Abstract The double-factored Markov decision process (DFMDP) is a new framework for addressing the Markov decision processes with uncertainty of parameter scenarios. The two factors in this framework refer to the physical state and the scenario belief, which describes the probability distribution of scenarios, and they compose the state-belief pair for the framework. This study focuses on infinite horizon DFMDPs and their application. The optimality equations for infinite horizon DFMDPs are formulated and they can be represented by a value function mapping, which is a contraction under the supremum norm. It is demonstrated that for a fixed state, the optimal value functions for finite horizon DFMDPs are piecewise linear and convex in a scenario belief space. This property is used to develop an algorithm named as the double-factored linear support for an approximate solution to infinite horizon DFMDPs. The principle and framework of the algorithm are described in detail. Three computational instances are presented to illustrate the performance of the algorithm and the applications of infinite horizon DFMDPs.

Keywords: Uncertainty modelling; Double-factored Markov decision process; Multi-scenario Markov decision process; Stochastic inventory control; Path programming (search for similar items in EconPapers)
Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
http://link.springer.com/10.1007/s43069-025-00468-3 Abstract (text/html)
Access to the full text of the articles in this series is restricted.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:snopef:v:6:y:2025:i:4:d:10.1007_s43069-025-00468-3

Ordering information: This journal article can be ordered from
https://www.springer.com/journal/43069

DOI: 10.1007/s43069-025-00468-3

Access Statistics for this article

SN Operations Research Forum is currently edited by Marco Lübbecke

More articles in SN Operations Research Forum from Springer
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().