Asymptotic Optimality of Semi-Open-Loop Policies in Markov Decision Processes with Large Lead Times

Xingyu Bai (), Xin Chen (), Menglong Li () and Alexander Stolyar ()
Additional contact information
Xingyu Bai: Industrial and Enterprise Systems Engineering, University of Illinois at Urbana-Champaign, Urbana, Illinois 61801
Xin Chen: H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, Georgia 30332
Menglong Li: Department of Management Sciences, City University of Hong Kong, Hong Kong
Alexander Stolyar: Industrial and Enterprise Systems Engineering & Coordinated Science Laboratory, University of Illinois at Urbana-Champaign, Urbana, Illinois 61801

Operations Research, 2023, vol. 71, issue 6, 2061-2077

Abstract: We consider a generic Markov decision process (MDP) with two controls: one control taking effect immediately and the other control whose effect is delayed by a positive lead time. As the lead time grows, one naturally expects that the effect of the delayed action only weakly depends on the current state, and decoupling the delayed action from the current state could provide good controls. The purpose of this paper is to substantiate this decoupling intuition by establishing asymptotic optimality of semi-open-loop policies, which specify open-loop controls for the delayed action and closed-loop controls for the immediate action. For MDPs defined on general spaces with uniformly bounded cost functions and a fast mixing property, we construct a periodic semi-open-loop policy for each lead time value and show that these policies are asymptotically optimal as the lead time goes to infinity. For MDPs defined on Euclidean spaces with linear dynamics and convex structures (convex cost functions and constraint sets), we impose another set of conditions under which semi-open-loop policies (actually, constant delayed-control policies) are asymptotically optimal. Moreover, we verify that these conditions hold for a broad class of inventory models, in which there are multiple controls with nonidentical lead times.

Keywords: Operations and Supply Chains; open-loop policy; asymptotic analysis; Markov decision process; lead time; inventory (search for similar items in EconPapers)
Date: 2023
References: Add references at CitEc
Citations:

Downloads: (external link)
http://dx.doi.org/10.1287/opre.2021.0088 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:inm:oropre:v:71:y:2023:i:6:p:2061-2077

Access Statistics for this article

More articles in Operations Research from INFORMS Contact information at EDIRC.
Bibliographic data for series maintained by Chris Asher ().