An Approximation Approach for the Deviation Matrix of Continuous-Time Markov Processes with Application to Markov Decision Theory

Nicole Leder (), Bernd Heidergott () and Arie Hordijk ()
Additional contact information
Nicole Leder: Department of Mathematics, University of Hamburg, Hamburg 20146, Germany
Bernd Heidergott: Department of Econometrics and Operations Research, and Tinbergen Institute, Vrije Universiteit Amsterdam, Amsterdam 1081 HV, The Netherlands
Arie Hordijk: Mathematical Institute, Leiden University, Leiden 2300 RA, The Netherlands

Operations Research, 2010, vol. 58, issue 4-part-1, 918-932

Abstract: We present an update formula that allows the expression of the deviation matrix of a continuous-time Markov process with denumerable state space having generator matrix Q * through a continuous-time Markov process with generator matrix Q . We show that under suitable stability conditions the algorithm converges at a geometric rate. By applying the concept to three different examples, namely, the M/M/1 queue with vacations, the M/G/1 queue, and a tandem network, we illustrate the broad applicability of our approach. For a problem in admission control, we apply our approximation algorithm to Markov decision theory for computing the optimal control policy. Numerical examples are presented to highlight the efficiency of the proposed algorithm.

Keywords: Markov decision processes; deviation matrix; algorithm; probability; Markov processes; queues; algorithms (search for similar items in EconPapers)
Date: 2010
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (1)

Downloads: (external link)
http://dx.doi.org/10.1287/opre.1090.0786 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:inm:oropre:v:58:y:2010:i:4-part-1:p:918-932

Access Statistics for this article

More articles in Operations Research from INFORMS Contact information at EDIRC.
Bibliographic data for series maintained by Chris Asher ().