 Stochastic approximations for finite-state Markov chainsD.-J. Ma, A.M. Makowski and A. Shwartz Stochastic Processes and their Applications, 1990, vol. 35, issue 1, 27-45 Abstract: This paper develops an a.s. convergence theory for a class of projected stochastic approximations driven by finite-state Markov chains. The conditions are mild and are given explicitly in terms of the model data, mainly the Lipschitz continuity of the one-step transition probabilities. The approach used here is a version of the ODE method as proposed by Métivier and Priouret. It combines the Kushner-Clark Lemma with properties of the Poisson equation associated with the underlying family of Markov chains. The class of algorithms studied here was motivated by implementation issues for constrained Markov decision problems, where the policies of interest often depend on quantities not readily available due either to insufficient knowledge of the model parameters or to computational difficulties. This naturally leads to the on-line estimation (or computation) problem investigated here. Several examples from the area of queueing systems are discussed. Keywords: stochastic; approximation; recursive; estimation; adaptive; control; Markov; chains; Poisson; equation; Lipschitz; continuity (search for similar items in EconPapers) Downloads: (external link) Related works: Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text Persistent link: https://EconPapers.repec.org/RePEc:eee:spapps:v:35:y:1990:i:1:p:27-45 Ordering information: This journal article can be ordered from Access Statistics for this article Stochastic Processes and their Applications is currently edited by T. Mikosch More articles in Stochastic Processes and their Applications from Elsevier |
Is your work missing from RePEc? Here is how to contribute.
Questions or problems? Check the EconPapers FAQ or send mail to .
EconPapers is hosted by the
School of Business at Örebro University.