The critical discount factor for finite Markovian decision processes with an absorbing set

K. Hinderer and K.-H. Waldmann

Mathematical Methods of Operations Research, 2003, vol. 57, issue 1, 19 pages

Abstract: This paper deals with a Markovian decision process with an absorbing set J 0 . We are interested in the largest number β * ≥1, called the critical discount factor, such that for all discount factors β smaller than β * the limit V of the N-stage value function V N for N →∞ exists and is finite for each choice of the one-stage reward function. Several representations of β * are given. The equality of 1/β * with the maximal Perron/Frobenius eigenvalue of the MDP links our problem and our results to topics studied intensively (mostly for β=1) in the literature. We derive in a unified way a large number of conditions, some of which are known, which are equivalent either to β>β * or to β * >1. In particular, the latter is equivalent to transience of the MDP. A few of our findings are extended with the aid of results in Rieder (1976) to models with standard Borel state and action space. We also complement an algorithm of policy iteration type, due to Mandl/Seneta (1969), for the computation of β * . Finally we determine β * explicitly in two models with stochastically monotone transition law. Copyright Springer-Verlag Berlin Heidelberg 2003

Keywords: MS classification 2000: 90C40; 47J10; Key words: transient Markovian decision processes; expected total reward criterion; stochastic shortest path problem; sublinear Perron/Frobenius theorem; spectral radius of transition matrices; stochastic monotonicity (search for similar items in EconPapers)
Date: 2003
References: Add references at CitEc
Citations: View citations in EconPapers (1)

Downloads: (external link)
http://hdl.handle.net/10.1007/s001860200252 (text/html)
Access to full text is restricted to subscribers.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:mathme:v:57:y:2003:i:1:p:1-19

Ordering information: This journal article can be ordered from
http://www.springer.com/economics/journal/00186

DOI: 10.1007/s001860200252

Access Statistics for this article

Mathematical Methods of Operations Research is currently edited by Oliver Stein

More articles in Mathematical Methods of Operations Research from Springer, Gesellschaft für Operations Research (GOR), Nederlands Genootschap voor Besliskunde (NGB)
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().