Discrete Markov Processes and Numerical Algorithms for Markov Chains

Dmitrii Lozovanu and Stefan Wolfgang Pickl
Additional contact information
Dmitrii Lozovanu: Moldowa Academy of Science
Stefan Wolfgang Pickl: Universität der Bundeswehr München

Chapter Chapter 1 in Markov Decision Processes and Stochastic Positional Games, 2024, pp 1-124 from Springer

Abstract: Abstract This chapter states the necessary classical results on discrete-time Markov processes and presents some approaches for determining the basic probabilistic characteristics of finite state space Markov chains. The main focus is on the elaboration of efficient numerical algorithms for computing the state-time probabilities, the limiting and differential matrices, as well as the average and expected total discounted rewards for discrete-time Markov processes. New algorithms for determining the limiting and differential matrices for such processes based on the z-transform are developed, and innovative procedures for calculating the state-time probabilities based on dynamic programming are substantiated.

Keywords: Markov chain; Markov processes with rewards; State-time probabilities; Limiting matrix; Differential matrix; Average expected rewards; Discounted expected rewards (search for similar items in EconPapers)
Date: 2024
References: Add references at CitEc
Citations:

There are no downloads for this item, see the EconPapers FAQ for hints about obtaining it.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:isochp:978-3-031-40180-0_1

Ordering information: This item can be ordered from
http://www.springer.com/9783031401800

DOI: 10.1007/978-3-031-40180-0_1

Access Statistics for this chapter

More chapters in International Series in Operations Research & Management Science from Springer
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().