Detailed information 
Original study plan 
Bachelor's programme Technical Mathematics 2022W 
Objectives 
A Markov chain is a mathematical model that is useful in the study of complex systems. The basic concepts of a Markov chain are the state of a system and the transition from one state to another. It is said that a system is in a certain state when random variables that fully describe the system take on the values assigned to that state. A transition of the system from one state to another occurs when the variables that describe the system change their values accordingly. The purpose of this course is to give an analytical structure to a Markov decision problem which at the same time describes the system sufficiently well and is still computationally usable.

Subject 
 Markovchain with a discrete time
 Controlled Markovchain
 Iterative solution for sequential decision processes
 The policyiteration for the solution of sequential decision processes
 Applications of the policyiteration algorithm
 The policyiteration algorithm for the processes with several ergodic classes
 The sequential decision processes with discounting
 Continuoustime Markovchains
 The controllable continuoustime Markovchains
 The continuous decision problems
 The continuous decision problems with discounting
 Conclusion

Criteria for evaluation 
Written exam

Methods 
Slides and blackboard presentation

Language 
English and French 
Study material 
 Lecture notes
 Howard R., Dynamic programming and Markov processes. Wiley Series, 1960.
 Puterman M., L. Markov decision process. Wiley series in Probability and Mathematical Statistics, 1994.

Changing subject? 
No 
Further information 
Until term 2022S known as: TM1WCVOMARK VL Markov chains
