Inhalt

[ 201WTMSMACV22 ] VL Markov Chains

Versionsauswahl
Workload Education level Study areas Responsible person Hours per week Coordinating university
3 ECTS B3 - Bachelor's programme 3. year Mathematics Dmitry Efrosinin 2 hpw Johannes Kepler University Linz
Detailed information
Original study plan Bachelor's programme Technical Mathematics 2023W
Objectives A Markov chain is a mathematical model that is useful in the study of complex systems. The basic concepts of a Markov chain are the state of a system and the transition from one state to another. It is said that a system is in a certain state when random variables that fully describe the system take on the values assigned to that state. A transition of the system from one state to another occurs when the variables that describe the system change their values accordingly. The purpose of this course is to give an analytical structure to a Markov decision problem which at the same time describes the system sufficiently well and is still computationally usable.
Subject
  1. Markov-chain with a discrete time
  2. Controlled Markov-chain
  3. Iterative solution for sequential decision processes
  4. The policy-iteration for the solution of sequential decision processes
  5. Applications of the policy-iteration algorithm
  6. The policy-iteration algorithm for the processes with several ergodic classes
  7. The sequential decision processes with discounting
  8. Continuous-time Markov-chains
  9. The controllable continuous-time Markov-chains
  10. The continuous decision problems
  11. The continuous decision problems with discounting
  12. Conclusion
Criteria for evaluation Written exam
Methods Slides and blackboard presentation
Language English and French
Study material
  • Lecture notes
  • Howard R., Dynamic programming and Markov processes. Wiley Series, 1960.
  • Puterman M., L. Markov decision process. Wiley series in Probability and Mathematical Statistics, 1994.
Changing subject? No
Earlier variants They also cover the requirements of the curriculum (from - to)
TM1WCVOMARK: VO Markov chains (2000S-2022S)
On-site course
Maximum number of participants -
Assignment procedure Direct assignment