Outline
Markov Chains and Stationary Distributions
Matt Williamson1
1Lane Department of Computer Science and Electrical Engineering
West Virginia University
March 19, 2012
Williamson
Markov Chains and Stationary Distributions
Outline
Outline
1 Stationary Distributions
Fundamental Theorem of Markov Chains
Computing Stationary Distributions
Example: A Simple Queue
Williamson
Markov Chains and Stationary Distributions
Outline
Outline
1 Stationary Distributions
Fundamental Theorem of Markov Chains
Computing Stationary Distributions
Example: A Simple Queue
2 Random Walks on Undirected Graphs
Application: An s-t Connectivity Algorithm
Williamson
Markov Chains and Stationary Distributions
Outline
Outline
1 Stationary Distributions
Fundamental Theorem of Markov Chains
Computing Stationary Distributions
Example: A Simple Queue
2 Random Walks on Undirected Graphs
Application: An s-t Connectivity Algorithm
3 Parrondo’s Paradox
Williamson
Markov Chains and Stationary Distributions
Stationary Distributions
Random Walks on Undirected Graphs
Parrondo’s Paradox
Fundamental Theorem of Markov Chains
Computing Stationary Distributions
Example: A Simple Queue
Outline
1 Stationary Distributions
Fundamental Theorem of Markov Chains
Computing Stationary Distributions
Example: A Simple Queue
2 Random Walks on Undirected Graphs
Application: An s-t Connectivity Algorithm
3 Parrondo’s Paradox
Williamson
Markov Chains and Stationary Distributions
Stationary Distributions
Random Walks on Undirected Graphs
Parrondo’s Paradox
Fundamental Theorem of Markov Chains
Computing Stationary Distributions
Example: A Simple Queue
Probability Matrix
Recall
Let P be a one-step probability matrix of a Markov chain such that
P =
P0,0 P0,1
P1,0 P1,1
...
...
Pi,1
Pi,0
...
...
· · · P0,j
· · · P1,j
...
...
· · ·
Pi,j
...
...
· · ·
· · ·
...
· · ·
...
.
Williamson
Markov Chains and Stationary Distributions
Stationary Distributions
Random Walks on Undirected Graphs
Parrondo’s Paradox
Fundamental Theorem of Markov Chains
Computing Stationary Distributions
Example: A Simple Queue
Probability Matrix
Recall
Let P be a one-step probability matrix of a Markov chain such that
P =
P0,0 P0,1
P1,0 P1,1
...
...
Pi,1
Pi,0
...
...
· · · P0,j
· · · P1,j
...
...
· · ·
Pi,j
...
...
· · ·
· · ·
...
· · ·
...
.
Let ¯p(t) = (p0(t), p1(t), p2(t), . . . ) be the vector giving the probability distribution of
the state of the chain at time t.
Williamson
Markov Chains and Stationary Distributions
Stationary Distributions
Random Walks on Undirected Graphs
Parrondo’s Paradox
Fundamental Theorem of Markov Chains
Computing Stationary Distributions
Example: A Simple Queue
Probability Matrix
Recall
Let P be a one-step probability matrix of a Markov chain such that
P =
P0,0 P0,1
P1,0 P1,1
...
...
Pi,1
Pi,0
...
...
· · · P0,j
· · · P1,j
...
...
· · ·
Pi,j
...
...
· · ·
· · ·
...
· · ·
...
.
Let ¯p(t) = (p0(t), p1(t), p2(t), . . . ) be the vector giving the probability distribution of
the state of the chain at time t. Then,
¯p(t) = ¯p(t − 1) · P
Williamson
Markov Chains and Stationary Distributions