Resume
Research
Learning
Blog
Teaching
Jokes
Kernel Papers

# Markov Process

## Definition

A Markov process is a discrete-time stochastic process. To constuct an MP, 3 components are necessary:

1. A set $$\mathcal{X}$$, commonly called “states”
2: An initial probability distribution over the set of states: $$p_0$$ 3: A transition operator $$T: \mathcal{X} \rightarrow \mathbb{P}(\mathcal{X}}$$, which specifies a conditional distribution
$T(X_t = x) := P(X_{t+1} = x' \lvert X_t = x})$

A Markov process is a set of random variables indexed by time $$\{X_t\}_{t=1, 2, 3...}$$ where the distribution is given by:

\begin{align*} X_1 &\sim p_0\\ X_{t+1} \lvert X_{t} &\sim T X_t \end{align*}

If the set $$\mathcal{X}$$ is finite, the transition operator can be written as a $$\lvert \mathcal{X} \lvert \times \lvert \mathcal{X} \lvert$$ dimensional matrix called a transition matrix or stochastic matrix.

## Properties

### Finite Dimensional

• Each column of $$T$$ must sum to 1

• The marginal probability of the $$k$$th variable $$X_k$$ is given by

$p(X_k) = T p(X_{k-1}) = T^{k-1} X_1$