Probability

Parametric Distributions

Many common distributions depend on specific parameters. Parameters are frequently classified into one of several possible types:

• location parameter: shifts the distribution e.g. Gaussian mean
• scale (inverse rate): stretches/squeezes the distribution e.g. Laplace diversity
• shape: changes the shape e.g. Beta \(\alpha, \beta\)

Some common discrete, continuous and special parametric distributions are:

Discrete Distributions

• Bernoulli
• Beta-Bernoulli
• Binomial
• Logarithmic
• Poisson

Continuous Distributions

• Beta
• Cauchy
• Continuous Bernoulli
• Exponential
• Kumaraswamy
• Gamma
• Normal/Gaussian
• von Mises-Fisher

Classes of Distribution

• Elliptical Distributions
• Exponential Family Distributions
• Levy-Stable Distributions
• Location-Scale Distributions

Random Variables

• Moment generating functions

Probability Theory

• Probability Space
• Probability Measure
• \(\sigma\)-Algebra
• Random Variables
• Change of variable theorem
• Notions of Convergence

Distances and Divergences

Probability distances and divergences have commonly encountered properties. Some common probability distances and divergences are

• Cramer Distance
• Jensen-Shannon Divergence
• Kullback-Leibler Divergence
• Total Variation Distance
• Wasserstein Distance

Probability Integral Transform

Theorem: For any random variable \(X\), its CDF \(F_X(x)\) is distributed uniformly over \((0,1)\). That is, if we define \(Y = F_X(x)\), then \(Y \sim \mathcal{U}(0,1)\).

Proof:

$$ \begin{align*} P(Y \leq y) &= P(F_X(x) \leq y)\\ &= P(x \leq F_X^{-1}(y))\\ &= F_X(F_X^{-1}(y))\\ &= y \end{align*} Since only $$\mathcal{U}(0,1)$$ has a CDF $$F_Y(y) = P(Y \leq y) = y$$, we conclude that $$Y$$ is distributed uniformly.