Bayesian Nonparametrics

Machine Learning Probability Statistics

Overview

Infinite-dimensional probabilistic models including Dirichlet processes, Gaussian processes, and related constructions.

Bayesian Nonparametrics (BNPs) are probabilistic models that leverage infinite dimensional distributions based on stochastic processes. Stochastic processes are a set of random variables indexed by some index set (e.g. time or integers). For instance, a sequence of stock prices \(\{ X_t \}_{0 \leq t \leq T }\) is a stochastic process where the index set is time \(t \in [0, T]\). The utility of defining distributions over stochastic processes is that it makes the model significantly more powerful; for instance, if the variables of the process are parameters for observations, then the model can have access to an unbounded number of parameters to use as appropriate. One interpretation is therefore that Bayesian Nonparametrics are distributions over function spaces, where the function maps the index set to the set of random variables.

Broadly, there are three approaches to using BNPs:

1. Exponential Family Conjugacy, which converts the integral into vector addition
2. Size-based representations, which offers exact inference via slice sampling
3. Marginalization, which integrates out the traits to create marginal processes e.g. Chinese Restaurant Process, Indian Buffet Process

Contents

• Bernoulli Process
• Beta Process
• Chinese Restaurant Process
• Completely Random Measures
• Dependent Dirichlet Process
• Dirichlet Process
• Distance Dependent Chinese Restaurant Process
• Distance Dependent Indian Buffet Process
• Gamma Process
• Gaussian Process
• Indian Buffet Process
• Infinite Hidden Markov Model
• Levy Process
• MacQueen Urn Scheme
• Poisson Point Process
• Polya Urn Scheme
• Stick Breaking Process

References

• Peter Orbanz’s
• Gershman and Blei’s A Tutorial on Bayesian Nonparametric Models