Information Theory

Introductory Quantities

• Entropy
• Cross Entropy
• Mutual Information
• KL Divergence

Inequalities

• Jensen’s Inequality
  • Gibb’s Inequality
• Log-Sum Inequality

Advanced Quantities

• f-Divergence