Information Theory

Mathematics Computer Science

Overview

Entropy, mutual information, coding theory, and information-theoretic concepts.

Introductory Quantities

• Entropy
• Cross Entropy
• Mutual Information
• KL Divergence

Inequalities

• Jensen’s Inequality
  • Gibb’s Inequality
• Log-Sum Inequality

Advanced Quantities

• f-Divergence