Chapter XIII · 24 min

Entropy and Information

Surprise, divergence, and why models train

“Information is the resolution of uncertainty.”
— After Claude Shannon, A Mathematical Theory of Communication (1948)

A probability distribution is a complete story about uncertainty. Often you need a number that says how much story is there: how surprised a source is on average, how different two stories are, how much one variable tells you about another. Those numbers are entropy, Kullback–Leibler divergence, and mutual information. They are the native language of modern training losses.

Surprise and entropy

If an event of probability $p$ occurs, a natural measure of surprise is $- lo g p$ . Certain events ( $p = 1$ ) are unsurprising; rare events are surprising. The logarithm makes independent surprises add. (The base chooses the unit: base 2 is bits, base $e$ is nats. Learning papers usually live in nats because calculus does.)

The entropy of a discrete random variable is expected surprise:

H (X) = - x \sum p_{X} (x) lo g p_{X} (x) = E [- lo g p_{X} (X)] .

Entropy is non-negative, zero only for a sure (deterministic) variable, and maximised — among distributions on a fixed finite set — by the uniform. A fair coin has entropy 1 bit; a coin with $p = 0.9$ has less, because it is easier to guess.

For a continuous variable the same idea yields differential entropy $h (X) = - \int f lo g f$ , which can be negative and is not invariant to changes of units. Treat it as a relative tool, not as a “number of bits in a real number” — that last quantity is infinite.

Joint, conditional, chain

Joint entropy $H (X, Y)$ is the surprise in the pair. Conditional entropy $H (Y ∣ X) = E [H (Y ∣ X = x)]$ is the leftover surprise in $Y$ once $X$ is known. The chain rule is the same shape as probability’s:

H (X, Y) = H (X) + H (Y ∣ X) = H (Y) + H (X ∣ Y) .

Conditioning cannot increase entropy: $H (Y ∣ X) \leq H (Y)$ . Information does not make the world noisier.

Mutual information

The reduction in surprise about $Y$ upon seeing $X$ is the mutual information

I (X; Y) = H (Y) - H (Y ∣ X) = H (X) + H (Y) - H (X, Y) .

It is symmetric, non-negative, and zero if and only if $X$ and $Y$ are independent. Feature selection, representation learning, and “does this embedding know the label?” are all questions about $I (X; Y)$ . Unlike correlation, mutual information sees nonlinear dependence.

KL divergence and cross-entropy

Given a true distribution $p$ and a model $q$ , the extra surprise of using $q$ to encode samples from $p$ is the Kullback–Leibler divergence

D_{KL} (p ∥ q) = x \sum p (x) lo g \frac{p ( x )}{q ( x )} = E_{x \sim p} [lo g p (x) - lo g q (x)] .

It is non-negative (Gibbs’ inequality / Jensen), and zero if and only if $p = q$ on a set of probability one. It is not a metric: it is asymmetric, and it does not obey the triangle inequality. $D_{KL} (p ∥ q)$ is large when $p$ puts mass where $q$ does not; the other direction punishes the reverse.

Cross-entropy is expected surprise under the model, with data from the truth:

H (p, q) = - E_{x \sim p} [lo g q (x)] = H (p) + D_{KL} (p ∥ q) .

For a fixed true distribution $p$ , minimising cross-entropy in $q$ is exactly minimising KL, which is exactly maximum likelihood. That is why a scored forecast, a multiple-choice model, or a classifier trained with softmax and cross-entropy is doing probability, whether or not the write-up says so.

Forward KL, reverse KL, and mode behaviour

Maximum likelihood minimises $D_{KL} (p_{data} ∥ p_{θ})$ — forward KL. The expectation is under the data, so the model is punished for missing mass that the data has: it tends to cover all modes, even if it smears. Minimising the reverse $D_{KL} (p_{θ} ∥ p_{data})$ is the variational / GAN-adjacent mood: the model is punished for putting mass where the data has none, and is allowed to drop modes. When a generated sample looks too average, or too sharp and incomplete, you are watching this asymmetry.