Chapter VII · 28 min

Expectation and Variance

A balance point, not a hope

“The expectation of the sum is the sum of the expectations, whether the variables are independent or not.”
— William Feller, An Introduction to Probability Theory, Vol. I (1950)

A distribution is a whole shape. Often you need a number: the fair price of a game, the centre of a cloud of data, the average error of a model. Expectation is that number when “average” is taken with respect to chance. Variance is how widely the distribution refuses to sit there.

The English word is a saboteur. “I expect the die to show 3.5” is nonsense: the die has no such face. “I expect to win” is a mood. $E [X]$ is neither. It is a balance point — the place a seesaw would rest if you piled probability as mass on the number line. Slow pictures first; the algebra is a caption.

Not “what you expect to happen”

Three things people mean by “expected”, only one of which is ours:

Typical outcome. The mode, or a value with a lot of mass. A die’s typical face is 1, 2, 3, 4, 5, or 6 — not 3.5.
Hope. What you would like. Irrelevant.
Centre of mass. Weight each possible value by its probability and add. That is $E [X]$ . It need not be a possible value, and it need not sit where most of the mass sits.

A weighted average, as a seesaw

If $X$ is discrete, taking values $x$ with masses $p_{X} (x)$ ,

E [X] = x \sum x p_{X} (x),

provided the sum converges absolutely. Physically: put a point mass $p_{X} (x)$ at each number $x$ on a rigid rod. The fulcrum that balances the rod is $E [X]$ .

A fair die as six equal masses. The rod balances halfway between 3 and 4. That point is E[X]. It is not a face, and it does not need to be.

If $X$ takes values on $Ω$ directly, the same quantity is $E [X] = \sum_{ω} X (ω) P ({ω})$ . For equally likely finite $Ω$ , it is the ordinary arithmetic mean of the list of values. A long-run reading, for later: if you roll the die many times, the average of the faces you saw will settle near 3.5. That is the law of large numbers, Chapter XI. Expectation is the number it settles to — defined now, justified then.

When the masses are unequal

Let $P (X = 1) = 0.8$ and $P (X = 10) = 0.2$ . Most of the time you see a 1. The mean is not 1:

E [X] = 1 \cdot 0.8 + 10 \cdot 0.2 = 2.8.

A heavy mass at 1, a light mass far away at 10. The fulcrum sits at 2.8: near the crowd, but tugged toward the rare large value. Means are not modes.

This is why “the average salary” can sit well above what most people earn, and why a model’s average error can look calm while a few examples are disasters. Rare large values have leverage. The seesaw is the whole lesson.

Linearity, with no independence

Independence would let you factor a product: $E [X Y] = E [X] E [Y]$ . Linearity is about sums, and sums do not ask permission. Mixing those two sentences is the second popular trap of this chapter.

Indicators are the bridge back to events: $E [1_{A}] = P (A)$ . Many “how many” questions are sums of indicators, hence sums of probabilities. If you can name the things you are counting, you can often name the expectation without naming the law.

Variance and spread

The mean says where the seesaw balances. It is silent about whether the masses sit next to the fulcrum or at the two far ends. Variance is the expected squared deviation from the mean,

Var (X) = E [(X - E X)^{2}] = E [X^{2}] - (E X)^{2} .

It is always non-negative, and zero only when $X$ is almost surely constant. The standard deviation $σ = Var (X)$ restores the original units — “how many pips away”, not “pips squared”. Unlike expectation, variance is not linear: $Var (X + Y) = Var X + Var Y + 2 Cov (X, Y)$ , and the covariance term vanishes when the variables are independent (more precisely, uncorrelated). Scaling: $Var (a X + b) = a^{2} Var (X)$ . Shifting a random variable never changes its spread; stretching it by 2 stretches the variance by 4.