Chapter XI · 22 min

Limit Theorems

Why averages settle

“The law of large numbers is the bridge between probability and the world of observed frequencies.”
— After Jakob Bernoulli and Émile Borel

A single coin is noise. A thousand coins are a fact. Limit theorems are the precise form of that slogan. They are why polling works, why sample means are worth computing, and why a clinic that has seen a hundred patients can say something about a disease it will never see in every person.

History

10,000 tosses in wartime: John Kerrich’s experiment

In 1940, while visiting Denmark, mathematician John Edmund Kerrich was caught there after the German invasion and interned as a British national. During his enforced confinement he turned probability into something almost stubbornly physical: with fellow internee Eric Christensen, he tossed a coin again and again and recorded the results. The experiment eventually reached 10,000 tosses.

The final count was 5,067 heads and 4,933 tails, so the observed proportion of heads was $5067/10000 = 0.5067$ . The important part is not that 0.5067 is close to 0.5; it is the path by which it got there. After 10 tosses the running proportion was 0.4; after 30 it was about 0.567; after 200 it was 0.49; by the thousands it hovered much more tightly around one half. The fluctuations did not disappear — their relative sizeshrank.

Kerrich later built these experiments into his 1946 book An Experimental Introduction to the Theory of Probability, written around experimental results begun during the internment. Long before a browser could simulate a million flips in a second, 10,000 hand tosses made the law of large numbers visible one mark at a time.

The law of large numbers

First, a piece of jargon you will meet on every later page. A sequence of random variables is i.i.d. when two things are true at once:

Independent — learning one does not change the chance of another (Chapter V). Separate tosses, not a sticky coin.
Identically distributed — each has the same distribution. The same coin, not a mix of fair and loaded. Same mean, same variance, same everything.

The letters stand for Independent and Identically Distributed. People say “i.i.d. sample” the way they say “fair die”: a modelling choice, not a fact about the world. A stacked deck is not i.i.d.; a well-shuffled one, dealt one card at a time without replacement, is not quite either (the cards are dependent), though for a small hand from a large deck it is close.

Let $X_{1}, X_{2}, \dots$ be i.i.d. with finite mean $μ = E [X_{1}]$ . Write $\overset{ˉ}{X}_{n} = (X_{1} + \dots + X_{n}) / n$ . Then $\overset{ˉ}{X}_{n} \to μ$ as $n \to \infty$ — in probability (the weak law) and, under the same hypotheses, almost surely (the strong law). Chebyshev plus vanishing variance of the mean, $Var (\overset{ˉ}{X}_{n}) = σ^{2} / n$ , already gives the weak law when the variance is finite.

Three running averages of coin flips. Early on, the paths wander; as n grows they settle toward 1/2. The gold dashed line is the mean they are trying to remember.

The law does not say that the next flip is due to compensate. Memorylessness of independent trials is still in force. It says the average so far forgets its start. Visit the settling averages lab and watch a path calm down.

The central limit theorem

The average settles; the shape of its error becomes universal. If the $X_{i}$ are i.i.d. with mean $μ$ and finite positive variance $σ^{2}$ , then

\frac{X ˉ _{n} - μ}{σ / n} d N (0, 1) .

Convergence in distribution means that probabilities of intervals settle to the corresponding normal probabilities. The left-hand side is the number of standard errors by which the sample mean misses $μ$ . For large $n$ , that number looks like a standard normal, even if each $X_{i}$ was a coin, a die, or a waiting time.

This is why the bell of Chapter 9 keeps returning. Add many independent small contributions, scale by $n$ , and the density of the sum forgets the density of the parts. (The hypotheses can fail: infinite variance, strong dependence, one term dominating. Then other limit laws — stable laws, not Gaussian — may appear.)

Foundations studio: make the idea yours

This extended studio deliberately slows the pace. It is for a first-time learner who wants to recognize the idea in a new story, not merely reproduce a formula. Work with pencil and paper. Predict before calculating; redraw the pictures; and finish every numerical answer with a sentence in ordinary language.

A mental map before more algebra

The LLN and CLT answer different questions. The LLN locates the average; the CLT describes the scale and approximate shape of its remaining random error.

A working map for Limit Theorems. Cover the labels and reconstruct the chain from memory.

Do not treat the arrows as a theorem. They are a study aid. A strong probability habit is to move back one box whenever a formula feels unmotivated: ask what the experiment is, what information is available, and what quantity the question actually requests.

Three formulas worth being able to narrate

ba r X_{n} t o m u

Read this line from left to right and explain what every symbol refers to in the experiment. If a symbol has no story, the model is not finished.

s q r t n (ba r X_{n} - m u) / s i g ma R i g h t a r r o w N (0, 1)

Now read the statement backwards: what would have to be known to use it? Backwards reading is often the difference between recognizing a formula and knowing when it applies.

o p er a t or nam e s d (ba r X_{n}) = s i g ma / s q r t n

Test the expression at an edge case or simple symmetric case. Probability formulas should survive sanity checks before you trust the arithmetic built on them.

Worked example ladder

A small experiment you can actually do

A five-step habit for every example in this chapter.

What usually goes wrong

When you notice this mistake, do not merely correct the final number. Return to the first line where the model became ambiguous. Probability errors are often representation errors wearing arithmetic clothing.

Questions beginners are right to ask

Why sqrt(n)?

Variances of independent sums grow like n, so standard deviations grow like sqrt(n); dividing an average by n leaves scale 1/sqrt(n).

Does CLT require normal data?

No. Its power is precisely that many non-normal populations yield approximately normal standardized sums under suitable conditions.

How large must n be?

There is no universal threshold. Skewness, heavy tails, dependence, and the quantity being approximated all matter.

Where the abstraction earns its keep

For each application, ask what counts as an outcome, what the model treats as random, and which assumptions are approximations. This is how probability becomes a modelling language instead of a catalogue of formulas.

Connections: do not store chapters in separate boxes

Problem-solving clinic: from recognition to fluency

There is a stage where every worked example looks clear but a fresh problem still feels foreign. The cure is not another formula; it is practice choosing the representation. Before equations, do a sixty-second scan: identify the experiment, what is known, what remains uncertain, the quantity being asked for, the assumption doing the heavy lifting, and one impossible answer that gives you a sanity bound.

The expert loop returns every calculation to the original story.

Case clinic A: Kerrich's 10,000 tosses

John Kerrich's wartime coin-toss record makes the LLN visible: the running proportion wanders early, then settles near one half without ever being forced to equal one half exactly.

Case clinic B: Polling

If independent sampled responses have finite variance, the standard error of an average shrinks roughly like 1/sqrt(n). Quadrupling the sample roughly halves the noise scale.

Solve or reason about it twice: once exactly and once with a rough estimate, simulation, or symmetry argument. If the two approaches disagree dramatically, investigate before trusting the more sophisticated calculation.

Debug a confident wrong answer

Two questions to answer without notes

Why sqrt(n)? Variances of independent sums grow like n, so standard deviations grow like sqrt(n); dividing an average by n leaves scale 1/sqrt(n).

Does CLT require normal data? No. Its power is precisely that many non-normal populations yield approximately normal standardized sums under suitable conditions.

A notebook protocol for proficiency

Give this chapter one notebook page divided into four quadrants: picture, formula, example, mistake. Redraw the main visual from memory, narrate one formula in English, invent a fresh story using the same mathematics, and record the most tempting wrong move. Revisit the page after two days and again after a week.

A mastery check before you move on

Try these without looking back. If one item feels slippery, return to the corresponding example and rebuild it rather than memorizing the answer.

Give a one-minute explanation of the chapter title to a curious teenager without a formula.
Invent a tiny example with at most six elementary outcomes and solve it completely by enumeration.
State one assumption that would make your example invalid and identify exactly which line would break.
Draw the mental map from memory and connect at least two boxes to an earlier or later chapter.
Write one question whose answer you still do not know. Good questions show that the concept has become active rather than passive.