Chapter IV · 32 min

Bayes' Theorem

The words, then the inversion

“The probability of any event is the ratio between the value at which an expectation depending on the happening of the event ought to be computed, and the value of the thing expected upon its happening.”

Bayes’ theorem is not a new axiom, and it is not a personality. It is the multiplication rule, solved for the factor you did not observe. You see a test result; you want the disease. You see an email; you want “spam or not”. You see data; you want the story that produced it. The theorem is permission to invert the arrow.

The formula is short. The words around it are what scare people — prior, likelihood, evidence, posterior, hypothesis. They sound like a secret handshake. They are not. Each one is a role in a three-line play: a story you might believe, a thing you just saw, and a number for how much the seeing should change the believing. This chapter names the cast before anyone has to act.

You already know the tree

Chapter III drew a tree: first the disease (rare), then the test. You read it root to leaf. Bayes is the same tree read leaf to root. The test result is in; you ask which first-branch you are probably on. No new arithmetic. A new question. If conditional probability still feels like a ratio of regions, stay there one more minute: Bayes is that ratio, solved for the factor you did not observe.

The words, one at a time

We will use a clinic as a running picture, but the words do not belong to medicine. They belong to any situation with a hidden story and a visible clue.

Hypothesis

A hypothesis is a story about the world that might be true. Not a guess you are married to — just a labelled branch. In the clinic the two hypotheses are “has the disease” ($D$) and “does not” ($D^c$). In an inbox they are “spam” and “ham”. In learning they are values of a parameter $\theta$. Call them what you like; they must partition the possibilities you are willing to entertain. If you leave a story out of the list, Bayes cannot resurrect it.

People also say cause, class, state, or model. Same job: the thing you cannot see directly, the thing you will have a degree of belief about.

Prior

The prior is how plausible each hypothesis was before this particular clue. Write $P(H)$. It is not a confession of bias, and it is not cheating. It is the base rate, the prevalence, what you would have said if someone had stopped you in the corridor yesterday. A disease that hits 1 in 100 has prior $0.01$. A fair-looking coin has prior “probably near one half”. A stranger’s email, before you read it, is probably ham.

If you skip the prior, you are answering a different question from the one you were asked. That is the whole tragedy of the 99% answer later in this chapter.

Data

The data (also: observation, measurement, clue, test result) is the thing that actually happened, the event you now condition on. A positive test $T^{+}$. The word “loan” in the subject line. Door 3 opening. Data is not a probability. It is an outcome, or a set of them. We will write $P(\text{data}\mid H)$ in a moment; do not let the notation trick you into thinking the data itself is a number between 0 and 1.

Likelihood

The likelihood is $P(\text{data}\mid H)$: if this story were true, how unsurprising is what we just saw? A good test of a disease has likelihood of a positive result, given disease, near 1. The same test, given health, has a small likelihood of positive — that small number is the false-positive rate.

Evidence — two meanings, unfortunately

English “evidence” means the clue: the positive test, the witness statement, the dataset. That is the data.

Bayesian jargon also calls $P(\text{data})$ “the evidence” (or marginal likelihood). That number is not the clue. It is how unsurprising the clue is if you have not committed to a hypothesis — the average of the likelihoods, weighted by the priors. It sits in the denominator and makes the posteriors add to one. When someone says “the evidence”, ask: do they mean the event, or $P(\text{event})$? This book will say data for the event and marginal probability of the data for the denominator, unless we are quoting the standard slogan.

Posterior

The posterior is how plausible each hypothesis is after the clue: $P(H\mid \text{data})$. It is the prior, reweighted by how well each story predicted what you saw, then renormalised so the stories still add to one. You came in with a prior. You leave with a posterior. Tomorrow, if more data arrives, today’s posterior is tomorrow’s prior. That is all “updating” means.

The inversion, with every piece labelled

Whenever $P(\text{data})>0$,

The denominator is usually expanded by total probability, because that is the expansion the tree already computed:

A slogan that is accurate enough to tape above a desk: posterior is proportional to likelihood times prior. The denominator is the same for every hypothesis; it only rescales. If you just want to know which story wins, you can compare the numerators and skip the division. If you want a probability you can say out loud, you divide.

The clinic, named slowly

A disease has prevalence 1%. That is the prior: $P(D)=0.01$, so $P(D^c)=0.99$. A test detects the disease with probability 0.99 when it is present (true-positive rate: the likelihood of a positive given disease) and falsely flags the healthy with probability 0.05 (false-positive rate: the likelihood of a positive given health). You test positive. That positive result is the data. What you want is the posterior $P(D\mid T^{+})$.

Almost everyone answers “about 99%”. They have quoted the likelihood of the data given disease, and dressed it as a posterior. The theorem answers otherwise.

Read the four numbers once more, with their badges on:

• $0.01$ — prior of disease.
• $0.99$ in the numerator — likelihood of a positive, if diseased.
• $0.05\times 0.99$ — likelihood of a positive if healthy, times the prior of being healthy. The false-alarm mass.
• $0.0594$ — marginal probability of a positive. All the ways the data happens.

One in six. The test is sharp; the disease is rare; most positive tests are false alarms from the huge healthy population. Prevalence is not a footnote. It is half the formula.

The clinic lab lets you drag prevalence and error rates until the surprise becomes ordinary. Do that before you memorise anything.

A second story, with no medicine in it

You have lost your keys. Hypothesis $K$: they are in the kitchen (prior 0.7, because that is where they usually are). Hypothesis $K^c$: they are in the hallway (prior 0.3). You glance at the kitchen hook. Data: the hook is empty. If the keys were in the kitchen you would have seen them 90% of the time; if they are in the hallway the hook is empty for sure. So

Same badges: 0.7 is the prior, 0.10 is the likelihood of an empty hook if they really are in the kitchen (you missed them), 0.37 is the marginal of the empty hook. After one look, “kitchen” has dropped from probably to roughly one in five. You have not proved the hallway. You have updated.

Likelihood is not a probability of the hypothesis

$P(\text{data}\mid H)$ is the chance of the data given a story. It is not the chance of the story. Those two numbers can point in opposite directions: a nearly perfect test still yields a modest posterior when the prior is tiny. Confusing likelihood with posterior is the error of the 99% answer above. It is also the error of quoting $P(\text{DNA match}\mid \text{innocent})$ in court when the jury asked for $P(\text{innocent}\mid \text{DNA match})$.

Yesterday’s posterior is today’s prior

Suppose you take a second, independent test, and it is also positive. You may start over with both tests as one datum, or you may take the 0.167 from the first test and use it as the prior for the second. Both routes give about 0.80. Sequential updating is the same theorem applied twice. “Learning from data”, in the Bayesian sense, is this habit: never throw the prior away, and never refuse to revise it.