Chapter X · 24 min

Joint Distributions

How variables move together

“Causal claims cannot be substantiated from associations alone.”
— After Judea Pearl, Causality (2000)

One number is rarely the whole experiment. Height and weight, rainfall and umbrellas, today’s error and yesterday’s — these come in pairs. A joint distribution is a probability on pairs. From it you recover each variable’s private distribution (the marginals) and the way they move together (covariance, and stronger notions).

Tables and marginals

For discrete $X, Y$ , the joint pmf is $p_{X, Y} (x, y) = P (X = x, Y = y)$ . Rows (or columns) sum to the marginals:

p_{X} (x) = y \sum p_{X, Y} (x, y) .

Conditionals are renormalised slices: $p_{Y ∣ X} (y ∣ x) = p_{X, Y} (x, y) / p_{X} (x)$ . Independence of the random variables is independence of all events they generate, and is equivalent to the product of the joint:

p_{X, Y} (x, y) = p_{X} (x) p_{Y} (y) for all x, y .

A joint table as a heat map. Darker cells carry more probability. A marginal is a row-sum or column-sum; independence would make every cell a product of its margins.

Covariance and correlation

Covariance is the expected product of centred variables,

Cov (X, Y) = E [(X - E X) (Y - E Y)] = E [X Y] - E X E Y .

It is positive when they tend to sit on the same side of their means, negative when they tend to sit on opposite sides, and zero when those tendencies cancel (or never existed). Correlation rescales to $[- 1, 1]$ :

ρ (X, Y) = \frac{Cov ( X , Y )}{Var X Var Y} .

Uncorrelated is weaker than independent: independence implies $ρ = 0$ (when the moments exist), but zero correlation only kills linear association. A symmetric banana-shaped cloud can have $ρ = 0$ and still be completely dependent.

Correlation is not causation

Ice cream sales and drowning deaths rise together. A third variable — heat — drives both. The joint distribution of sales and drownings is real; the arrow from cones to casualties is not. Probability on observables cannot, by itself, name a mechanism. You need an assumption about how the world was generated: an experiment, an instrumental variable, a causal graph. Pearl’s book in Further Reading is the long form of this paragraph.

Foundations studio: make the idea yours

This extended studio deliberately slows the pace. It is for a first-time learner who wants to recognize the idea in a new story, not merely reproduce a formula. Work with pencil and paper. Predict before calculating; redraw the pictures; and finish every numerical answer with a sentence in ordinary language.

A mental map before more algebra

Marginal distributions tell us how each variable behaves alone; the joint distribution tells us how they move together. Dependence lives in what the marginals omit.

A working map for Joint Distributions. Cover the labels and reconstruct the chain from memory.

Do not treat the arrows as a theorem. They are a study aid. A strong probability habit is to move back one box whenever a formula feels unmotivated: ask what the experiment is, what information is available, and what quantity the question actually requests.

Three formulas worth being able to narrate

p_{X, Y} (x, y) = ma t hbb P (X = x, Y = y)

Read this line from left to right and explain what every symbol refers to in the experiment. If a symbol has no story, the model is not finished.

p_{X} (x) = s u m_{y} p_{X, Y} (x, y)

Now read the statement backwards: what would have to be known to use it? Backwards reading is often the difference between recognizing a formula and knowing when it applies.

o p er a t or nam e C o v (X, Y) = ma t hbb E [(X - m u_{X}) (Y - m u_{Y})]

Test the expression at an edge case or simple symmetric case. Probability formulas should survive sanity checks before you trust the arithmetic built on them.

Worked example ladder

A small experiment you can actually do

A five-step habit for every example in this chapter.

What usually goes wrong

When you notice this mistake, do not merely correct the final number. Return to the first line where the model became ambiguous. Probability errors are often representation errors wearing arithmetic clothing.

Questions beginners are right to ask

What is a marginal?

It is the distribution of one variable obtained by summing or integrating out the others.

What is covariance measuring?

A signed average of how deviations from the means co-move; it captures linear association, not all forms of dependence.

Why condition inside a joint model?

Conditioning slices the joint distribution and renormalizes the slice, revealing subgroup behavior.

Where the abstraction earns its keep

For each application, ask what counts as an outcome, what the model treats as random, and which assumptions are approximations. This is how probability becomes a modelling language instead of a catalogue of formulas.

Connections: do not store chapters in separate boxes

Problem-solving clinic: from recognition to fluency

There is a stage where every worked example looks clear but a fresh problem still feels foreign. The cure is not another formula; it is practice choosing the representation. Before equations, do a sixty-second scan: identify the experiment, what is known, what remains uncertain, the quantity being asked for, the assumption doing the heavy lifting, and one impossible answer that gives you a sanity bound.

The expert loop returns every calculation to the original story.

Case clinic A: Two dice

The two die faces have a joint distribution on a 6 by 6 grid. The sum X+Y depends on the diagonal counts, while each die marginal remains uniform.

Case clinic B: Same marginals, different dependence

Two variables can each be fair bits yet be identical, opposite, or independent. Their marginals are the same in all three stories, but their joint laws are completely different.

Solve or reason about it twice: once exactly and once with a rough estimate, simulation, or symmetry argument. If the two approaches disagree dramatically, investigate before trusting the more sophisticated calculation.

Debug a confident wrong answer

Two questions to answer without notes

What is a marginal? It is the distribution of one variable obtained by summing or integrating out the others.

What is covariance measuring? A signed average of how deviations from the means co-move; it captures linear association, not all forms of dependence.

A notebook protocol for proficiency

Give this chapter one notebook page divided into four quadrants: picture, formula, example, mistake. Redraw the main visual from memory, narrate one formula in English, invent a fresh story using the same mathematics, and record the most tempting wrong move. Revisit the page after two days and again after a week.

A mastery check before you move on

Try these without looking back. If one item feels slippery, return to the corresponding example and rebuild it rather than memorizing the answer.

Give a one-minute explanation of the chapter title to a curious teenager without a formula.
Invent a tiny example with at most six elementary outcomes and solve it completely by enumeration.
State one assumption that would make your example invalid and identify exactly which line would break.
Draw the mental map from memory and connect at least two boxes to an earlier or later chapter.
Write one question whose answer you still do not know. Good questions show that the concept has become active rather than passive.