Chapter X · 18 min

Joint Distributions

How variables move together

“Causal claims cannot be substantiated from associations alone.”
— After Judea Pearl, Causality (2000)

One number is rarely the whole experiment. Height and weight, a feature and a label, today’s error and yesterday’s — these come in pairs. A joint distribution is a probability on pairs. From it you recover each variable’s private distribution (the marginals) and the way they move together (covariance, and stronger notions).

Tables and marginals

For discrete $X, Y$ , the joint pmf is $p_{X, Y} (x, y) = P (X = x, Y = y)$ . Rows (or columns) sum to the marginals:

p_{X} (x) = y \sum p_{X, Y} (x, y) .

Conditionals are renormalised slices: $p_{Y ∣ X} (y ∣ x) = p_{X, Y} (x, y) / p_{X} (x)$ . Independence of the random variables is independence of all events they generate, and is equivalent to the product of the joint:

p_{X, Y} (x, y) = p_{X} (x) p_{Y} (y) for all x, y .

A joint table as a heat map. Darker cells carry more probability. A marginal is a row-sum or column-sum; independence would make every cell a product of its margins.

Covariance and correlation

Covariance is the expected product of centred variables,

Cov (X, Y) = E [(X - E X) (Y - E Y)] = E [X Y] - E X E Y .

It is positive when they tend to sit on the same side of their means, negative when they tend to sit on opposite sides, and zero when those tendencies cancel (or never existed). Correlation rescales to $[- 1, 1]$ :

ρ (X, Y) = \frac{Cov ( X , Y )}{Var X Var Y} .

Uncorrelated is weaker than independent: independence implies $ρ = 0$ (when the moments exist), but zero correlation only kills linear association. A symmetric banana-shaped cloud can have $ρ = 0$ and still be completely dependent.

Correlation is not causation

Ice cream sales and drowning deaths rise together. A third variable — heat — drives both. The joint distribution of sales and drownings is real; the arrow from cones to casualties is not. Probability on observables cannot, by itself, name a mechanism. You need an assumption about how the world was generated: an experiment, an instrumental variable, a causal graph. Pearl’s book in Further Reading is the long form of this paragraph.