Chapter XII · 24 min

Likelihood and Models

Fitting a story to data

“A communication system is a system for transmitting information; inference is a system for recovering it.”

Having a model and some data, how do you turn the knobs? A clinic fitting a prevalence, a pollster fitting a turnout, a biologist fitting a capture–recapture count, and — among other trades — a learning algorithm fitting a network are running the same two procedures: maximise a likelihood, or maximise a posterior. This chapter names them. The theorems you now own are why any of it is more than a recipe.

Likelihood

A parametric model is a family of distributions with a knob (or a few knobs). The knob is the parameter $\theta$ — a coin’s bias, a Poisson’s rate, a line’s slope. Write $P(x\mid \theta )$ for “the chance of the data, if the knob is set to this value.” Having seen $x$, the likelihood is that same function, now viewed as a function of $\theta$. It is not a probability distribution on parameters — it need not even integrate to one in $\theta$. Chapter IV already used the word; here the knob may be continuous.

The maximum likelihood estimator is any maximiser

For i.i.d. observations — independent repeats of the same experiment — the joint likelihood multiplies, and one maximises the log to turn the product into a sum. A fair-looking coin that showed 7 heads in 10 flips has Bernoulli likelihood $p^7(1-p{)}^3$, maximised at $\hat{p}=0.7$.

MAP and the return of the prior

MAP is short for maximum a posteriori: pick the parameter that maximises the posterior instead of the likelihood alone.

The evidence $P(x)$ can be dropped because it does not depend on $\theta$. A prior that prefers small knobs is the Bayesian name for what some fitters call a penalty: taking logs, MAP for a bell-shaped prior on the knob is “likelihood minus a square of the knob.” Regularisation — shrinking wild estimates toward zero — is not a hack that probability is silent about. It is a prior, written in the loss.

Naïve Bayes, again

A classifier needs $P(y\mid x)$. Bayes’ theorem says

Naïve Bayes estimates the label prior from frequencies and estimates $P(x\mid y)$ as a product of per-feature distributions. Prediction is argmax of the product. The independence is false; the decision often works. You can now state, precisely, which assumption you are buying.

Cross-entropy is negative log-likelihood

For a discrete label, the log-likelihood of a model that emits probabilities $q_{\theta }(y\mid x)$ on a dataset is

up to a convention on the constant. Scoring a forecast with cross-entropy (Chapter XIII) is maximum likelihood under a categorical model — a model whose values are named types, not numbers. Calibration — whether a predicted 0.9 is right nine times in ten — is a separate question about the same object.

• A monsoon desk scoring rain/shine with a log-score: MLE for a two-type (categorical) likelihood.
• Fitting a line by least squares (Stranger in LA (Stranger in LA, X) ): MLE if the vertical scatter is modelled as a bell of constant width.
• Among other trades, a classifier that turns scores into chances that add to 1 (a “softmax”) and then uses cross-entropy is doing this same MLE.

Generative and discriminative

A generative classifier specifies $P(x,y)=P(x\mid y)P(y)$ and uses Bayes’ theorem at prediction time. Naïve Bayes, linear discriminant analysis, and most language models (as next-token distributions) are generative. A discriminative classifier specifies $P(y\mid x)$ directly — logistic regression, softmax networks — and never claims a density on $x$. Both are probability. They answer different questions. If you need to sample features, detect outliers, or handle missing inputs, you want a joint. If you only need a label, the conditional is the shorter story, and often the better fit, because it does not spend capacity on $P(x)$.

Logistic regression is the discriminative twin of a naïve Bayes with Gaussian (or Bernoulli) class-conditionals: the posterior as a function of $x$ is a sigmoid of a linear score. Same decision boundary family, different training objective.

Calibration and the evidence

A predicted probability is calibrated when, among all cases given score $q$, a fraction about $q$ are positive. Cross-entropy training encourages this but does not guarantee it, especially after a temperature or a stop-gradient. Reliability diagrams and proper scoring rules (log loss, Brier) are how you check. Accuracy is silent on calibration: a model that says 0.51 on every true item and 0.49 on every false one can be perfectly accurate and useless as a probability.

The evidence $P(x)=\int P(x\mid \theta )P(\theta )d\theta$ is the normaliser of Bayes’ theorem and a model-comparison score. A more flexible model can always fit, but it spreads prior mass thinner, and the evidence notices. This is Occam’s razor as arithmetic — MacKay’s book, in Further Reading, is the long form.

Overfitting is a confusion of sample and world

A model that interpolates the training set has made the empirical distribution certain. The true risk is an expectation under the world, not under the sample. When the hypothesis class is rich enough, the empirical minimum need not track the expectation — the LLN was stated for a fixed function, and you chose the function after seeing the data. Generalisation theory repairs this with uniform laws of large numbers. Regularisation, held-out sets, and cross-validation are the practical repairs. All of them are ways of remembering that the sample is one outcome in a sample space of datasets.