Chapter XII · 24 min

Likelihood and Models

Fitting a story to data

“A communication system is a system for transmitting information; inference is a system for recovering it.”
— After Claude Shannon, A Mathematical Theory of Communication (1948)

Having a model and some data, how do you turn the knobs? Scientists fitting a constant, pollsters fitting a turnout, doctors fitting a risk score, and learning algorithms fitting a network are running the same two procedures: maximise a likelihood, or maximise a posterior. This chapter names them. The theorems you now own — Bayes, linearity, concentration — are why any of it is more than a recipe.

Likelihood

A parametric model is a family $P (x ∣ θ)$ of distributions on data, indexed by a parameter. Having seen $x$ , the likelihood is that same function, now viewed as a function of $θ$ . It is not a probability distribution on parameters — it need not even integrate to one in $θ$ .

The maximum likelihood estimator is any maximiser

\hat{θ}_{MLE} (x) \in ar g θ max P (x ∣ θ) .

For i.i.d. observations the joint likelihood multiplies, and one maximises the log to turn the product into a sum. A fair-looking coin that showed 7 heads in 10 flips has Bernoulli likelihood $p^{7} (1 - p)^{3}$ , maximised at $\overset{p}{^} = 0.7$ .

Bernoulli likelihood after 7 heads and 3 tails. The curve is a function of p, peaked at the observed frequency. MLE reads off that peak.

MAP and the return of the prior

Maximum a posteriori estimation maximises the posterior instead:

\hat{θ}_{MAP} (x) \in ar g θ max P (x ∣ θ) P (θ) .

The evidence $P (x)$ can be dropped because it does not depend on $θ$ . A prior that prefers small weights is the Bayesian name for what optimisation people call an $ℓ_{2}$ penalty: taking logs, MAP for a Gaussian prior is likelihood minus a squared-norm regulariser. Regularisation is not a hack that probability is silent about. It is a prior, written in the loss.

Naïve Bayes, again

A classifier needs $P (y ∣ x)$ . Bayes’ theorem says

P (y ∣ x) \propto P (x ∣ y) P (y) .

Naïve Bayes estimates the label prior from frequencies and estimates $P (x ∣ y)$ as a product of per-feature distributions. Prediction is argmax of the product. The independence is false; the decision often works. You can now state, precisely, which assumption you are buying.

Cross-entropy is negative log-likelihood

For a discrete label, the log-likelihood of a model that emits probabilities $q_{θ} (y ∣ x)$ on a dataset is

i \sum lo g q_{θ} (y_{i} ∣ x_{i}) = - i \sum H (δ_{y_{i}}, q_{θ} (\cdot ∣ x_{i})),

up to a convention on the constant. Training with cross-entropy is maximum likelihood under a categorical model. Calibration — whether a predicted 0.9 is right nine times in ten — is a separate, probabilistic question about the same object.

Classification with softmax + cross-entropy: MLE for a categorical likelihood.
Linear regression with squared loss: MLE for Gaussian noise of constant variance.
Dropout, data augmentation, weight decay: extra randomness or extra priors, i.e. extra $P$ .

Generative and discriminative

A generative classifier specifies $P (x, y) = P (x ∣ y) P (y)$ and uses Bayes’ theorem at prediction time. Naïve Bayes, linear discriminant analysis, and most language models (as next-token distributions) are generative. A discriminative classifier specifies $P (y ∣ x)$ directly — logistic regression, softmax networks — and never claims a density on $x$ . Both are probability. They answer different questions. If you need to sample features, detect outliers, or handle missing inputs, you want a joint. If you only need a label, the conditional is the shorter story, and often the better fit, because it does not spend capacity on $P (x)$ .

Logistic regression is the discriminative twin of a naïve Bayes with Gaussian (or Bernoulli) class-conditionals: the posterior as a function of $x$ is a sigmoid of a linear score. Same decision boundary family, different training objective.

Calibration and the evidence

A predicted probability is calibrated when, among all cases given score $q$ , a fraction about $q$ are positive. Cross-entropy training encourages this but does not guarantee it, especially after a temperature or a stop-gradient. Reliability diagrams and proper scoring rules (log loss, Brier) are how you check. Accuracy is silent on calibration: a model that says 0.51 on every true item and 0.49 on every false one can be perfectly accurate and useless as a probability.

The evidence $P (x) = \int P (x ∣ θ) P (θ) d θ$ is the normaliser of Bayes’ theorem and a model-comparison score. A more flexible model can always fit, but it spreads prior mass thinner, and the evidence notices. This is Occam’s razor as arithmetic — MacKay’s book, in Further Reading, is the long form.

Overfitting is a confusion of sample and world

A model that interpolates the training set has made the empirical distribution certain. The true risk is an expectation under the world, not under the sample. When the hypothesis class is rich enough, the empirical minimum need not track the expectation — the LLN was stated for a fixed function, and you chose the function after seeing the data. Generalisation theory repairs this with uniform laws of large numbers. Regularisation, held-out sets, and cross-validation are the practical repairs. All of them are ways of remembering that the sample is one outcome in a sample space of datasets.