Chapter XIV · 24 min

Estimation

From a sample to a claim about the world

“The object of statistical methods is the reduction of data.”

A sample is one outcome in a space of datasets. An estimator is a function of that outcome, aimed at a property of the world: a mean, a weight vector, a whole distribution. This chapter makes the aim precise — bias, variance, consistency — and then returns to maximum likelihood as a method with theorems, not just a slogan. Generalisation, in the last pages, is estimation of risk.

Estimators

Let $X_1,\dots ,X_n$ be i.i.d. from a distribution indexed by $\theta$, and let $g(\theta )$ be a quantity you care about (often $g(\theta )=\theta$ itself). An estimator is a random variable $\hat{g}=T(X_1,\dots ,X_n)$. It is a function of the sample, hence random; the thing it estimates is not.

• The sample mean ${\bar{X}}_n$ estimates $\mathbb{E}[X]$.
• The empirical risk $\hat{R}(f)$ estimates the true risk $R(f)$.
• A trained network $f_{\hat{\theta }}$ is an estimator of a prediction function — a much larger $g$.

Bias, variance, mean squared error

The bias is the systematic error, $\mathrm{Bias}(\hat{g})=\mathbb{E}[\hat{g}]-g(\theta )$. Unbiased means the expectation hits the target; it does not mean any one sample is close. The variance is $\mathrm{Var}(\hat{g})$. They combine in the mean squared error:

This is the same algebra as Chapter 7, now read as a design tradeoff. A more flexible model class lowers bias (you can hit more targets) and typically raises variance (the sample’s idiosyncrasies get fit). Regularisation, early stopping, and smaller architectures are variance-control. The popular “bias–variance tradeoff” in learning is this identity, applied to a predictor’s risk rather than to a scalar parameter.

Consistency

An estimator is consistent for $g(\theta )$ if $\hat{g}\to g(\theta )$ in probability as $n\to \infty$. The sample mean is consistent for the mean (LLN). Consistency is a large-sample promise: with enough data, the estimator forgets its start. It says nothing, by itself, about $n=40$.

Maximum likelihood, more carefully

Under regularity (the support does not depend on $\theta$, the log-likelihood is differentiable, identifiability, …), the MLE is consistent, asymptotically unbiased, and asymptotically normal:

where $I(\theta )$ is the Fisher information,

Information, here, is the expected curvature of the log-likelihood. A sharp peak (large $I$) means the data distinguish nearby parameters well, so the MLE’s variance is small. The Cramér–Rao bound says that, among unbiased estimators, you cannot beat $1/(nI(\theta ))$ in variance. MLE attains this bound asymptotically. That is the precise sense in which “likelihood is efficient”.

MAP, regularisation, and misspecification

MAP is MLE with a prior. As $n$ grows, a continuous prior is usually washed out and MAP and MLE agree. At small $n$, the prior is doing the variance-control of the MSE decomposition. Weight decay is a Gaussian prior; an ${\ell }_1$ penalty is a Laplace prior. The vocabulary is interchangeable.

All of the efficiency theorems assume the model is well specified: some $\theta$ really generated the data. If not, MLE still minimises KL to the nearest member of the family (Chapter 13). That nearest member may be a useful lie, or a disaster. Model checking is the act of asking which.

Estimating risk: holdout and concentration

For a fixed predictor $f$, the empirical risk on an i.i.d. test set of size $n$ is an unbiased estimator of true risk, with variance shrinking as $1/n$. For bounded losses, Hoeffding’s inequality gives a tail:

(for loss in $[0,1]$). This is why a test set of a few thousand examples already makes accuracy a stable number — for one fixed $f$. If you selected $f$ by looking at the same data, the estimator is no longer unbiased for the selected model’s risk. That is the reason for a held-out set, and the reason a training loss is not a reportable number.