Point Estimation

Summary

Point estimation uses a sample statistic to infer the value of an unknown population parameter. An estimator is the rule viewed as a random variable before data are collected; an estimate is the numerical value obtained after sampling. Key properties are unbiasedness, consistency, and efficiency, and common construction methods include the method of moments and maximum likelihood.^[1]

Prerequisites

Definition / Notation

Let $X_{1}, \dots, X_{n}$ be a random sample from a distribution that depends on an unknown parameter $θ$ .

Symbol	Meaning
$θ$	Unknown population parameter
${\hat{θ}}_{n}$	Estimator of $θ$ , a statistic $T (X_{1}, \dots, X_{n})$
${\hat{θ}}_{n, obs}$	Estimate, the observed value of the estimator
$Bias ({\hat{θ}}_{n})$	$E [{\hat{θ}}_{n}] - θ$
$MSE ({\hat{θ}}_{n})$	$E [({\hat{θ}}_{n} - θ)^{2}]$

The mean squared error decomposes as:

MSE ({\hat{θ}}_{n}) = Var ({\hat{θ}}_{n}) + Bias ({\hat{θ}}_{n})^{2}

Parameters / Assumptions

The sample $X_{1}, \dots, X_{n}$ is independent and identically distributed (i.i.d.).
The parameter space is known and the model is identifiable.
The estimator has finite first and second moments.

Essential Result

A good estimator should satisfy one or more of the following:

Unbiasedness: $E [{\hat{θ}}_{n}] = θ$ .
Consistency: ${\hat{θ}}_{n} \overset{P}{\to} θ$ as $n \to \infty$ .
Efficiency: among unbiased estimators, ${\hat{θ}}_{n}$ has the smallest possible variance; under regularity conditions the variance attains the Cramér–Rao lower bound.
Method of moments: equate sample moments to population moments and solve for the parameters.
Maximum likelihood: choose the parameter value that maximizes the likelihood of the observed sample.^[2]

Worked Example

Suppose $X_{1}, \dots, X_{n} \sim Bernoulli (p)$ and we use $\hat{p} = \bar{X}$ . Then

E [\hat{p}] = p, Var (\hat{p}) = \frac{p (1 - p)}{n} .

Thus $\hat{p}$ is unbiased. By Chebyshev's inequality,

P (| \bar{X} - p | \geq ε) \leq \frac{p (1 - p)}{n ε^{2}} \to 0,

so $\hat{p}$ is also consistent.

Common Mistakes

Calling an observed number an estimator; it is an estimate.
Thinking that unbiasedness alone guarantees a small error; variance matters too.
Forgetting that consistency is a property of a sequence of estimators as $n$ grows.
Applying the method of moments when the required moments do not exist.

Connections

References

OpenStax, Introductory Statistics, "Point Estimation", https://openstax.org/details/books/introductory-statistics ↩︎
MIT OCW, Introduction to Probability and Statistics, "Estimation", https://ocw.mit.edu/courses/mathematics/ ↩︎