Maximum Likelihood Estimation

Summary

Maximum likelihood estimation chooses the parameter value that makes the observed data most probable under the assumed model. The likelihood function has the same functional form as the joint density or mass function, but is viewed as a function of the parameter. Maximizing the log-likelihood is usually easier and gives the same estimate.^[1]

Prerequisites

• Point Estimation
• Probability
• Bernoulli Distribution
• Normal Distribution

Definition / Notation

For a random sample $X_1,\dots ,X_n$ with density or mass function $f(x;\theta )$ :

The MLE is

Parameters / Assumptions

• The data are i.i.d. from a parametric family $f(x;\theta )$ .
• The parameter space is identifiable.
• The support of $f$ does not depend on $\theta$ in the regular case, and the log-likelihood is differentiable at the maximum.
• Boundary maxima must be checked separately.

Essential Result

• For $X_i\sim \mathrm{Bernoulli}(p)$ , $\ell (p)=\sum _i[x_i\ln p+(1-x_i)\ln (1-p)]$ . Solving $U(p)=0$ gives $\hat{p}=\overline{x}$ .
• For $X_i\sim N(\mu ,{\sigma }^2)$ , the MLEs are $\hat{\mu }=\overline{x}$ and ${\hat{\sigma }}^2=\frac{1}{n}\sum _{i=1}^n(x_i-\overline{x}{)}^2$ .
• Under regularity conditions, the MLE is consistent and asymptotically normal:

This justifies approximate confidence intervals and tests.^[2]

Worked Example

A coin is tossed 20 times and shows 7 heads. Model the outcomes as $\mathrm{Bernoulli}(p)$ . The log-likelihood is

Setting ${\ell }^{\prime }(p)=7/p-13/(1-p)=0$ yields $\hat{p}=7/20=0.35$ . The second derivative is negative, confirming a maximum.

Common Mistakes

• Treating the likelihood as a probability distribution over the parameter.
• Using $n-1$ instead of $n$ in the normal variance MLE.
• Ignoring the boundary of the parameter space.
• Trusting the asymptotic normal approximation in very small samples.

Connections

• Point Estimation
• Confidence Intervals
• Central Limit Theorem

References