Simple Linear Regression

Summary

Simple linear regression models the mean response of a continuous outcome $Y$ as a linear function of a single predictor $x$ . The coefficients are usually estimated by ordinary least squares, which minimizes the sum of squared residuals. Under standard assumptions, these estimators are the best linear unbiased estimators, and normal-theory inference applies to the slopes.^[1]

Prerequisites

Definition / Notation

The model is

Y_{i} = β_{0} + β_{1} x_{i} + ε_{i}, i = 1, \dots, n .

Symbol	Meaning
$Y_{i}$	Response variable
$x_{i}$	Predictor, treated as fixed or conditioned upon
$β_{0}, β_{1}$	Intercept and slope parameters
$ε_{i}$	Random error
${\hat{Y}}_{i}$	Fitted value, ${\hat{β}}_{0} + {\hat{β}}_{1} x_{i}$
$e_{i}$	Residual, $Y_{i} - {\hat{Y}}_{i}$
$R^{2}$	Coefficient of determination

The least-squares estimates are

{\hat{β}}_{1} = \frac{\sum_{i = 1}^{n} (x_{i} - \bar{x}) (Y_{i} - \bar{Y})}{\sum_{i = 1}^{n} (x_{i} - \bar{x})^{2}}, {\hat{β}}_{0} = \bar{Y} - {\hat{β}}_{1} \bar{x} .

Parameters / Assumptions

The predictor values $x_{i}$ are fixed or conditioned upon.
Errors have mean zero, constant variance $σ^{2}$ , and are uncorrelated (Gauss-Markov assumptions).
For inference on slopes, errors are additionally assumed i.i.d. normal.
The predictor is not constant, so $\sum (x_{i} - \bar{x})^{2} > 0$ .

Essential Result

The Gauss-Markov theorem states that, under the assumptions above, the least-squares estimators are the best linear unbiased estimators (BLUE). Under normality,

{\hat{β}}_{1} \sim N (β_{1}, \frac{σ^{2}}{S_{x x}}),

where $S_{x x} = \sum_{i = 1}^{n} (x_{i} - \bar{x})^{2}$ . In practice $σ^{2}$ is estimated by $s^{2} = \frac{\sum e_{i}^{2}}{n - 2}$ , and

t = \frac{{\hat{β}}_{1} - β_{1, 0}}{s / \sqrt{S_{x x}}} \sim t_{n - 2}

under $H_{0} : β_{1} = β_{1, 0}$ . The coefficient of determination is

R^{2} = 1 - \frac{\sum e_{i}^{2}}{\sum (Y_{i} - \bar{Y})^{2}} .

Worked Example

For data $x = (1, 2, 3, 4)$ and $y = (2, 4, 5, 7)$ :

\bar{x} = 2.5, \bar{y} = 4.5, S_{x x} = 5, S_{x y} = 7.

Thus

{\hat{β}}_{1} = \frac{7}{5} = 1.4, {\hat{β}}_{0} = 4.5 - 1.4 (2.5) = 1.0 .

The fitted line is $\hat{y} = 1.0 + 1.4 x$ .^[2]

Common Mistakes

Confusing correlation with causation.
Extrapolating beyond the observed range of $x$ .
Ignoring non-constant variance, outliers, or influential points.
Assuming a statistically significant slope implies a large practical effect.

Connections

References

OpenStax, Introductory Statistics, "Linear Regression", https://openstax.org/details/books/introductory-statistics ↩︎
MIT OCW, Introduction to Probability and Statistics, "Simple Linear Regression", https://ocw.mit.edu/courses/mathematics/ ↩︎