Orthogonality and Projections

Summary

Orthogonality means a zero inner product. Orthonormal bases make coordinates and projections simple. Orthogonal projections solve nearest-point problems and are the geometric engine of least squares.

Prerequisites

Vector Spaces and Bases, Vectors and Dot Product. Hub: Linear Algebra.

Definition / Statement

Vectors $\mathbf{u},\mathbf{v}$ in ${\mathbb{R}}^n$ are orthogonal if $\mathbf{u}\cdot \mathbf{v}=0$ . A set is orthogonal if every pair of distinct vectors is orthogonal; it is orthonormal if it is orthogonal and each vector has length $1$ .

A subspace $W\subseteq {\mathbb{R}}^n$ has orthogonal complement

The orthogonal projection of $\mathbf{b}$ onto $W$ is the unique $\hat{\mathbf{b}}\in W$ such that $\mathbf{b}-\hat{\mathbf{b}}\in W^{\perp }$ .

Objects and Dimensions

Notation

Conditions / Assumptions

• Standard Euclidean inner product on ${\mathbb{R}}^n$ .
• Projection onto $\mathrm{span}\{\mathbf{a}\}$ with $\mathbf{a}\ne \mathbf{0}$ uses the formula below.
• For a matrix $A$ with full column rank, the projection onto $\mathrm{Col}(A)$ uses the normal equations (see Least Squares and QR).

Matrix / Vector Form

Projection onto a line spanned by $\mathbf{a}\ne \mathbf{0}$ :

Projection onto $\mathrm{Col}(A)$ when $A$ has linearly independent columns:

The matrix $P=A(A^{T}A{)}^{-1}{A}^T$ is the orthogonal projection matrix onto $\mathrm{Col}(A)$ : $P^T=P$ and $P^2=P$ .

Orthonormal columns $Q$ : projection simplifies to

Pythagorean theorem: if $\mathbf{u}\perp \mathbf{v}$ , then $\parallel \mathbf{u}+\mathbf{v}{\parallel }^2=\parallel \mathbf{u}{\parallel }^2+\parallel \mathbf{v}{\parallel }^2$ .

Geometric Interpretation

• $\hat{\mathbf{b}}$ is the closest point in $W$ to $\mathbf{b}$ .
• Residual $\mathbf{b}-\hat{\mathbf{b}}$ is perpendicular to every vector in $W$ .
• Orthonormal frames measure lengths of coordinates by the ordinary Euclidean norm of the coefficient vector.

Worked Example

Project $\mathbf{b}=⟨3,4⟩$ onto $\mathbf{a}=⟨1,0⟩$ :

Common Mistakes

• Using ${\mathbf{a}}^T\mathbf{b}/\parallel \mathbf{a}\parallel$ instead of ${\mathbf{a}}^T\mathbf{b}/\parallel \mathbf{a}{\parallel }^2$ (or ${\mathbf{a}}^T\mathbf{a}$ ).
• Forgetting that $Q{Q}^T$ is the projection only when columns of $Q$ are orthonormal (for orthogonal but not unit columns, scale correctly).
• Confusing $Q^{T}Q=I_k$ (always for orthonormal columns) with $Q{Q}^T=I_n$ (only if $k=n$ ).

Connections

• Related: Vectors and Dot Product, Vector Spaces and Bases, Least Squares and QR
• Next: Least Squares and QR
• Spectral: real symmetric matrices have orthonormal eigenbases (Eigenvalues and Eigenvectors)

References

Orthogonality and projections follow MIT 18.06.^[1]