6  Orthogonal Projections and Complements

“It is frequent in mathematics that every instance of a concept of seemingly great generality is in essence the same as a small and concrete special case.”
– Paul Halmos

6.1 Orthogonal Projections

Orthogonal Projections
  1. If a vector x is orthogonal to every vector in subspace W of \mathbb{R}^n, then x is said to be orthogonal to the subspace W.
  2. The orthogonal projection of \mathbf{v} \in V onto the subspace W \subset V is the element \mathbf{v}_W that makes the difference \mathbf{z}=\mathbf{v}-\mathbf{v}_W orthogonal to W.
Theorem: Orthogonal Projections wrt Orthonormal Basis

Let \mathbf{u}_1,\mathbf{u}_2, \cdots, \mathbf{u}_n be an orthonormal basis for a subspace W \subset V. Then the orthogonal projection of \mathbf{v} \in V onto \mathbf{v}_W \in W is given by \mathbf{v}_W= c_1\mathbf{u}_1+c_2\mathbf{u}_2+ \cdots + c_n\mathbf{u}_n, where c_i= \langle \mathbf{v}, \mathbf{u}_i \rangle.

Exercise 6.1 Find the orthogonal projection of \mathbf{v}=\begin{bmatrix} 1 & 2 & -1 & 2 \end{bmatrix}^T onto the following subspaces:

  1. The span of \{\begin{bmatrix} 1 & -1 & 2 & 1 \end{bmatrix}^T, \begin{bmatrix} 2 & 1 & 0 & -1 \end{bmatrix}^T\}
  2. The subspace orthogonal to \begin{bmatrix} 1 & -1 & 0 & 1 \end{bmatrix}^T

6.2 Orthogonal Subspaces and Orthogonal Complements

Orthogonal Complements
  1. Two subspaces U,W \subset V are said to be orthogonal if every vector \mathbf{v} \in V is orthogonal to every vector \mathbf{w} \in W.
  2. The set of all vectors of V that are orthogonal to the subspace W \subset V is said to be the orthogonal complement of W, and is written as W^{\perp}.

Exercise 6.2  

  1. Given that W is a subspace of V, show that the subset W^{\perp} \subset V is a subspace of V.
  2. Show that W^{\perp} \cap W =\{0\}.

Let W be a finite dimensional subspace of (inner product space)V. Then:

  1. Every vector \mathbf{v} \in V can be uniquely decomposed as \mathbf{v} = \mathbf{v}_W + \mathbf{z}, where \mathbf{v}_W \in W and \mathbf{z} \in W^\perp.
  2. (W^\perp)^\perp=W.

These results are false in infinite dimensions.

6.3 The Four Fundamenental Subspaces

The Four Fundamental Subspaces (Fundamental Theorem of Linear Algebra)

For a m \times n matrix \mathbf{A}, {(\mathcal{C}(\mathbf{A}))}^{\perp}= \mathcal{N} (\mathbf{A}^T) & {(\mathcal{C}(\mathbf{A}^T))}^{\perp}= \mathcal{N} (\mathbf{A})

(The Fredholm Alternative) A linear system \mathbf{Ax}=\mathbf{b} has a solution iff \mathbf{b} \perp \mathbf{y} for every \mathbf{y} \in \mathcal{N} (\mathbf{A}^T).