Let [math]v[/math] be a vector in [math]\mathbb{R}^3[/math]. We define the [b]orthogonal projection of [math]v[/math] onto a plane [math]W[/math][/b] through the origin, denoted by [math]\text{proj}_W v[/math], as follows:[br][br][math]\text{proj}_W v=w[/math], where [math]w[/math] is a vector in [math]W[/math] such that [math]v-w[/math] is in [math]W^\perp[/math].[br][br]The intuition for this definition can be illustrated by the applet below. [math]v[/math] is the black vector and the plane [math]W[/math] through the origin is spanned by the orthogonal basis [math]\left\{w_1,w_2\right\}[/math]. [br][br]In fact, [math]w=\text{proj}_W v[/math] can be explicitly computed as follows:[br][br][math]w=\frac{v\cdot w_1}{w_1\cdot w_1}w_1+\frac{v\cdot w_2}{w_2\cdot w_2}w_2[/math][br][br]We can verify that [math]v-w[/math] is in [math]W^\perp[/math]:[br][br][math](v-w)\cdot w_1=v\cdot w_1-\frac{v\cdot w_1}{w_1\cdot w_1}w_1\cdot w_1-\frac{v\cdot w_2}{w_2\cdot w_2}w_2 \cdot w_1=v\cdot w_1-v\cdot w_1=0[/math][br][math](v-w)\cdot w_2=v\cdot w_1-\frac{v\cdot w_1}{w_1\cdot w_1}w_1\cdot w_2-\frac{v\cdot w_2}{w_2\cdot w_2}w_2 \cdot w_2=v\cdot w_2-v\cdot w_2=0[/math]
In general, let [math]v[/math] be a vector in [math]\mathbb{R}^n[/math] and let [math]W[/math] be a subspace in [math]\mathbb{R}^n[/math] with an orthogonal basis [math]\left\{w_1,w_2,\ldots,w_p\right\}[/math]. Then we have the following theorem:[br][br][u]Orthogonal Decomposition Theorem[/u]: [math]v[/math] can be decomposed uniquely as the sum of two vectors[math]v=w+z[/math], where [math]w[/math] is a vector in [math]W[/math] and [math]z=v-w[/math] is a vector in [math]W^\perp[/math]. In fact, [math]w=\text{proj}_W v[/math] i.e. the orthogonal projection of [math]v[/math] onto [math]W[/math], which can be explicitly expressed as follows:[br][br][math]w=\frac{v\cdot w_1}{w_1\cdot w_1}w_1+\frac{v\cdot w_2}{w_2\cdot w_2}w_2+\cdots+\frac{v\cdot w_p}{w_p\cdot w_p}w_p[/math][br][br][u]Proof[/u]: Let [math]w[/math] be the vector in [math]W[/math] defined by the above formula. It suffices to show that [math]v-w[/math] is in [math]W^\perp[/math]. Similar to what we have done before, we have[br][br][math](v-w)\cdot w_j=v\cdot w_j - \frac{v\cdot w_j}{w_j\cdot w_j}w_j\cdot w_j=0[/math][br][br]for [math]j=1,2,\ldots,p[/math], which implies [math]v-w[/math] is in [math]W^{\perp}[/math].[br][br]As for the uniqueness, we let [math]v=w'+z'[/math] be another decomposition such that [math]w'[/math] is in [math]W[/math] and [math]z'[/math] is in [math]W^{\perp}[/math]. Then [math]w'+z'=w+z[/math], which implies [math]w'-w=z-z'[/math]. So [math]w'-w[/math] is in [math]W^{\perp}[/math]. But [math]w'-w[/math] is also in [math]W[/math]. Hence, [math]\|w'-w\|^2=(w'-w)\cdot(w'-w)=0[/math], which means [math]w'=w[/math]. Then [math]z'=z[/math] and the decomposition is thus unique.[br][br]
Let [math]w_1=\begin{pmatrix}2\\5\\-1\end{pmatrix}[/math] and [math]w_2=\begin{pmatrix}-4\\2\\2\end{pmatrix}[/math].[br][list=1][*]Show that [math]\left\{w_1,w_2\right\}[/math] is an orthogonal basis for [math]W=\text{Span}\left\{w_1,w_2\right\}[/math].[/*][*]Write [math]v=\begin{pmatrix}3\\4\\5\end{pmatrix}[/math] as the sum of a vector in [math]W[/math] and a vector in [math]W^\perp[/math].[/*][/list]
From the applet above, you can see that when the vector [math]v[/math] in [math]\mathbb{R}^3[/math] is written as the orthogonal decomposition [math]v=w+z[/math], where [math]w=\text{proj}_W v[/math], [math]\|z\|=\|v-w\|[/math] can be regarded as the [u]closest distance[/u] from the point at the arrowhead of [math]v[/math] to the plane [math]W[/math]. This intuition gives rise to the following theorem:[br][br][u]The Best Approximation Theorem[/u]: Let [math]W[/math] be a subspace of [math]\mathbb{R}^n[/math], let [math]v[/math] be any vector in [math]\mathbb{R}^n[/math]. Suppose [math]w=\text{proj}_W v[/math]. Then [math]w[/math] is the [b]closest point[/b] in [math]W[/math] to [math]v[/math] in the following sense:[br][br][math] \|v-w\|<\|v-u\|[/math] for any [math]u[/math] in [math]W[/math] distinct from [math]w[/math].[br][br]Proof: Let [math]u[/math] be any vector in [math]W[/math] distinct from [math]w[/math]. Then [math]w-u[/math] is a nonzero vector in [math]W[/math].[br][br]Consider [math]v-u=\left(v-w\right)+\left(w-u\right)[/math]. By definition, [math]v-w[/math] is in [math]W^{\perp}[/math] and [math]w-u[/math] is in [math]W[/math]. So [math]\left(v-w\right)\perp\left(w-u\right)[/math]. By Pythagorean theorem, we have[br][br][math]\|v-u\|^2=\|v-w\|^2+\|w-u\|^2[/math][br][br]Since [math]w-u[/math] is nonzero, we have [math]\|v-w\|^2<\|v-u\|^2[/math], which implies [math] \|v-w\|<\|v-u\|[/math].[br][br][br]This theorem is particularly useful when we discuss the least-squares method later.[br]