SVD is closely related to the diagonalization of symmetric matrices.[br][br]Given any [math]m\times n[/math] matrix [math]A[/math]. We consider [math]A^TA[/math], which is an [math]n\times n[/math] symmetric matrix (Note: [math](A^TA)^T=A^T(A^T)^T=A^TA[/math]). Then we can diagonalize it as follows:[br][br][math]A^TA=VDV^T[/math], where [math]V[/math] is an [math]n\times n[/math] orthogonal matrix and [math]D[/math] is an [math]n\times n[/math] diagonal matrix. We denote the diagonal entries of [math]D[/math] by [math]\lambda_i[/math] and arrange them in descending order and the column vectors of [math]V[/math] are [math]\left\{v_1,v_2,\ldots,v_n\right\}[/math], which form an orthonormal basis because [math]V[/math] is orthogonal.[br][br]Notice that for any [math]v_i, v_j[/math],[br][br][math]Av_i \cdot Av_j=(Av_i)^T(Av_j)=v_i^TA^TAv_j=v_i^T(\lambda_jv_j)=\lambda_j(v_i\cdot v_j)[/math][br][br]Therefore, [math]\left\{Av_1,Av_2,\ldots,Av_n\right\}[/math] is orthogonal. Moreover, when [math]i=j[/math], we have [math]\|Av_i\|^2=\lambda_i[/math], which means all eigenvalues [math]\lambda_i[/math] are nonnegative. Suppose [math]A\ne 0[/math]. Let [math]\lambda_1\geq \lambda_2\geq \cdots \lambda_r>0[/math] and [math]\lambda_i=0[/math] for [math]i>r[/math]. Then we normalize [math]Av_i[/math] for [math]i=1,\ldots,r[/math] [br][br][math]u_i=\frac{Av_i}{\|Av_i\|}=\frac 1{\sqrt{\lambda_i}}Av_i[/math], where [math]i=1,\ldots,r[/math]. [br][br]Hence [math]\left\{u_1,u_2,\ldots,u_r\right\}[/math] is an orthonormal set. Extend it to [math]\left\{u_1,u_2,\ldots,u_r, u_{r+1},\ldots, u_m\right\}[/math] to form an orthonormal basis for [math]\mathbb{R}^m[/math] if [math] k < m[/math].[br][br]Define [math] \sigma_i = \sqrt{\lambda_i}[/math] for [math]i=1,...,r[/math] and [math]\sigma_i=0[/math] for [math]i=1+r,...,m[/math]. Hence [math]Av_i=\sigma_iu_i[/math] for [math]i=1,\ldots,m[/math] and we have[br][br][math]U\Sigma = AV[/math], where [math]U[/math] is the [math]m\times m[/math] orthogonal matrix formed by [math]\left\{u_1,u_2,\ldots, u_m\right\}[/math] and [math]\Sigma[/math] is the [math]m\times n[/math] diagonal matrix with diagonal entries [math] \sigma_i[/math].[br][br]Hence we obtain [math]A=U\Sigma V^T[/math] i.e. the SVD of [math]A[/math].[br]

From the above construction, we have the following four [b]fundamental subspaces[/b]:[br][br][math]\text{Col}(A)=\left\{u_1,\ldots,u_r\right\}[/math][br][math](\text{Col}(A))^\perp=\left\{u_{r+1},\ldots,u_m\right\}[/math][br][math]\text{Nul}(A)=\left\{v_{r+1},\ldots,v_n\right\}[/math][br][math](\text{Nul}(A))^{\perp}=\left\{v_1,\ldots,v_r\right\}[/math]