Skip to content
Geometry and Algebra

Inner product and orthogonality: projections, Gram-Schmidt and the spectral theorem

Inner product and norm, Pythagoras, orthogonal projection and least squares, orthonormal bases, the Gram-Schmidt process, orthogonal matrices and the spectral theorem.

5 min read 0 comments

Abstract vector spaces have neither lengths nor angles. To get them you need one more structure, the inner product. With it come orthogonality, projections, orthonormal bases and one of the finest theorems of the course, the spectral theorem. These are the tools behind least squares, Fourier series and JPEG compression.

Inner product and norm

An inner product on a real space V is an operation \langle\mathbf u, \mathbf v\rangle that gives a number and is symmetric, linear in each argument and positive definite: \langle\mathbf v, \mathbf v\rangle \gt 0 for every \mathbf v \ne \mathbf 0. The standard one on \mathbb R^n is \mathbf u\cdot\mathbf v = \sum u_i v_i. On continuous functions on [a, b] an inner product is \langle f, g\rangle = \int_a^b f(x)g(x)\,dx.

The norm is \|\mathbf v\| = \sqrt{\langle\mathbf v, \mathbf v\rangle}, and the Cauchy-Schwarz and triangle inequalities hold (proved in the lesson on vectors, with an argument that uses only these properties). Two vectors are orthogonal if \langle\mathbf u, \mathbf v\rangle = 0.

Pythagoras’ theorem

If \mathbf u\perp\mathbf v, then \|\mathbf u + \mathbf v\|^2 = \langle\mathbf u + \mathbf v, \mathbf u + \mathbf v\rangle = \|\mathbf u\|^2 + 2\langle\mathbf u, \mathbf v\rangle + \|\mathbf v\|^2 = \|\mathbf u\|^2 + \|\mathbf v\|^2. In any space with an inner product, even one made of functions.

Non-zero orthogonal vectors are independent. If \sum\lambda_i\mathbf v_i = \mathbf 0, taking the inner product with \mathbf v_j leaves only \lambda_j\|\mathbf v_j\|^2 = 0, so \lambda_j = 0.

Orthogonal projection

Orthogonal projection of a vector v onto the line spanned by u
The projection of \mathbf v onto the direction of \mathbf u is the vector on that line closest to \mathbf v. The difference \mathbf v - \operatorname{proj}_{\mathbf u}\mathbf v is perpendicular to \mathbf u.

Proof of the formula

We look for the multiple c\,\mathbf u such that \mathbf v - c\,\mathbf u is orthogonal to \mathbf u:

    \[ \langle\mathbf v - c\,\mathbf u, \mathbf u\rangle = 0 \quad\Rightarrow\quad c = \frac{\langle\mathbf v, \mathbf u\rangle}{\langle\mathbf u, \mathbf u\rangle},\qquad \operatorname{proj}_{\mathbf u}\mathbf v = \frac{\langle\mathbf v, \mathbf u\rangle}{\langle\mathbf u, \mathbf u\rangle}\,\mathbf u \]

It is also the closest point: for any other multiple d\,\mathbf u, the vector \mathbf v - d\,\mathbf u is the sum of \mathbf v - c\,\mathbf u and (c - d)\,\mathbf u, which are orthogonal, and by Pythagoras \|\mathbf v - d\,\mathbf u\|^2 = \|\mathbf v - c\,\mathbf u\|^2 + (c - d)^2\|\mathbf u\|^2 \ge \|\mathbf v - c\,\mathbf u\|^2.

The same holds for projection onto a subspace W: the point of W closest to \mathbf v is the one for which the difference is orthogonal to all of W. This is the principle of least squares: when a system A\mathbf x = \mathbf b has no solution (too many equations, noisy data), you look for \mathbf x such that A\mathbf x is the projection of \mathbf b onto the column space. The orthogonality condition gives the normal equations A^TA\,\mathbf x = A^T\mathbf b. This is how a regression line through experimental data is computed.

Orthonormal bases

A basis is orthonormal if its vectors have norm 1 and are pairwise orthogonal. The advantage is huge: coordinates are computed with an inner product, without solving systems.

    \[ \mathbf v = \sum_i\langle\mathbf v, \mathbf e_i\rangle\,\mathbf e_i \]

Indeed, taking the inner product of \mathbf v = \sum c_i\mathbf e_i with \mathbf e_j leaves only c_j. A Fourier series is exactly this, in a space of functions: sines and cosines are an orthogonal basis with respect to \int f g, and the Fourier coefficients are inner products.

The Gram-Schmidt process

From any basis \mathbf v_1, \dots, \mathbf v_n you build an orthonormal basis of the same space by removing from each vector its projections onto the previous ones:

    \[ \mathbf u_1 = \mathbf v_1,\qquad \mathbf u_k = \mathbf v_k - \sum_{j \lt k}\frac{\langle\mathbf v_k, \mathbf u_j\rangle}{\langle\mathbf u_j, \mathbf u_j\rangle}\,\mathbf u_j,\qquad \mathbf e_k = \frac{\mathbf u_k}{\|\mathbf u_k\|} \]

Example. \mathbf v_1 = (1, 1, 0), \mathbf v_2 = (1, 0, 1). \mathbf u_1 = (1, 1, 0). \langle\mathbf v_2, \mathbf u_1\rangle = 1, \langle\mathbf u_1, \mathbf u_1\rangle = 2, so \mathbf u_2 = (1, 0, 1) - \tfrac12(1, 1, 0) = (\tfrac12, -\tfrac12, 1). Check: \mathbf u_1\cdot\mathbf u_2 = \tfrac12 - \tfrac12 + 0 = 0. Normalising: \mathbf e_1 = \tfrac{1}{\sqrt2}(1, 1, 0), \mathbf e_2 = \tfrac{1}{\sqrt6}(1, -1, 2).

Orthogonal matrices

A square matrix Q is orthogonal if its columns are an orthonormal basis, that is if Q^TQ = I. Then Q^{-1} = Q^T: the inverse costs as much as a transpose. Orthogonal matrices preserve inner products, lengths and angles, \langle Q\mathbf u, Q\mathbf v\rangle = \mathbf u^TQ^TQ\mathbf v = \langle\mathbf u, \mathbf v\rangle: they are the rotations (\det Q = 1) and the reflections (\det Q = -1).

The spectral theorem

Theorem. Every real symmetric matrix has real eigenvalues and can be diagonalised with an orthonormal basis of eigenvectors: there is an orthogonal Q such that

    \[ Q^TAQ = D \]

A key step of the proof is easy to see: eigenvectors of distinct eigenvalues are orthogonal. If A\mathbf u = \lambda\mathbf u and A\mathbf v = \mu\mathbf v, using A = A^T:

    \[ \lambda\langle\mathbf u, \mathbf v\rangle = \langle A\mathbf u, \mathbf v\rangle = \langle\mathbf u, A\mathbf v\rangle = \mu\langle\mathbf u, \mathbf v\rangle \]

so (\lambda - \mu)\langle\mathbf u, \mathbf v\rangle = 0 and, if \lambda \ne \mu, \langle\mathbf u, \mathbf v\rangle = 0. You can see it in the matrix \begin{pmatrix}2 & 1\\ 1 & 2\end{pmatrix} from the lesson on eigenvalues: its eigenvectors (1, 1) and (1, -1) are perpendicular.

The spectral theorem has consequences everywhere: the principal stress directions in a beam, the principal axes of inertia of a rigid body, principal component analysis (PCA), which reduces data to the directions of greatest variance, and the classification of conics and quadrics that we will see in the lesson on plane geometry.

Related articles

Subscribe
Notify of
guest
0 Comments
Oldest
Newest Most Voted
0
Would love your thoughts, please comment.x
()
x