Visualizing linear projection
Introduction
The projection of one vector onto another is the geometry behind least squares: the fitted values in a regression of \(y\) on a single regressor \(b\) are the projection of \(y\) onto the line spanned by \(b\), and the residual is what the projection leaves over. The derivation is three lines long, and all of it is ordinary algebra except for one geometric input, namely that the residual meets the regressor at a right angle and that a right angle means a dot product of zero. This post states the derivation, illustrates it with two interactive figures, and then removes the geometric input altogether: minimizing the squared error returns the same coefficient without a picture, and orthogonality drops out as a consequence rather than an assumption.
What the projection is allowed to be
Let \(b\) and \(y\) be vectors in \(\mathbb{R}^2\) with \(b \neq 0\). The projection \(p\) of \(y\) onto the line spanned by \(b\) has to lie on that line, and every point on that line is a multiple of \(b\). Hence
\[ p = \hat{x} b \]
for some scalar \(\hat{x}\). That is the whole content of projecting onto a line: one unknown number rather than two, which is why the answer comes out as a scalar and not as a vector. The first figure shows the situation. Dragging the tip of \(b\) changes the line that \(y\) is projected on; dragging the tip of \(y\) changes the point being projected.
Figure 1. The line through \(b\) is dashed grey, the projection \(p\) green, and the error \(e = y - p\) dashed orange. Drag the circles at the tips of \(b\) and \(y\).
What pins down which multiple
The error is \(e = y - p\), drawn as the dashed orange vector. The projection is the closest point on the line to \(y\), and in the figure that closest point is exactly where the error meets the line at a right angle. Orthogonality of \(b\) and \(e\) is, in dot-product notation,
\[ b^{T} e = 0 . \]
That single condition is enough to determine \(\hat{x}\). Substituting \(e = y - \hat{x} b\) gives
\[ b^{T}(y - \hat{x} b) = 0 , \]
and expanding by linearity of the dot product,
\[ b^{T} y - \hat{x} \, (b^{T} b) = 0 . \]
Both \(b^{T} y\) and \(b^{T} b\) are numbers and \(\hat{x}\) is a number, so what remains is ordinary algebra:
\[ \hat{x} = \frac{b^{T} y}{b^{T} b} . \]
Everything in the derivation hinges on the geometric fact \(b^{T} e = 0\); the rest is rearrangement. The fourth readout in Figure 1 reports \(b^{T} e\), and it stays at zero however the two vectors are dragged.
Reading the formula
Two features of the result are worth noting. Substituting \(\hat{x}\) back into \(p\) gives
\[ p = b \hat{x} = \frac{b (b^{T} y)}{b^{T} b} = \left[ \frac{b b^{T}}{b^{T} b} \right] y , \]
where the bracketed object is the projection matrix \(P\), a \(2 \times 2\) matrix that acts on any \(y\). The order matters: \(b b^{T}\) is a matrix and \(b^{T} b\) is a scalar, so the same two symbols produce different objects depending on their placement.
Second, since \(b^{T} y = \lVert b \rVert \lVert y \rVert \cos \theta\) and \(b^{T} b = \lVert b \rVert^{2}\), the coefficient can be written as \(\hat{x} = \lVert y \rVert \cos \theta / \lVert b \rVert\), so the length of \(p\) is \(\lVert y \rVert \cos \theta\). That is the answer trigonometry gives, and it agrees.
Three behaviours are visible in Figure 1. Dragging \(y\) onto the dashed line reduces \(\hat{x}\) to the ratio of the two lengths and collapses \(e\) to zero. Dragging \(y\) until it is perpendicular to \(b\) sends \(\hat{x}\) to zero and \(p\) to the origin. Stretching \(b\) along its own direction does not move \(p\) at all, because \(\hat{x}\) halves as \(b\) doubles: \(b\) appears once in the numerator and twice in the denominator, so only the direction of \(b\) matters. That is the right behaviour for a projection onto a line, since the line is unchanged by rescaling the vector that spans it.
Why orthogonal vectors have a dot product of zero
The condition \(b^{T} e = 0\) is the one place where geometry entered the argument, so it is worth asking where it comes from. The link is Pythagoras, and the second figure makes it visible: the two right-hand numbers coincide exactly when the corner at the origin reaches ninety degrees.
Figure 2. The triangle has sides \(a\), \(b\) and \(a - b\). The two right-hand readouts are equal precisely when \(a^{T} b = 0\), at which point the corner at the origin is marked as a right angle.
The only geometric input is the notion of length. In coordinates, \(\lVert v \rVert^{2} = v_{1}^{2} + v_{2}^{2}\), which is the Pythagorean theorem applied to the right triangle formed by a vector and the two axes, and \(v_{1}^{2} + v_{2}^{2}\) is exactly \(v^{T} v\). The identity
\[ \lVert v \rVert^{2} = v^{T} v \]
is therefore not a definition introduced for convenience: it is Pythagoras written in dot-product notation, and it is the only geometric fact the argument below needs.
The third side of the triangle in Figure 2 can now be expanded algebraically, in the same way that \((p - q)^{2} = p^{2} - 2pq + q^{2}\) works for ordinary numbers:
\[ \lVert a - b \rVert^{2} = (a - b)^{T}(a - b) = a^{T}a - 2 a^{T} b + b^{T} b = \lVert a \rVert^{2} + \lVert b \rVert^{2} - 2 a^{T} b . \]
Pythagoras, in its usual form, says that the triangle has a right angle at the origin exactly when \(\lVert a - b \rVert^{2} = \lVert a \rVert^{2} + \lVert b \rVert^{2}\). Lining the two statements up, they agree if and only if the leftover term \(-2 a^{T} b\) vanishes, so
\[ a \perp b \iff a^{T} b = 0 . \]
The dot product is thus a measure of the amount by which a triangle fails to be right-angled, and that is what the bottom row of Figure 2 reports: the gap between the two squared quantities is always \(-2 a^{T} b\), and it closes only at ninety degrees. The familiar expression \(a^{T} b = \lVert a \rVert \lVert b \rVert \cos \theta\) is the same statement in different dress, since it follows from the law of cosines, which is Pythagoras with a correction term, and the correction disappears at \(\cos 90^{\circ} = 0\).
Orthogonality as a consequence rather than an assumption
Two assumptions remain in the argument as given. Pythagoras itself was taken as given, which is unobjectionable but still an input, and the closest point on the line was assumed to be the foot of the perpendicular, which the figure makes plausible without establishing it. Both can be removed at once by discarding the geometry and minimizing the error directly. Define
\[ f(x) = \lVert y - x b \rVert^{2} = x^{2}(b^{T} b) - 2 x (b^{T} y) + \lVert y \rVert^{2} , \]
using the same expansion as before. This is an upward-opening parabola in the single variable \(x\), since \(b^{T} b > 0\) for \(b \neq 0\). Setting \(f'(x) = 2x(b^{T}b) - 2(b^{T}y) = 0\) gives
\[ \hat{x} = \frac{b^{T} y}{b^{T} b} , \]
the same coefficient as before, obtained without a picture and without assuming perpendicularity. Orthogonality then follows rather than being imposed: at the minimizer, \(b^{T} e = b^{T} y - \hat{x}(b^{T} b) = 0\).
Conclusion
Projection onto a line reduces to a single scalar because the projection is constrained to be a multiple of the vector spanning the line, and that scalar is \(\hat{x} = b^{T} y / b^{T} b\). The derivation can be started from either end. Assuming that the residual is orthogonal to \(b\) yields the coefficient by algebra, with orthogonality justified by Pythagoras through the identity \(\lVert v \rVert^{2} = v^{T} v\). Conversely, minimizing the squared error yields the same coefficient by calculus and delivers orthogonality as a by-product. Since each route derives the other, the perpendicular picture in Figure 1 can be relied on rather than merely used as an illustration.