Linear Regression as Geometry

We begin with five observed pairs. The horizontal coordinate is the predictor x, and the vertical coordinate is the response y. A regression line will summarize how the response changes as x changes. Here is one candidate line. For this example its height at the centre is fixed at three, and m controls its tilt. We have not yet chosen m. At each observed x value, the fitted line predicts a height. The vertical segment from that fitted height to the observed point is the residual. The sign records direction. A point above the line has a positive residual, and a point below it has a negative residual. But a good fit should penalize a large miss in either direction. So square every residual and add the squares. This sum of squared errors is nonnegative, it treats equal misses above and below equally, and large misses count much more than small ones. Notice what we are minimizing. These are vertical differences in the response coordinate. Ordinary regression of y on x does not use the shortest distance to the line; it holds each observed x fixed. With a horizontal line, m is zero. The five residuals have squared sum sixteen. The line is clearly missing the upward pattern in the data. To choose the tilt without calculus, let us watch every candidate and record its squared error. The same live value of m will drive both pictures. On the right, the red curve records the sum of squared errors for every slope in view. The yellow dot begins at slope zero and error sixteen. Tilt the fitted line upward to slope zero point six. The residual pattern tightens, and the squared error falls from sixteen to five point two. Continue to slope one point two. Several vertical misses are now small, and the dot reaches the bottom of the error curve. The squared error is one point six. Go past that slope to one point eight. Some residuals change sign, but their squares grow again. The error rises back to five point two. Return to the bottom. The line with slope one point two has the smallest sum of squared vertical residuals among this whole family. That is the least-squares fit. The curve on the right is not an appeal to differentiation. It is a geometric record of what the yellow segments on the left are doing. Tilting away from the best line makes their combined squared length grow. For this centred predictor, the intercept stayed at the mean response, three, while the slope changed. In a general fit, both intercept and slope are chosen together to minimize the same squared residual total. This data picture explains what least squares measures. To understand why the optimum has a clean algebraic form, we now change the geometry completely.

Now replace the data plot by observation space. To draw it, we use a miniature data set with three observations. Three observations give three coordinates, so the whole construction fits in ordinary three-dimensional space. The first row records response one, the second response two, and the third response four. Read down the response column, not across a plotted axis. Stack those three responses into one vector y: one, two, four. Each coordinate belongs to one observation, so this single arrow represents the entire response data set. The design matrix also has one coordinate per observation in each column. Its intercept column is one, one, one, because the intercept contributes equally to every fitted response. The predictor column is minus one, zero, one. It records the three predictor values after centring. This green arrow is that whole column. A coefficient beta zero scales the blue column. A coefficient beta one scales the green column. Adding the two scaled arrows produces every fitted response vector this model is capable of making. Those linear combinations fill a plane through the origin. This is the column space of X. Changing the regression coefficients moves the fitted vector somewhere inside this plane, but never outside it. The picture is three-dimensional only because this example has three observations. With n observations, y lives in n-dimensional observation space. With p predictors and an intercept, the columns span a subspace of dimension at most p plus one. The yellow response vector usually misses the predictor plane. An exact fit would put y inside the column space, but these three observations do not lie on one exact regression line. Among all points in the plane, choose the one closest to y. The green arrow reaches that point. It is the fitted-value vector, hat y. Its three coordinates are the three fitted responses. Returning to the original data plot would place those fitted values on the regression line at the three observed predictor values. What remains is the red vector from hat y to y. Coordinate by coordinate, it contains observed response minus fitted response, so it is exactly the residual vector e. The closest route from a point to a plane is perpendicular to the plane. Therefore the residual stands at a right angle to every direction the predictor columns span. This is least squares in one sentence. The fitted vector is the orthogonal projection of y onto the column space of X. The word shadow is useful. Imagine light travelling perpendicular to the predictor plane. The response arrow casts its endpoint onto the plane, and the shadow point is hat y. The red leftover is perpendicular to the shadow plane. Pythagoras then says no other point in the plane can be closer to y. Moving away from hat y adds an in-plane displacement at right angles to the residual. That geometric fact is the minimization. We did not differentiate a sum of squares. We identified the unique shortest vector from y to the model subspace. The same picture survives with many observations and many predictors. Only the dimension changes. The response is one point, the possible fits form a column space, and least squares drops a perpendicular to it.

Now use the right angle to calculate the coefficients. The fitted vector is X times beta hat, because it is a linear combination of the columns of X. The residual is observed y minus fitted y. Substituting the matrix form gives e equals y minus X beta hat. Perpendicular vectors have dot product zero. Since e is perpendicular to the entire column space, it is perpendicular to the intercept column, the first predictor column, and every other predictor column. For one column c sub j, perpendicularity reads c sub j transpose e equals zero. Write one such equation for every column. Stack all those dot products. The transpose X transpose places the predictor columns into the rows of one multiplication, so the whole collection becomes X transpose e equals zero. This compact zero is the algebraic form of the visible right angle. Every coordinate of X transpose e is one column's dot product with the residual. Carry that perpendicularity forward, and replace e by y minus X beta hat. It becomes X transpose times y minus X beta hat equals zero. Distribute X transpose. The first term is X transpose y. The second is X transpose X beta hat. Move the second term to the other side. We obtain X transpose X beta hat equals X transpose y. These are the normal equations. They are called normal because the residual is normal, meaning perpendicular, to the predictor subspace. The name is geometric before it is algebraic. The intercept equation has a familiar consequence. Its column is the all-ones vector, so its dot product with e is the sum of the residuals. With an intercept, that sum must be zero. Each predictor equation says something similar: the residual has zero dot product with that predictor column. No linear pattern available to the model remains inside the residual. If the columns are linearly independent, X transpose X is invertible. Multiplying by its inverse gives the familiar coefficient formula. But the inverse formula is not the fundamental fact. The fundamental fact is X transpose e equals zero. Numerical methods usually solve the projection problem without explicitly forming an inverse. If the columns are dependent, the fitted projection can still be defined even though the coefficient vector need not be unique. That distinction will matter when we discuss multicollinearity. So the normal equations come straight from one right angle: fitted values lie in the column space, residuals leave it perpendicularly, and every predictor column has zero residual dot product.

R squared uses the same projection picture, but first we must identify what counts as total variation. With an intercept in the model, the baseline prediction is the sample mean of y. Subtract that mean from every observed response. The centred vector y sub c measures how far the data sit from the mean-only fit. Centre the fitted vector in the same way. Because the regression model includes the intercept direction, the centred fitted vector remains inside the predictor subspace. The centred response splits into two pieces. The green piece is the variation explained by the predictors. The red piece is the residual variation left unexplained. Projection makes those pieces perpendicular. The three vectors therefore form a right triangle: explained component, residual component, and total centred response. Pythagoras applies to their squared lengths. Total sum of squares equals regression sum of squares plus residual sum of squares. R squared is the explained squared length divided by the total squared length. Equivalently, it is one minus the unexplained squared length divided by the total. In the three-observation example, the total squared length is fourteen thirds. The explained squared length is nine halves, and the residual squared length is one sixth. Nine halves plus one sixth is fourteen thirds, exactly as the right triangle requires. The resulting R squared is twenty-seven over twenty-eight, about zero point nine six four. There is also an angle interpretation. Let theta be the angle between the centred response and its fitted projection. Then R squared is cosine squared theta. A small angle means the response lies close to the model subspace, so the projection keeps nearly all its length. A large angle means a larger perpendicular residual. With ordinary least squares and an intercept, this training-sample ratio lies between zero and one. Zero means the predictors improve nothing beyond the mean. One means every residual is zero. The intercept condition matters. Without the mean direction in the model, this familiar centred Pythagorean decomposition need not hold, and the usual interpretation of R squared needs care. R squared measures alignment with the fitted subspace, not causation, and not whether the model will predict new data well. It is a geometric summary of in-sample explained squared length. The projection picture has now given us residual minimization, the normal equations, and R squared. One final geometric question remains: what if the predictor directions themselves almost collapse together?

To see multicollinearity, first remove the intercept direction by centring, and scale two predictor columns to unit length. Then their relationship is summarized by the angle theta between their vectors. At sixty degrees, the blue and green predictors provide clearly different directions. Together they span the plane, and a fitted vector has coordinates beta one and beta two in this predictor basis. Draw the coefficient contributions tip to tail. The blue contribution first moves along x one. The green contribution then moves along x two, and their sum lands exactly at the red fitted vector. These coefficients are coordinates, not intrinsic lengths. Their values depend on how well the chosen predictor directions form a coordinate system. Now make the green predictor more like the blue one. At thirty degrees, the span is still the whole plane, and the fitted vector has not moved. But the two coefficient contributions must grow and cancel more strongly. Bring the angle down to ten degrees. The predictor arrows are now nearly parallel. To reach the same modest red vector, one long component runs left and the other almost cancels it while climbing to the target. This is multicollinearity. Two columns of X supply nearly the same direction in observation space, so the model has difficulty deciding how much of the fit to assign to each one. The fitted vector itself can still be perfectly clear. What becomes unstable is its description in this nearly collapsed coordinate system. Here is a small perturbation. Raise the fitted target's second coordinate from one point zero zero to one point zero two, only two hundredths. The red endpoint barely moves, but each coefficient changes by about zero point one one. The coordinate change is almost six times the size of the data perturbation, with opposite signs. The algebra reports the same geometry. For standardized columns, X transpose X has ones on the diagonal and cosine theta off the diagonal. Its determinant is one minus cosine squared theta, which is sine squared theta. As the predictor angle shrinks, this determinant approaches zero. At exactly zero degrees the two columns coincide. The span collapses from a plane to a line, X loses rank, and infinitely many coefficient pairs can describe the same fitted vector. Near zero, the inverse still exists in exact arithmetic, but it must magnify the thin direction enormously. That sensitivity is called ill conditioning. For two standardized predictors, this condition number is the square root of one plus cosine theta over one minus cosine theta. At ten degrees, the Gram determinant is about zero point zero three and the condition number is already about eleven point four. A condition number near one means the predictor directions are well separated. A large value means small changes in y, rounding, or sampling can create much larger changes in the estimated coefficients. This does not automatically make the fitted values unstable. Many different large, cancelling coefficient pairs can land at nearly the same projection point. Prediction within the observed predictor pattern may therefore look steady while individual coefficient interpretations do not. The first lesson is geometric: nearly parallel columns provide almost the same direction. The second is about coordinates: reaching one moderate fit may require large contributions with opposite signs. The third is statistical: a small change in the sample can move those large coefficients substantially, increasing their uncertainty. And the fourth separates two questions. The model may predict a combined effect reasonably while the separate effects of the correlated predictors remain poorly determined. Possible responses include collecting data where the predictors vary more independently, combining redundant predictors, choosing one of them, or using a regularized fit. Each response changes how the thin direction is handled, so it should follow the scientific purpose of the model. The whole lecture is now one geometric story. Data points create vertical residuals. Least squares chooses the shortest residual vector in observation space. Perpendicularity gives the normal equations. Pythagoras turns the same projection into R squared. And when predictor directions nearly align, the spanning coordinates become ill conditioned even if the projected fit barely changes.