{"version":1,"lectureId":"01M14V05J1Q2H560M850WHHDZA","attempt":1,"publication":{"slug":"least-squares-is-a-projection","title":"Linear Regression as Geometry","subject":"statistics","summary":"Linear regression developed as a geometric projection. The lecture begins with observed data, vertical residuals, and a fitted line whose squared error visibly falls to a minimum as its slope changes. It then moves into observation space, where predictor columns span a subspace, fitted values are the orthogonal shadow of the response, and the residual is the perpendicular component. That right angle yields the normal equations directly. The same decomposition gives R squared as a ratio of squared lengths, while nearly parallel predictor columns reveal multicollinearity, unstable coefficients, and ill conditioning.","metaDescription":"Learn linear regression through residual geometry, orthogonal projection, normal equations, R squared, and nearly parallel predictors.","transcript":"We begin with five observed pairs. The horizontal coordinate is the predictor x, and the vertical coordinate is the response y. A regression line will summarize how the response changes as x changes. Here is one candidate line. For this example its height at the centre is fixed at three, and m controls its tilt. We have not yet chosen m. At each observed x value, the fitted line predicts a height. The vertical segment from that fitted height to the observed point is the residual. The sign records direction. A point above the line has a positive residual, and a point below it has a negative residual. But a good fit should penalize a large miss in either direction. So square every residual and add the squares. This sum of squared errors is nonnegative, it treats equal misses above and below equally, and large misses count much more than small ones. Notice what we are minimizing. These are vertical differences in the response coordinate. Ordinary regression of y on x does not use the shortest distance to the line; it holds each observed x fixed. With a horizontal line, m is zero. The five residuals have squared sum sixteen. The line is clearly missing the upward pattern in the data. To choose the tilt without calculus, let us watch every candidate and record its squared error. The same live value of m will drive both pictures. On the right, the red curve records the sum of squared errors for every slope in view. The yellow dot begins at slope zero and error sixteen. Tilt the fitted line upward to slope zero point six. The residual pattern tightens, and the squared error falls from sixteen to five point two. Continue to slope one point two. Several vertical misses are now small, and the dot reaches the bottom of the error curve. The squared error is one point six. Go past that slope to one point eight. Some residuals change sign, but their squares grow again. The error rises back to five point two. Return to the bottom. The line with slope one point two has the smallest sum of squared vertical residuals among this whole family. That is the least-squares fit. The curve on the right is not an appeal to differentiation. It is a geometric record of what the yellow segments on the left are doing. Tilting away from the best line makes their combined squared length grow. For this centred predictor, the intercept stayed at the mean response, three, while the slope changed. In a general fit, both intercept and slope are chosen together to minimize the same squared residual total. This data picture explains what least squares measures. To understand why the optimum has a clean algebraic form, we now change the geometry completely. Now replace the data plot by observation space. To draw it, we use a miniature data set with three observations. Three observations give three coordinates, so the whole construction fits in ordinary three-dimensional space. The first row records response one, the second response two, and the third response four. Read down the response column, not across a plotted axis. Stack those three responses into one vector y: one, two, four. Each coordinate belongs to one observation, so this single arrow represents the entire response data set. The design matrix also has one coordinate per observation in each column. Its intercept column is one, one, one, because the intercept contributes equally to every fitted response. The predictor column is minus one, zero, one. It records the three predictor values after centring. This green arrow is that whole column. A coefficient beta zero scales the blue column. A coefficient beta one scales the green column. Adding the two scaled arrows produces every fitted response vector this model is capable of making. Those linear combinations fill a plane through the origin. This is the column space of X. Changing the regression coefficients moves the fitted vector somewhere inside this plane, but never outside it. The picture is three-dimensional only because this example has three observations. With n observations, y lives in n-dimensional observation space. With p predictors and an intercept, the columns span a subspace of dimension at most p plus one. The yellow response vector usually misses the predictor plane. An exact fit would put y inside the column space, but these three observations do not lie on one exact regression line. Among all points in the plane, choose the one closest to y. The green arrow reaches that point. It is the fitted-value vector, hat y. Its three coordinates are the three fitted responses. Returning to the original data plot would place those fitted values on the regression line at the three observed predictor values. What remains is the red vector from hat y to y. Coordinate by coordinate, it contains observed response minus fitted response, so it is exactly the residual vector e. The closest route from a point to a plane is perpendicular to the plane. Therefore the residual stands at a right angle to every direction the predictor columns span. This is least squares in one sentence. The fitted vector is the orthogonal projection of y onto the column space of X. The word shadow is useful. Imagine light travelling perpendicular to the predictor plane. The response arrow casts its endpoint onto the plane, and the shadow point is hat y. The red leftover is perpendicular to the shadow plane. Pythagoras then says no other point in the plane can be closer to y. Moving away from hat y adds an in-plane displacement at right angles to the residual. That geometric fact is the minimization. We did not differentiate a sum of squares. We identified the unique shortest vector from y to the model subspace. The same picture survives with many observations and many predictors. Only the dimension changes. The response is one point, the possible fits form a column space, and least squares drops a perpendicular to it. Now use the right angle to calculate the coefficients. The fitted vector is X times beta hat, because it is a linear combination of the columns of X. The residual is observed y minus fitted y. Substituting the matrix form gives e equals y minus X beta hat. Perpendicular vectors have dot product zero. Since e is perpendicular to the entire column space, it is perpendicular to the intercept column, the first predictor column, and every other predictor column. For one column c sub j, perpendicularity reads c sub j transpose e equals zero. Write one such equation for every column. Stack all those dot products. The transpose X transpose places the predictor columns into the rows of one multiplication, so the whole collection becomes X transpose e equals zero. This compact zero is the algebraic form of the visible right angle. Every coordinate of X transpose e is one column's dot product with the residual. Carry that perpendicularity forward, and replace e by y minus X beta hat. It becomes X transpose times y minus X beta hat equals zero. Distribute X transpose. The first term is X transpose y. The second is X transpose X beta hat. Move the second term to the other side. We obtain X transpose X beta hat equals X transpose y. These are the normal equations. They are called normal because the residual is normal, meaning perpendicular, to the predictor subspace. The name is geometric before it is algebraic. The intercept equation has a familiar consequence. Its column is the all-ones vector, so its dot product with e is the sum of the residuals. With an intercept, that sum must be zero. Each predictor equation says something similar: the residual has zero dot product with that predictor column. No linear pattern available to the model remains inside the residual. If the columns are linearly independent, X transpose X is invertible. Multiplying by its inverse gives the familiar coefficient formula. But the inverse formula is not the fundamental fact. The fundamental fact is X transpose e equals zero. Numerical methods usually solve the projection problem without explicitly forming an inverse. If the columns are dependent, the fitted projection can still be defined even though the coefficient vector need not be unique. That distinction will matter when we discuss multicollinearity. So the normal equations come straight from one right angle: fitted values lie in the column space, residuals leave it perpendicularly, and every predictor column has zero residual dot product. R squared uses the same projection picture, but first we must identify what counts as total variation. With an intercept in the model, the baseline prediction is the sample mean of y. Subtract that mean from every observed response. The centred vector y sub c measures how far the data sit from the mean-only fit. Centre the fitted vector in the same way. Because the regression model includes the intercept direction, the centred fitted vector remains inside the predictor subspace. The centred response splits into two pieces. The green piece is the variation explained by the predictors. The red piece is the residual variation left unexplained. Projection makes those pieces perpendicular. The three vectors therefore form a right triangle: explained component, residual component, and total centred response. Pythagoras applies to their squared lengths. Total sum of squares equals regression sum of squares plus residual sum of squares. R squared is the explained squared length divided by the total squared length. Equivalently, it is one minus the unexplained squared length divided by the total. In the three-observation example, the total squared length is fourteen thirds. The explained squared length is nine halves, and the residual squared length is one sixth. Nine halves plus one sixth is fourteen thirds, exactly as the right triangle requires. The resulting R squared is twenty-seven over twenty-eight, about zero point nine six four. There is also an angle interpretation. Let theta be the angle between the centred response and its fitted projection. Then R squared is cosine squared theta. A small angle means the response lies close to the model subspace, so the projection keeps nearly all its length. A large angle means a larger perpendicular residual. With ordinary least squares and an intercept, this training-sample ratio lies between zero and one. Zero means the predictors improve nothing beyond the mean. One means every residual is zero. The intercept condition matters. Without the mean direction in the model, this familiar centred Pythagorean decomposition need not hold, and the usual interpretation of R squared needs care. R squared measures alignment with the fitted subspace, not causation, and not whether the model will predict new data well. It is a geometric summary of in-sample explained squared length. The projection picture has now given us residual minimization, the normal equations, and R squared. One final geometric question remains: what if the predictor directions themselves almost collapse together? To see multicollinearity, first remove the intercept direction by centring, and scale two predictor columns to unit length. Then their relationship is summarized by the angle theta between their vectors. At sixty degrees, the blue and green predictors provide clearly different directions. Together they span the plane, and a fitted vector has coordinates beta one and beta two in this predictor basis. Draw the coefficient contributions tip to tail. The blue contribution first moves along x one. The green contribution then moves along x two, and their sum lands exactly at the red fitted vector. These coefficients are coordinates, not intrinsic lengths. Their values depend on how well the chosen predictor directions form a coordinate system. Now make the green predictor more like the blue one. At thirty degrees, the span is still the whole plane, and the fitted vector has not moved. But the two coefficient contributions must grow and cancel more strongly. Bring the angle down to ten degrees. The predictor arrows are now nearly parallel. To reach the same modest red vector, one long component runs left and the other almost cancels it while climbing to the target. This is multicollinearity. Two columns of X supply nearly the same direction in observation space, so the model has difficulty deciding how much of the fit to assign to each one. The fitted vector itself can still be perfectly clear. What becomes unstable is its description in this nearly collapsed coordinate system. Here is a small perturbation. Raise the fitted target's second coordinate from one point zero zero to one point zero two, only two hundredths. The red endpoint barely moves, but each coefficient changes by about zero point one one. The coordinate change is almost six times the size of the data perturbation, with opposite signs. The algebra reports the same geometry. For standardized columns, X transpose X has ones on the diagonal and cosine theta off the diagonal. Its determinant is one minus cosine squared theta, which is sine squared theta. As the predictor angle shrinks, this determinant approaches zero. At exactly zero degrees the two columns coincide. The span collapses from a plane to a line, X loses rank, and infinitely many coefficient pairs can describe the same fitted vector. Near zero, the inverse still exists in exact arithmetic, but it must magnify the thin direction enormously. That sensitivity is called ill conditioning. For two standardized predictors, this condition number is the square root of one plus cosine theta over one minus cosine theta. At ten degrees, the Gram determinant is about zero point zero three and the condition number is already about eleven point four. A condition number near one means the predictor directions are well separated. A large value means small changes in y, rounding, or sampling can create much larger changes in the estimated coefficients. This does not automatically make the fitted values unstable. Many different large, cancelling coefficient pairs can land at nearly the same projection point. Prediction within the observed predictor pattern may therefore look steady while individual coefficient interpretations do not. The first lesson is geometric: nearly parallel columns provide almost the same direction. The second is about coordinates: reaching one moderate fit may require large contributions with opposite signs. The third is statistical: a small change in the sample can move those large coefficients substantially, increasing their uncertainty. And the fourth separates two questions. The model may predict a combined effect reasonably while the separate effects of the correlated predictors remain poorly determined. Possible responses include collecting data where the predictors vary more independently, combining redundant predictors, choosing one of them, or using a regularized fit. Each response changes how the thin direction is handled, so it should follow the scientific purpose of the model. The whole lecture is now one geometric story. Data points create vertical residuals. Least squares chooses the shortest residual vector in observation space. Perpendicularity gives the normal equations. Pythagoras turns the same projection into R squared. And when predictor directions nearly align, the spanning coordinates become ill conditioned even if the projected fit barely changes.","watch":{"version":1,"scenes":[{"title":"Residuals and the Best Line","start":0,"end":195.44677083333335,"objects":{"candidate":"a Line [green] labelled \"hat(y)=3+m x\" drawn in data_axes (start=(-2.0, (3.0 - (2.0 * slope))), end=(2.0, (3.0 + (2.0 * slope))))","data_axes":"an Axes (x_range=(-2.5, 2.5), y_range=(-1.0, 7.0), x_ticks_every=1.0)","heading":"a Heading that says \"Observed Data and a Candidate Line\"","heading_2":"a Heading that says \"Tilt the Line and Watch the Error\"","left_label":"a Tex [text] that says \"Data and residuals\"","line_family":"a Math [text] that says \"$hat(y)_i = 3 + m x_i$\"","minimum_point":"a PlotPoint [green] labelled \"upright(\"minimum\")\" drawn in sse_axes (target='sse_curve', x=1.2)","point_1":"a Point [blue] drawn in data_axes (location=(-2.0, 1.0))","point_2":"a Point [blue] drawn in data_axes (location=(-1.0, 2.0))","point_3":"a Point [blue] drawn in data_axes (location=(0.0, 2.0))","point_4":"a Point [blue] drawn in data_axes (location=(1.0, 4.0))","point_5":"a Point [blue] drawn in data_axes (location=(2.0, 6.0))","residual_1":"a Line [yellow] drawn in data_axes (start=(-2.0, (3.0 - (2.0 * slope))), end=(-2.0, 1.0))","residual_2":"a Line [yellow] drawn in data_axes (start=(-1.0, (3.0 - slope)), end=(-1.0, 2.0))","residual_3":"a Line [yellow] drawn in data_axes (start=(0.0, 3.0), end=(0.0, 2.0))","residual_4":"a Line [yellow] drawn in data_axes (start=(1.0, (3.0 + slope)), end=(1.0, 4.0))","residual_5":"a Line [yellow] drawn in data_axes (start=(2.0, (3.0 + (2.0 * slope))), end=(2.0, 6.0))","residual_definition":"a Math [text] that says \"$e_i = y_i - hat(y)_i$\"","right_label":"a Tex [text] that says \"Squared error versus slope\"","slope":"a VariableNumber (format_spec='.1f')","squared_error":"a Math [text] that says \"$upright(\"SSE\")(m) = sum_i e_i^2$\"","sse_axes":"an Axes (x_range=(-0.2, 2.4), y_range=(0.0, 20.0), x_ticks_every=0.5)","sse_curve":"a FunctionPlot [red] drawn in sse_axes (function=<function>, x_range=(-0.1, 2.3))","sse_tracker":"a PlotPoint [yellow] labelled \"0.0\" drawn in sse_axes (target='sse_curve', x=<VariableNumber slope = 1.2>)"},"beats":[{"start":0,"say":"We begin with five observed pairs. The horizontal coordinate is the predictor x, and the vertical coordinate is the response y. A regression line will summarize how the response changes as x changes.","live":[],"does":[[0,"heading is shown on the screen, written out."],[1.207,"data_axes is shown on the screen, written out."],[1.649,"point_1 is shown on the screen, written out."],[1.7690000000000001,"point_2 is shown on the screen, written out."],[1.889,"point_3 is shown on the screen, written out."],[2.009,"point_4 is shown on the screen, written out."],[2.129,"point_5 is shown on the screen, written out."]]},{"start":14.253499999999999,"say":"Here is one candidate line. For this example its height at the centre is fixed at three, and m controls its tilt. We have not yet chosen m.","live":["data_axes","heading","point_1","point_2","point_3","point_4","point_5"],"does":[[15.437999999999999,"candidate is shown on the screen, written out."],[21.091,"data_axes moves to a new place on the board."],[21.091,"line_family is shown on the screen, written out."]]},{"start":25.4065,"say":"At each observed x value, the fitted line predicts a height. The vertical segment from that fitted height to the observed point is the residual.","live":["line_family","data_axes","heading","point_1","point_2","point_3","point_4","point_5","candidate"],"does":[[30.515,"residual_1 is shown on the screen, written out."],[30.635,"residual_2 is shown on the screen, written out."],[30.755,"residual_3 is shown on the screen, written out."],[30.875,"residual_4 is shown on the screen, written out."],[30.995,"residual_5 is shown on the screen, written out."],[33.812000000000005,"residual_definition is shown on the screen, written out."]]},{"start":35.4915,"say":"The sign records direction. A point above the line has a positive residual, and a point below it has a negative residual. But a good fit should penalize a large miss in either direction.","live":["line_family","residual_definition","data_axes","heading","point_1","point_2","point_3","point_4","point_5","candidate","residual_1","residual_2","residual_3","residual_4","residual_5"],"does":[[38.475,"residual_definition (the \"y_i\" part) is emphasized."],[39.19500000000001,"residual_definition (the \"hat(y)_i\" part) is emphasized."],[39.19500000000001,"residual_definition (the \"y_i\" part) is no longer emphasized."],[48.727000000000004,"residual_definition (the \"hat(y)_i\" part) is no longer emphasized."]]},{"start":49.327,"say":"So square every residual and add the squares. This sum of squared errors is nonnegative, it treats equal misses above and below equally, and large misses count much more than small ones.","live":null,"does":[[50.092999999999996,"squared_error is shown on the screen, written out."],[52.403,"squared_error (the \"e_i^2\" part) is emphasized."],[63.096,"squared_error (the \"e_i^2\" part) is no longer emphasized."]]},{"start":63.696,"say":"Notice what we are minimizing. These are vertical differences in the response coordinate. Ordinary regression of y on x does not use the shortest distance to the line; it holds each observed x fixed.","live":["line_family","residual_definition","squared_error","data_axes","heading","point_1","point_2","point_3","point_4","point_5","candidate","residual_1","residual_2","residual_3","residual_4","residual_5"],"does":[[66.981,"residual_2 is indicated — a transient flash."],[77.094,"residual_4 is indicated — a transient flash."]]},{"start":78.53,"say":"With a horizontal line, m is zero. The five residuals have squared sum sixteen. The line is clearly missing the upward pattern in the data.","live":null,"does":[[79.19199999999998,"candidate is indicated — a transient flash."],[83.54599999999998,"squared_error is indicated — a transient flash."]]},{"start":89.834,"say":"To choose the tilt without calculus, let us watch every candidate and record its squared error. The same live value of m will drive both pictures.","live":null,"does":[[99.6215,"data_axes moves to a new place on the board."],[99.6215,"heading is hidden from the screen — left the board."],[99.6215,"line_family is hidden from the screen — left the board."],[99.6215,"residual_definition is hidden from the screen — left the board."],[99.6215,"squared_error is hidden from the screen — left the board."]]},{"start":100.8215,"say":"On the right, the red curve records the sum of squared errors for every slope in view. The yellow dot begins at slope zero and error sixteen.","live":["data_axes","point_1","point_2","point_3","point_4","point_5","candidate","residual_1","residual_2","residual_3","residual_4","residual_5"],"does":[[100.8215,"sse_axes is shown on the screen, written out."],[102.36500000000001,"sse_curve is shown on the screen, written out."],[107.055,"sse_tracker is shown on the screen, written out."]]},{"start":111.5565,"say":"Tilt the fitted line upward to slope zero point six. The residual pattern tightens, and the squared error falls from sixteen to five point two.","live":["data_axes","point_1","point_2","point_3","point_4","point_5","candidate","residual_1","residual_2","residual_3","residual_4","residual_5","sse_axes","sse_curve","sse_tracker"],"does":[[111.905,"candidate is redrawn as the numbers it depends on change."],[111.905,"residual_1 is redrawn as the numbers it depends on change."],[111.905,"residual_2 is redrawn as the numbers it depends on change."],[111.905,"residual_4 is redrawn as the numbers it depends on change."],[111.905,"residual_5 is redrawn as the numbers it depends on change."],[111.905,"sse_tracker is redrawn as the numbers it depends on change."],[111.905,"slope ticks to 0.6."]]},{"start":121.5725,"say":"Continue to slope one point two. Several vertical misses are now small, and the dot reaches the bottom of the error curve. The squared error is one point six.","live":null,"does":[[121.828,"candidate is redrawn as the numbers it depends on change."],[121.828,"residual_1 is redrawn as the numbers it depends on change."],[121.828,"residual_2 is redrawn as the numbers it depends on change."],[121.828,"residual_4 is redrawn as the numbers it depends on change."],[121.828,"residual_5 is redrawn as the numbers it depends on change."],[121.828,"sse_tracker is redrawn as the numbers it depends on change."],[121.828,"slope ticks to 1.2."],[127.74900000000001,"minimum_point is shown on the screen, written out."]]},{"start":132.993,"say":"Go past that slope to one point eight. Some residuals change sign, but their squares grow again. The error rises back to five point two.","live":["data_axes","point_1","point_2","point_3","point_4","point_5","candidate","residual_1","residual_2","residual_3","residual_4","residual_5","sse_axes","sse_curve","sse_tracker","minimum_point"],"does":[[133.58499999999998,"candidate is redrawn as the numbers it depends on change."],[133.58499999999998,"residual_1 is redrawn as the numbers it depends on change."],[133.58499999999998,"residual_2 is redrawn as the numbers it depends on change."],[133.58499999999998,"residual_4 is redrawn as the numbers it depends on change."],[133.58499999999998,"residual_5 is redrawn as the numbers it depends on change."],[133.58499999999998,"sse_tracker is redrawn as the numbers it depends on change."],[133.58499999999998,"slope ticks to 1.8."]]},{"start":143.2645,"say":"Return to the bottom. The line with slope one point two has the smallest sum of squared vertical residuals among this whole family. That is the least-squares fit.","live":null,"does":[[143.613,"candidate is redrawn as the numbers it depends on change."],[143.613,"residual_1 is redrawn as the numbers it depends on change."],[143.613,"residual_2 is redrawn as the numbers it depends on change."],[143.613,"residual_4 is redrawn as the numbers it depends on change."],[143.613,"residual_5 is redrawn as the numbers it depends on change."],[143.613,"sse_tracker is redrawn as the numbers it depends on change."],[143.613,"slope ticks to 1.2."],[147.607,"minimum_point is indicated — a transient flash."]]},{"start":155.323,"say":"The curve on the right is not an appeal to differentiation. It is a geometric record of what the yellow segments on the left are doing. Tilting away from the best line makes their combined squared length grow.","live":null,"does":[[161.709,"residual_1 is indicated — a transient flash."],[161.709,"residual_5 is indicated — a transient flash."],[167.839,"sse_curve is indicated — a transient flash."]]},{"start":169.263,"say":"For this centred predictor, the intercept stayed at the mean response, three, while the slope changed. In a general fit, both intercept and slope are chosen together to minimize the same squared residual total.","live":null,"does":[[171.005,"candidate is indicated — a transient flash."]]},{"start":183.94650000000001,"say":"This data picture explains what least squares measures. To understand why the optimum has a clean algebraic form, we now change the geometry completely.","live":null,"does":[[194.4051041666667,"data_axes is hidden from the screen — left the board."],[194.4051041666667,"point_1 is hidden from the screen — data_axes left the board."],[194.4051041666667,"point_2 is hidden from the screen — data_axes left the board."],[194.4051041666667,"point_3 is hidden from the screen — data_axes left the board."],[194.4051041666667,"point_4 is hidden from the screen — data_axes left the board."],[194.4051041666667,"point_5 is hidden from the screen — data_axes left the board."],[194.4051041666667,"candidate is hidden from the screen — data_axes left the board."],[194.4051041666667,"residual_1 is hidden from the screen — data_axes left the board."],[194.4051041666667,"residual_2 is hidden from the screen — data_axes left the board."],[194.4051041666667,"residual_3 is hidden from the screen — data_axes left the board."],[194.4051041666667,"residual_4 is hidden from the screen — data_axes left the board."],[194.4051041666667,"residual_5 is hidden from the screen — data_axes left the board."],[194.4051041666667,"sse_axes is hidden from the screen — left the board."],[194.4051041666667,"sse_curve is hidden from the screen — sse_axes left the board."],[194.4051041666667,"sse_tracker is hidden from the screen — sse_axes left the board."],[194.4051041666667,"minimum_point is hidden from the screen — sse_axes left the board."]]}]},{"title":"The Fit as a Shadow","start":195.44677083333335,"end":421.9515,"objects":{"column_plane":"a Plane [blue] labelled \"upright(\"col\")(X)\" drawn in space (normal=(1.0, -2.0, 1.0), edge_direction=(1.0, 1.0, 1.0), opacity=0.16)","data_table":"a Table [text] that says \"Observation $y$ $1$ $x$ 1 1 1 -1 2 2 1 0 3 4 1 1\" (rows=(('Observation', '$y$', '$1$', '$x$'), ('1', '1', '1', '-1'), (…, header=True)","decomposition":"a Math [text] that says \"$y = hat(y) + e$\"","fitted_vector":"a Vector [green] labelled \"hat(y)\" drawn in space (start=(0.0, 0.0, 0.0), end=(0.8333333333333334, 2.3333333333333335, 3.8333333333333335))","heading":"a Heading that says \"Three Observations Become Three Coordinates\"","heading_2":"a Heading that says \"The Predictor Columns Span the Possible Fits\"","heading_3":"a Heading that says \"Least Squares Is Orthogonal Projection\"","intercept_vector":"a Vector [blue] labelled \"c_0\" drawn in space (start=(0.0, 0.0, 0.0), end=(1.0, 1.0, 1.0))","linear_combinations":"a Math [text] that says \"$hat(y) = beta_0 c_0 + beta_1 c_1$\"","matrix_form":"a Math [text] that says \"$hat(y) = X hat(beta)$\"","predictor_vector":"a Vector [green] labelled \"c_1\" drawn in space (start=(0.0, 0.0, 0.0), end=(-1.0, 0.0, 1.0))","projection_note":"a Panel that says \"The least-squares fitted vector is the point in the column space closest to the observed response vector.\"","projection_rule":"a Math [text] that says \"$hat(y) = upright(\"proj\")_(upright(\"col\")(X)) y$\"","residual_vector":"a Vector [red] labelled \"e\" drawn in space (start=(0.8333333333333334, 2.3333333333333335, 3.8333333333333335), end=(1.0, 2.0, 4.0))","response_vector":"a Vector [yellow] labelled \"y\" drawn in space (start=(0.0, 0.0, 0.0), end=(1.0, 2.0, 4.0))","right_angle":"an Angle [yellow] drawn in space (vertex=(0.8333333333333334, 2.3333333333333335, 3.8333333333333335), sides=((0.0, 0.0, 0.0), (1.0, 2.0, 4.0)), right_angle=True)","space":"an Axes3D (x_range=(-1.5, 2.0), y_range=(-0.5, 3.0), z_range=(-0.5, 4.5))"},"beats":[{"start":195.44677083333335,"say":"Now replace the data plot by observation space. To draw it, we use a miniature data set with three observations. Three observations give three coordinates, so the whole construction fits in ordinary three-dimensional space.","live":[],"does":[[195.44677083333335,"heading is shown on the screen, written out."],[201.62377083333334,"data_table is shown on the screen, written out."],[208.06677083333335,"space is shown on the screen, written out."]]},{"start":210.21077083333336,"say":"The first row records response one, the second response two, and the third response four. Read down the response column, not across a plotted axis.","live":["space","heading"],"does":[[210.75577083333334,"data_table is shown on the screen, written out."],[213.15977083333334,"data_table is shown on the screen, written out."],[214.80777083333334,"data_table is shown on the screen, written out."],[217.35077083333334,"data_table (the \"column=2\" part) is emphasized."]]},{"start":221.14377083333335,"say":"Stack those three responses into one vector y: one, two, four. Each coordinate belongs to one observation, so this single arrow represents the entire response data set.","live":null,"does":[[223.60477083333333,"response_vector is shown on the screen, written out."],[233.34527083333336,"data_table (the \"column=2\" part) is no longer emphasized."]]},{"start":233.94527083333335,"say":"The design matrix also has one coordinate per observation in each column. Its intercept column is one, one, one, because the intercept contributes equally to every fitted response.","live":["space","heading","response_vector"],"does":[[239.33177083333334,"intercept_vector is shown on the screen, written out."]]},{"start":246.81727083333334,"say":"The predictor column is minus one, zero, one. It records the three predictor values after centring. This green arrow is that whole column.","live":["space","heading","response_vector","intercept_vector"],"does":[[254.20077083333334,"predictor_vector is shown on the screen, written out."],[256.19827083333337,"data_table is hidden from the screen — left the board."],[256.19827083333337,"heading is hidden from the screen — left the board."]]},{"start":256.79827083333333,"say":"A coefficient beta zero scales the blue column. A coefficient beta one scales the green column. Adding the two scaled arrows produces every fitted response vector this model is capable of making.","live":["space","response_vector","intercept_vector","predictor_vector"],"does":[[256.79827083333333,"linear_combinations is shown on the screen, written out."],[259.03877083333333,"intercept_vector is indicated — a transient flash."],[262.60277083333335,"predictor_vector is indicated — a transient flash."]]},{"start":270.62127083333337,"say":"Those linear combinations fill a plane through the origin. This is the column space of X. Changing the regression coefficients moves the fitted vector somewhere inside this plane, but never outside it.","live":["space","response_vector","intercept_vector","predictor_vector","linear_combinations"],"does":[[271.29477083333336,"linear_combinations is indicated — a transient flash."],[272.7577708333333,"column_plane is shown on the screen, written out."]]},{"start":284.21327083333335,"say":"The picture is three-dimensional only because this example has three observations. With n observations, y lives in n-dimensional observation space. With p predictors and an intercept, the columns span a subspace of dimension at most p plus one.","live":["space","response_vector","intercept_vector","predictor_vector","linear_combinations","column_plane"],"does":[[285.43277083333334,"space turns in its own slot."]]},{"start":301.54327083333334,"say":"The yellow response vector usually misses the predictor plane. An exact fit would put y inside the column space, but these three observations do not lie on one exact regression line.","live":null,"does":[[302.0657708333333,"response_vector is indicated — a transient flash."],[304.82877083333335,"column_plane is indicated — a transient flash."]]},{"start":313.67227083333336,"say":"Among all points in the plane, choose the one closest to y. The green arrow reaches that point. It is the fitted-value vector, hat y.","live":null,"does":[[318.5367708333333,"fitted_vector is shown on the screen, written out."],[321.31177083333336,"matrix_form is shown on the screen, written out."],[323.6567708333333,"linear_combinations is hidden from the screen."]]},{"start":324.25677083333335,"say":"Its three coordinates are the three fitted responses. Returning to the original data plot would place those fitted values on the regression line at the three observed predictor values.","live":["space","response_vector","intercept_vector","predictor_vector","matrix_form","column_plane","fitted_vector"],"does":[[326.24177083333336,"fitted_vector is indicated — a transient flash."]]},{"start":335.5727708333334,"say":"What remains is the red vector from hat y to y. Coordinate by coordinate, it contains observed response minus fitted response, so it is exactly the residual vector e.","live":null,"does":[[336.9667708333334,"residual_vector is shown on the screen, written out."],[344.90777083333336,"decomposition is shown on the screen, written out."]]},{"start":347.77127083333335,"say":"The closest route from a point to a plane is perpendicular to the plane. Therefore the residual stands at a right angle to every direction the predictor columns span.","live":["space","response_vector","intercept_vector","predictor_vector","matrix_form","decomposition","column_plane","fitted_vector","residual_vector"],"does":[[349.7797708333334,"column_plane is indicated — a transient flash."],[354.59877083333333,"right_angle is shown on the screen, written out."],[357.94177083333335,"decomposition moves to a new place on the board."],[357.94177083333335,"matrix_form is hidden from the screen."]]},{"start":359.14177083333334,"say":"This is least squares in one sentence. The fitted vector is the orthogonal projection of y onto the column space of X.","live":["space","response_vector","intercept_vector","predictor_vector","decomposition","column_plane","fitted_vector","residual_vector","right_angle"],"does":[[359.14177083333334,"projection_note is shown on the screen, written out."],[362.52077083333336,"fitted_vector is indicated — a transient flash."],[364.1687708333334,"projection_rule is shown on the screen, written out."]]},{"start":368.3682708333333,"say":"The word shadow is useful. Imagine light travelling perpendicular to the predictor plane. The response arrow casts its endpoint onto the plane, and the shadow point is hat y.","live":["space","response_vector","intercept_vector","predictor_vector","decomposition","column_plane","fitted_vector","residual_vector","right_angle","projection_note","projection_rule"],"does":[[369.1927708333333,"fitted_vector is indicated — a transient flash."],[374.82377083333336,"response_vector is indicated — a transient flash."]]},{"start":380.5317708333333,"say":"The red leftover is perpendicular to the shadow plane. Pythagoras then says no other point in the plane can be closer to y. Moving away from hat y adds an in-plane displacement at right angles to the residual.","live":null,"does":[[381.03077083333335,"residual_vector is indicated — a transient flash."],[392.0947708333333,"right_angle is indicated — a transient flash."]]},{"start":394.40177083333333,"say":"That geometric fact is the minimization. We did not differentiate a sum of squares. We identified the unique shortest vector from y to the model subspace.","live":null,"does":[[402.4937708333334,"projection_note (the \"closest\" part) is emphasized."],[405.6402708333333,"projection_note (the \"closest\" part) is no longer emphasized."]]},{"start":406.24027083333334,"say":"The same picture survives with many observations and many predictors. Only the dimension changes. The response is one point, the possible fits form a column space, and least squares drops a perpendicular to it.","live":null,"does":[[420.9098333333334,"decomposition is hidden from the screen — left the board."],[420.9098333333334,"projection_note is hidden from the screen — left the board."],[420.9098333333334,"projection_rule is hidden from the screen — left the board."],[420.9098333333334,"space is hidden from the screen — left the board."],[420.9098333333334,"response_vector is hidden from the screen — space left the board."],[420.9098333333334,"intercept_vector is hidden from the screen — space left the board."],[420.9098333333334,"predictor_vector is hidden from the screen — space left the board."],[420.9098333333334,"column_plane is hidden from the screen — space left the board."],[420.9098333333334,"fitted_vector is hidden from the screen — space left the board."],[420.9098333333334,"residual_vector is hidden from the screen — space left the board."],[420.9098333333334,"right_angle is hidden from the screen — space left the board."]]}]},{"title":"Perpendicularity and the Normal Equations","start":421.9515,"end":607.6031666666667,"objects":{"column_0":"a Vector [blue] labelled \"c_0\" drawn in space (start=(0.0, 0.0, 0.0), end=(1.0, 1.0, 1.0))","column_1":"a Vector [green] labelled \"c_1\" drawn in space (start=(0.0, 0.0, 0.0), end=(-1.0, 0.0, 1.0))","condition":"a Text [text] that says \"This inverse formula requires linearly independent columns. The normal equations themselves remain the central geometric statement.\"","fit":"a Vector [green] labelled \"hat(y)\" drawn in space (start=(0.0, 0.0, 0.0), end=(0.8333333333333334, 2.3333333333333335, 3.8333333333333335))","heading":"a Heading that says \"Turn the Right Angle into Equations\"","heading_2":"a Heading that says \"The Normal Equations\"","normal_work":"a Derivation [text] that says \"$X^T (y - X hat(beta)) &= 0 \\ X^T y - X^T X hat(beta) &= 0 \\ X^T X hat(beta) &= X^T y$\"","orthogonality_fact":"a Math [text] that says \"$X^T e = 0$\"","orthogonality_work":"a Derivation [text] that says \"$hat(y) &= X hat(beta) \\ e &= y - X hat(beta) \\ c_j^T e &= 0, quad j=0,1,dots,p \\ X^T e &= 0$\"","plane":"a Plane [blue] labelled \"upright(\"col\")(X)\" drawn in space (normal=(1.0, -2.0, 1.0), edge_direction=(1.0, 1.0, 1.0), opacity=0.16)","residual":"a Vector [red] labelled \"e\" drawn in space (start=(0.8333333333333334, 2.3333333333333335, 3.8333333333333335), end=(1.0, 2.0, 4.0))","right_angle":"an Angle [yellow] drawn in space (vertex=(0.8333333333333334, 2.3333333333333335, 3.8333333333333335), sides=((0.0, 0.0, 0.0), (1.0, 2.0, 4.0)), right_angle=True)","solution":"a Math [text] that says \"$hat(beta) = (X^T X)^(-1) X^T y$\"","space":"an Axes3D (x_range=(-1.5, 2.0), y_range=(-0.5, 3.0), z_range=(-0.5, 4.5))"},"beats":[{"start":421.9515,"say":"Now use the right angle to calculate the coefficients. The fitted vector is X times beta hat, because it is a linear combination of the columns of X.","live":[],"does":[[421.9515,"heading is shown on the screen, written out."],[421.9515,"space is shown on the screen, written out."],[425.7825,"fit is shown on the screen, written out."],[429.0455,"space moves to a new place on the board."],[429.0455,"orthogonality_work is shown on the screen, written out."],[430.2875,"plane is shown on the screen, written out."],[430.2875,"column_0 is shown on the screen, written out."],[430.2875,"column_1 is shown on the screen, written out."]]},{"start":432.2805,"say":"The residual is observed y minus fitted y. Substituting the matrix form gives e equals y minus X beta hat.","live":["space","heading","plane","column_0","column_1","fit"],"does":[[432.8145,"residual is shown on the screen, written out."],[432.8145,"right_angle is shown on the screen, written out."],[434.5675,"orthogonality_work is shown on the screen, written out."]]},{"start":442.226,"say":"Perpendicular vectors have dot product zero. Since e is perpendicular to the entire column space, it is perpendicular to the intercept column, the first predictor column, and every other predictor column.","live":["space","heading","plane","column_0","column_1","fit","residual","right_angle"],"does":[[443.9565,"orthogonality_work is shown on the screen, written out."],[450.6085,"column_0 is indicated — a transient flash."],[451.89750000000004,"column_1 is indicated — a transient flash."]]},{"start":456.1315,"say":"For one column c sub j, perpendicularity reads c sub j transpose e equals zero. Write one such equation for every column.","live":null,"does":[[459.7185,"orthogonality_work (the \"c_j^T e\" part) is emphasized."],[467.1605,"orthogonality_work (the \"c_j^T e\" part) is no longer emphasized."]]},{"start":467.7605,"say":"Stack all those dot products. The transpose X transpose places the predictor columns into the rows of one multiplication, so the whole collection becomes X transpose e equals zero.","live":null,"does":[[468.0625,"orthogonality_work is shown on the screen, written out."],[476.4805,"orthogonality_work (the \"X^T e\" part) is emphasized."],[480.149,"orthogonality_work (the \"X^T e\" part) is no longer emphasized."]]},{"start":480.749,"say":"This compact zero is the algebraic form of the visible right angle. Every coordinate of X transpose e is one column's dot product with the residual.","live":null,"does":[[481.9685,"orthogonality_work is indicated — a transient flash."],[484.1975,"right_angle is indicated — a transient flash."],[490.8385,"heading is hidden from the screen — left the board."],[490.8385,"orthogonality_work is hidden from the screen — left the board."]]},{"start":491.4385,"say":"Carry that perpendicularity forward, and replace e by y minus X beta hat. It becomes X transpose times y minus X beta hat equals zero.","live":["space","plane","column_0","column_1","fit","residual","right_angle"],"does":[[491.4385,"orthogonality_fact is shown on the screen, written out."],[494.0625,"normal_work is shown on the screen, written out."]]},{"start":502.377,"say":"Distribute X transpose. The first term is X transpose y. The second is X transpose X beta hat.","live":["space","plane","column_0","column_1","fit","residual","right_angle","orthogonality_fact"],"does":[[502.71950000000004,"normal_work is shown on the screen, written out."],[505.1815,"normal_work (the \"X^T y\" part) is emphasized."],[508.1995,"normal_work (the \"X^T X hat(beta)\" part) is emphasized."],[508.1995,"normal_work (the \"X^T y\" part) is no longer emphasized."],[510.968,"normal_work (the \"X^T X hat(beta)\" part) is no longer emphasized."]]},{"start":511.568,"say":"Move the second term to the other side. We obtain X transpose X beta hat equals X transpose y. These are the normal equations.","live":null,"does":[[511.9105,"normal_work is shown on the screen, written out."],[520.2115,"A box is drawn around normal_work."]]},{"start":522.2865,"say":"They are called normal because the residual is normal, meaning perpendicular, to the predictor subspace. The name is geometric before it is algebraic.","live":null,"does":[[524.2365,"residual is indicated — a transient flash."],[526.0945,"right_angle is indicated — a transient flash."]]},{"start":532.8245000000001,"say":"The intercept equation has a familiar consequence. Its column is the all-ones vector, so its dot product with e is the sum of the residuals. With an intercept, that sum must be zero.","live":null,"does":[[537.2015,"column_0 is indicated — a transient flash."],[544.6555000000001,"orthogonality_fact is indicated — a transient flash."]]},{"start":546.126,"say":"Each predictor equation says something similar: the residual has zero dot product with that predictor column. No linear pattern available to the model remains inside the residual.","live":null,"does":[[546.7525,"column_1 is indicated — a transient flash."],[549.4115,"residual is indicated — a transient flash."]]},{"start":558.2545,"say":"If the columns are linearly independent, X transpose X is invertible. Multiplying by its inverse gives the familiar coefficient formula.","live":null,"does":[[562.3405,"solution is shown on the screen, written out."]]},{"start":568.119,"say":"But the inverse formula is not the fundamental fact. The fundamental fact is X transpose e equals zero. Numerical methods usually solve the projection problem without explicitly forming an inverse.","live":["space","plane","column_0","column_1","fit","residual","right_angle","orthogonality_fact","solution"],"does":[[569.8605,"condition is shown on the screen, written out."],[575.1315,"orthogonality_fact is indicated — a transient flash."]]},{"start":582.2555,"say":"If the columns are dependent, the fitted projection can still be defined even though the coefficient vector need not be unique. That distinction will matter when we discuss multicollinearity.","live":["space","plane","column_0","column_1","fit","residual","right_angle","orthogonality_fact","solution","condition"],"does":[[584.6595,"plane is indicated — a transient flash."],[586.8415,"solution is indicated — a transient flash."]]},{"start":594.117,"say":"So the normal equations come straight from one right angle: fitted values lie in the column space, residuals leave it perpendicularly, and every predictor column has zero residual dot product.","live":null,"does":[[596.6365000000001,"right_angle is indicated — a transient flash."],[599.1905,"plane is indicated — a transient flash."],[606.5615,"condition is hidden from the screen — left the board."],[606.5615,"normal_work is hidden from the screen — left the board."],[606.5615,"orthogonality_fact is hidden from the screen — left the board."],[606.5615,"solution is hidden from the screen — left the board."],[606.5615,"space is hidden from the screen — left the board."],[606.5615,"plane is hidden from the screen — space left the board."],[606.5615,"column_0 is hidden from the screen — space left the board."],[606.5615,"column_1 is hidden from the screen — space left the board."],[606.5615,"fit is hidden from the screen — space left the board."],[606.5615,"residual is hidden from the screen — space left the board."],[606.5615,"right_angle is hidden from the screen — space left the board."]]}]},{"title":"R Squared as a Length Ratio","start":607.6031666666667,"end":789.1013541666666,"objects":{"angle_form":"a Math [text] that says \"$R^2 = cos^2 theta$\"","decomposition":"a Derivation [text] that says \"$y_c &= y - overline(y) bold(1) \\ hat(y)_c &= hat(y) - overline(y) bold(1) \\ y_c &= hat(y)_c + e \\ y_c dot y_c &= hat(y)_c dot hat(y)_c + e dot e$\"","explained_arrow":"a Vector [green] labelled \"hat(y)_c\" drawn in triangle (end=(2.1213203436, 0.0))","heading":"a Heading that says \"Centre the Response First\"","heading_2":"a Heading that says \"The Three-Observation Example\"","numeric_0":"a Math [text] that says \"$y_c dot y_c = frac(14, 3)$\"","numeric_1":"a Math [text] that says \"$hat(y)_c dot hat(y)_c = frac(9, 2)$\"","numeric_2":"a Math [text] that says \"$e dot e = frac(1, 6)$\"","numeric_3":"a Math [text] that says \"$R^2 = frac(9/2, 14/3) = frac(27, 28) approx 0.964$\"","r_squared":"a Math [text] that says \"$R^2 = frac(hat(y)_c dot hat(y)_c, y_c dot y_c) = 1 - frac(e dot e, y_c dot y_c)$\"","residual_arrow":"a Vector [red] labelled \"e\" drawn in triangle (start=(2.1213203436, 0.0), end=(2.1213203436, 0.4082482905))","right_angle":"an Angle [yellow] drawn in triangle (vertex=(2.1213203436, 0.0), sides=((0.0, 0.0), (2.1213203436, 0.4082482905)), right_angle=True)","theta_angle":"an Angle [blue] labelled \"theta\" drawn in triangle (sides=((2.1213203436, 0.0), (2.1213203436, 0.4082482905)), radius=0.48)","total_arrow":"a Vector [yellow] labelled \"y_c\" drawn in triangle (end=(2.1213203436, 0.4082482905))","triangle":"a Figure (x_range=(-0.25, 2.75), y_range=(-0.4, 1.4), aspect=(3.0, 1.8))"},"beats":[{"start":607.6031666666667,"say":"R squared uses the same projection picture, but first we must identify what counts as total variation. With an intercept in the model, the baseline prediction is the sample mean of y.","live":[],"does":[[607.6031666666667,"heading is shown on the screen, written out."],[617.5991666666666,"decomposition is shown on the screen, written out."]]},{"start":619.7436666666666,"say":"Subtract that mean from every observed response. The centred vector y sub c measures how far the data sit from the mean-only fit.","live":["heading"],"does":[[621.2291666666666,"decomposition (the \"overline(y) bold(1)\" part) is emphasized."],[624.1201666666667,"total_arrow is shown on the screen, written out."],[629.4491666666667,"decomposition (the \"overline(y) bold(1)\" part) is no longer emphasized."]]},{"start":630.0491666666667,"say":"Centre the fitted vector in the same way. Because the regression model includes the intercept direction, the centred fitted vector remains inside the predictor subspace.","live":["heading","total_arrow"],"does":[[630.3511666666667,"decomposition is shown on the screen, written out."],[630.8151666666666,"explained_arrow is shown on the screen, written out."]]},{"start":641.0861666666666,"say":"The centred response splits into two pieces. The green piece is the variation explained by the predictors. The red piece is the residual variation left unexplained.","live":["heading","total_arrow","explained_arrow"],"does":[[642.6191666666666,"decomposition is shown on the screen, written out."],[648.5981666666667,"residual_arrow is shown on the screen, written out."]]},{"start":652.5646666666667,"say":"Projection makes those pieces perpendicular. The three vectors therefore form a right triangle: explained component, residual component, and total centred response.","live":["heading","total_arrow","explained_arrow","residual_arrow"],"does":[[657.9401666666666,"right_angle is shown on the screen, written out."]]},{"start":664.7626666666666,"say":"Pythagoras applies to their squared lengths. Total sum of squares equals regression sum of squares plus residual sum of squares.","live":["heading","total_arrow","explained_arrow","residual_arrow","right_angle"],"does":[[665.1111666666667,"decomposition is shown on the screen, written out."],[668.2221666666667,"decomposition (the \"y_c dot y_c\" part) is emphasized."],[669.8711666666667,"decomposition (the \"hat(y)_c dot hat(y)_c\" part) is emphasized."],[669.8711666666667,"decomposition (the \"y_c dot y_c\" part) is no longer emphasized."],[672.1231666666666,"decomposition (the \"e dot e\" part) is emphasized."],[672.1231666666666,"decomposition (the \"hat(y)_c dot hat(y)_c\" part) is no longer emphasized."],[673.9461666666666,"decomposition (the \"e dot e\" part) is no longer emphasized."]]},{"start":674.5461666666666,"say":"R squared is the explained squared length divided by the total squared length. Equivalently, it is one minus the unexplained squared length divided by the total.","live":null,"does":[[674.8941666666667,"r_squared is shown on the screen, written out."],[675.9621666666667,"r_squared (the \"frac(hat(y)_c dot hat(y)_c, y_c dot y_c)\" part) is emphasized."],[679.8281666666667,"r_squared (the \"1 - frac(e dot e, y_c dot y_c)\" part) is emphasized."],[679.8281666666667,"r_squared (the \"frac(hat(y)_c dot hat(y)_c, y_c dot y_c)\" part) is no longer emphasized."],[685.0181666666666,"r_squared moves to a new place on the board."],[685.0181666666666,"decomposition is hidden from the screen — left the board."],[685.0181666666666,"heading is hidden from the screen — left the board."],[685.0181666666666,"r_squared (the \"1 - frac(e dot e, y_c dot y_c)\" part) is no longer emphasized."]]},{"start":686.2181666666667,"say":"In the three-observation example, the total squared length is fourteen thirds. The explained squared length is nine halves, and the residual squared length is one sixth.","live":["r_squared","total_arrow","explained_arrow","residual_arrow","right_angle"],"does":[[689.8281666666667,"numeric_0 is shown on the screen, written out."],[692.8701666666667,"numeric_1 is shown on the screen, written out."],[695.4941666666666,"numeric_2 is shown on the screen, written out."]]},{"start":697.2436666666666,"say":"Nine halves plus one sixth is fourteen thirds, exactly as the right triangle requires. The resulting R squared is twenty-seven over twenty-eight, about zero point nine six four.","live":["r_squared","total_arrow","explained_arrow","residual_arrow","right_angle","numeric_0","numeric_1","numeric_2"],"does":[[703.8841666666667,"numeric_3 is shown on the screen, written out."],[708.5521666666666,"A box is drawn around numeric_3."]]},{"start":709.9996666666666,"say":"There is also an angle interpretation. Let theta be the angle between the centred response and its fitted projection. Then R squared is cosine squared theta.","live":["r_squared","total_arrow","explained_arrow","residual_arrow","right_angle","numeric_0","numeric_1","numeric_2","numeric_3"],"does":[[711.1141666666666,"theta_angle is shown on the screen, written out."],[719.6351666666667,"angle_form is shown on the screen, written out."]]},{"start":722.0116666666667,"say":"A small angle means the response lies close to the model subspace, so the projection keeps nearly all its length. A large angle means a larger perpendicular residual.","live":["r_squared","angle_form","total_arrow","explained_arrow","residual_arrow","right_angle","numeric_0","numeric_1","numeric_2","numeric_3","theta_angle"],"does":[[723.6371666666666,"total_arrow is indicated — a transient flash."],[731.6011666666667,"residual_arrow is indicated — a transient flash."]]},{"start":733.2346666666666,"say":"With ordinary least squares and an intercept, this training-sample ratio lies between zero and one. Zero means the predictors improve nothing beyond the mean. One means every residual is zero.","live":null,"does":[[737.5891666666666,"r_squared is indicated — a transient flash."],[740.4681666666667,"residual_arrow is indicated — a transient flash."]]},{"start":747.5111666666667,"say":"The intercept condition matters. Without the mean direction in the model, this familiar centred Pythagorean decomposition need not hold, and the usual interpretation of R squared needs care.","live":null,"does":[[750.7851666666667,"total_arrow is indicated — a transient flash."]]},{"start":761.1951666666666,"say":"R squared measures alignment with the fitted subspace, not causation, and not whether the model will predict new data well. It is a geometric summary of in-sample explained squared length.","live":null,"does":[[762.8091666666667,"angle_form is indicated — a transient flash."]]},{"start":774.3226666666667,"say":"The projection picture has now given us residual minimization, the normal equations, and R squared. One final geometric question remains: what if the predictor directions themselves almost collapse together?","live":null,"does":[[788.0596875,"angle_form is hidden from the screen — left the board."],[788.0596875,"numeric_0 is hidden from the screen — left the board."],[788.0596875,"numeric_1 is hidden from the screen — left the board."],[788.0596875,"numeric_2 is hidden from the screen — left the board."],[788.0596875,"numeric_3 is hidden from the screen — left the board."],[788.0596875,"r_squared is hidden from the screen — left the board."]]}]},{"title":"When Predictor Directions Nearly Align","start":789.1013541666666,"end":1081.7938333333332,"objects":{"basis_angle":"an Angle [yellow] labelled \"theta\" drawn in figure (sides=((1.0, 0.0), (cos(theta), sin(theta))), radius=0.34)","coefficients":"a Math [text] that says \"$hat(beta) approx vec(-0.38, 1.15)$\"","component_1":"a Vector [blue] labelled \"hat(beta)_1 x_1\" drawn in figure (end=((0.2 - ((target_y / sin(theta)) * cos(theta))), 0.0))","component_2":"a Vector [green] labelled \"hat(beta)_2 x_2\" drawn in figure (start=((0.2 - ((target_y / sin(theta)) * cos(theta))), 0.0), end=(0.2, <VariableNumber target_y = 1.02>))","condition_number":"a Math [text] that says \"$kappa(X) = sqrt(frac(1+cos theta, 1-cos theta))$\"","determinant":"a Math [text] that says \"$d = 1 - cos^2 theta = sin^2 theta$\"","exact_dependence":"a Math [text] that says \"$theta=0 arrow.r x_2=x_1 arrow.r upright(\"rank\")(X)<2$\"","figure":"a Figure (x_range=(-6.25, 1.25), y_range=(-0.4, 1.7), aspect=(7.5, 2.1))","fit_arrow":"a Vector [red] labelled \"hat(y)\" drawn in figure (end=(0.2, <VariableNumber target_y = 1.02>))","gram":"a Math [text] that says \"$X^T X = mat(1, cos theta; cos theta, 1)$\"","heading":"a Heading that says \"Two Predictor Directions\"","heading_2":"a Heading that says \"A Small Change in the Target\"","heading_3":"a Heading that says \"Why the Coordinates Become Unstable\"","heading_4":"a Heading that says \"What Multicollinearity Means\"","item_1":"a Text [text] that says \"Nearly parallel columns provide almost the same direction.\"","item_2":"a Text [text] that says \"Large opposite coefficients can produce one modest fitted vector.\"","item_3":"a Text [text] that says \"Small data changes can cause much larger coefficient changes.\"","item_4":"a Text [text] that says \"Predictions may remain stable while individual effects are unstable.\"","numeric_det":"a Math [text] that says \"$theta=10 degree: quad d approx 0.030$\"","numeric_kappa":"a Math [text] that says \"$kappa(X) approx 11.4$\"","perturb_beta":"a Math [text] that says \"$hat(beta) approx vec(-5.47, 5.76)$\"","perturb_fit":"a Math [text] that says \"$hat(y) = vec(0.20, 1.00)$\"","target_y":"a VariableNumber (initial_value=1.0)","theta":"a VariableNumber (initial_value=1.0471975511965976)","x_1":"a Vector [blue] labelled \"x_1\" drawn in figure","x_2":"a Vector [green] labelled \"x_2\" drawn in figure (end=(cos(theta), sin(theta)))"},"beats":[{"start":789.1013541666666,"say":"To see multicollinearity, first remove the intercept direction by centring, and scale two predictor columns to unit length. Then their relationship is summarized by the angle theta between their vectors.","live":[],"does":[[789.1013541666666,"heading is shown on the screen, written out."],[793.8733541666667,"x_1 is shown on the screen, written out."],[793.8733541666667,"x_2 is shown on the screen, written out."],[798.6103541666666,"basis_angle is shown on the screen, written out."],[800.0723541666666,"figure is shown on the screen, written out."]]},{"start":801.7523541666666,"say":"At sixty degrees, the blue and green predictors provide clearly different directions. Together they span the plane, and a fitted vector has coordinates beta one and beta two in this predictor basis.","live":["figure","heading","x_1","x_2","basis_angle"],"does":[[809.2523541666666,"fit_arrow is shown on the screen, written out."]]},{"start":814.2058541666667,"say":"Draw the coefficient contributions tip to tail. The blue contribution first moves along x one. The green contribution then moves along x two, and their sum lands exactly at the red fitted vector.","live":["figure","heading","x_1","x_2","basis_angle","fit_arrow"],"does":[[817.9903541666666,"component_1 is shown on the screen, written out."],[821.0783541666666,"component_2 is shown on the screen, written out."],[825.6063541666666,"coefficients is shown on the screen, written out."]]},{"start":827.5998541666667,"say":"These coefficients are coordinates, not intrinsic lengths. Their values depend on how well the chosen predictor directions form a coordinate system.","live":["figure","coefficients","heading","x_1","x_2","basis_angle","fit_arrow","component_1","component_2"],"does":[[829.0973541666666,"component_1 is indicated — a transient flash."],[829.0973541666666,"component_2 is indicated — a transient flash."]]},{"start":837.0228541666667,"say":"Now make the green predictor more like the blue one. At thirty degrees, the span is still the whole plane, and the fitted vector has not moved. But the two coefficient contributions must grow and cancel more strongly.","live":null,"does":[[840.6103541666666,"x_2 is redrawn as the numbers it depends on change."],[840.6103541666666,"basis_angle is redrawn as the numbers it depends on change."],[840.6103541666666,"component_1 is redrawn as the numbers it depends on change."],[840.6103541666666,"component_2 is redrawn as the numbers it depends on change."],[840.6103541666666,"theta ticks to 0.5235987755982988."],[849.6663541666667,"coefficients becomes \"$hat(beta) approx vec(-1.53, 2.00)$\"."]]},{"start":850.2663541666666,"say":"Bring the angle down to ten degrees. The predictor arrows are now nearly parallel. To reach the same modest red vector, one long component runs left and the other almost cancels it while climbing to the target.","live":null,"does":[[850.5213541666666,"x_2 is redrawn as the numbers it depends on change."],[850.5213541666666,"basis_angle is redrawn as the numbers it depends on change."],[850.5213541666666,"component_1 is redrawn as the numbers it depends on change."],[850.5213541666666,"component_2 is redrawn as the numbers it depends on change."],[850.5213541666666,"theta ticks to 0.17453292519943295."],[859.9143541666666,"component_1 is indicated — a transient flash."],[861.3763541666666,"component_2 is indicated — a transient flash."],[863.4088541666666,"coefficients becomes \"$hat(beta) approx vec(-5.47, 5.76)$\"."]]},{"start":864.0088541666667,"say":"This is multicollinearity. Two columns of X supply nearly the same direction in observation space, so the model has difficulty deciding how much of the fit to assign to each one.","live":null,"does":[[866.3193541666666,"x_1 is indicated — a transient flash."],[868.3153541666666,"x_2 is indicated — a transient flash."]]},{"start":875.5098541666666,"say":"The fitted vector itself can still be perfectly clear. What becomes unstable is its description in this nearly collapsed coordinate system.","live":null,"does":[[875.9863541666666,"fit_arrow is indicated — a transient flash."],[880.7113541666666,"component_1 is indicated — a transient flash."],[883.9503541666667,"figure moves to a new place on the board."],[883.9503541666667,"coefficients is hidden from the screen — left the board."],[883.9503541666667,"heading is hidden from the screen — left the board."]]},{"start":885.1503541666666,"say":"Here is a small perturbation. Raise the fitted target's second coordinate from one point zero zero to one point zero two, only two hundredths.","live":["figure","x_1","x_2","basis_angle","fit_arrow","component_1","component_2"],"does":[[885.1503541666666,"perturb_fit is shown on the screen, written out."],[885.1503541666666,"perturb_beta is shown on the screen, written out."],[887.8783541666667,"fit_arrow is redrawn as the numbers it depends on change."],[887.8783541666667,"component_1 is redrawn as the numbers it depends on change."],[887.8783541666667,"component_2 is redrawn as the numbers it depends on change."],[887.8783541666667,"target_y ticks to 1.02."]]},{"start":895.4098541666666,"say":"The red endpoint barely moves, but each coefficient changes by about zero point one one. The coordinate change is almost six times the size of the data perturbation, with opposite signs.","live":["figure","x_1","x_2","basis_angle","fit_arrow","component_1","component_2","perturb_fit","perturb_beta"],"does":[[897.9643541666667,"perturb_beta becomes \"$hat(beta) approx vec(-5.59, 5.87)$\"."],[905.3253541666667,"component_1 is indicated — a transient flash."],[905.8123541666666,"component_2 is indicated — a transient flash."],[906.6833541666666,"figure moves to a new place on the board."],[906.6833541666666,"perturb_beta is hidden from the screen — left the board."],[906.6833541666666,"perturb_fit is hidden from the screen — left the board."]]},{"start":907.8833541666667,"say":"The algebra reports the same geometry. For standardized columns, X transpose X has ones on the diagonal and cosine theta off the diagonal.","live":["figure","x_1","x_2","basis_angle","fit_arrow","component_1","component_2"],"does":[[907.8833541666667,"gram is shown on the screen, written out."],[916.0453541666667,"gram (the \"cos theta\" part) is emphasized."],[918.4368541666666,"gram (the \"cos theta\" part) is no longer emphasized."]]},{"start":919.0368541666667,"say":"Its determinant is one minus cosine squared theta, which is sine squared theta. As the predictor angle shrinks, this determinant approaches zero.","live":["figure","x_1","x_2","basis_angle","fit_arrow","component_1","component_2","gram"],"does":[[919.6293541666666,"determinant is shown on the screen, written out."],[922.6823541666666,"determinant (the \"sin^2 theta\" part) is emphasized."],[928.7773541666667,"determinant (the \"sin^2 theta\" part) is no longer emphasized."]]},{"start":929.3773541666667,"say":"At exactly zero degrees the two columns coincide. The span collapses from a plane to a line, X loses rank, and infinitely many coefficient pairs can describe the same fitted vector.","live":["figure","x_1","x_2","basis_angle","fit_arrow","component_1","component_2","gram","determinant"],"does":[[930.4103541666666,"exact_dependence is shown on the screen, written out."],[931.8733541666667,"x_1 is indicated — a transient flash."],[931.8733541666667,"x_2 is indicated — a transient flash."]]},{"start":942.6093541666667,"say":"Near zero, the inverse still exists in exact arithmetic, but it must magnify the thin direction enormously. That sensitivity is called ill conditioning.","live":["figure","x_1","x_2","basis_angle","fit_arrow","component_1","component_2","exact_dependence","gram","determinant"],"does":[[950.4923541666666,"condition_number is shown on the screen, written out."]]},{"start":953.5308541666666,"say":"For two standardized predictors, this condition number is the square root of one plus cosine theta over one minus cosine theta. At ten degrees, the Gram determinant is about zero point zero three and the condition number is already about eleven point four.","live":["figure","x_1","x_2","basis_angle","fit_arrow","component_1","component_2","exact_dependence","gram","determinant","condition_number"],"does":[[956.0373541666667,"numeric_kappa is shown on the screen, written out."],[963.4563541666666,"numeric_det is shown on the screen, written out."],[968.9363541666667,"numeric_kappa (the \"11.4\" part) is indicated — a transient flash."]]},{"start":970.3378541666666,"say":"A condition number near one means the predictor directions are well separated. A large value means small changes in y, rounding, or sampling can create much larger changes in the estimated coefficients.","live":["figure","x_1","x_2","basis_angle","fit_arrow","component_1","component_2","exact_dependence","gram","determinant","condition_number","numeric_det","numeric_kappa"],"does":[[975.6093541666667,"condition_number is indicated — a transient flash."]]},{"start":983.8948541666666,"say":"This does not automatically make the fitted values unstable. Many different large, cancelling coefficient pairs can land at nearly the same projection point. Prediction within the observed predictor pattern may therefore look steady while individual coefficient interpretations do not.","live":null,"does":[[985.7633541666667,"fit_arrow is indicated — a transient flash."],[988.9443541666666,"component_1 is indicated — a transient flash."],[988.9443541666666,"component_2 is indicated — a transient flash."],[1000.1013541666666,"condition_number is hidden from the screen — left the board."],[1000.1013541666666,"determinant is hidden from the screen — left the board."],[1000.1013541666666,"exact_dependence is hidden from the screen — left the board."],[1000.1013541666666,"figure is hidden from the screen — left the board."],[1000.1013541666666,"x_1 is hidden from the screen — figure left the board."],[1000.1013541666666,"x_2 is hidden from the screen — figure left the board."],[1000.1013541666666,"basis_angle is hidden from the screen — figure left the board."],[1000.1013541666666,"fit_arrow is hidden from the screen — figure left the board."],[1000.1013541666666,"component_1 is hidden from the screen — figure left the board."],[1000.1013541666666,"component_2 is hidden from the screen — figure left the board."],[1000.1013541666666,"gram is hidden from the screen — left the board."],[1000.1013541666666,"numeric_det is hidden from the screen — left the board."],[1000.1013541666666,"numeric_kappa is hidden from the screen — left the board."]]},{"start":1001.3013541666667,"say":"The first lesson is geometric: nearly parallel columns provide almost the same direction.","live":[],"does":[[1001.8353541666667,"item_1 is shown on the screen, written out."]]},{"start":1007.9268541666667,"say":"The second is about coordinates: reaching one moderate fit may require large contributions with opposite signs.","live":["item_1"],"does":[[1008.4613541666666,"item_2 is shown on the screen, written out."]]},{"start":1015.6903541666666,"say":"The third is statistical: a small change in the sample can move those large coefficients substantially, increasing their uncertainty.","live":["item_1","item_2"],"does":[[1016.1893541666666,"item_3 is shown on the screen, written out."]]},{"start":1024.3593541666667,"say":"And the fourth separates two questions. The model may predict a combined effect reasonably while the separate effects of the correlated predictors remain poorly determined.","live":["item_1","item_2","item_3"],"does":[[1024.9863541666666,"item_4 is shown on the screen, written out."]]},{"start":1035.3033541666666,"say":"Possible responses include collecting data where the predictors vary more independently, combining redundant predictors, choosing one of them, or using a regularized fit. Each response changes how the thin direction is handled, so it should follow the scientific purpose of the model.","live":["item_1","item_2","item_3","item_4"],"does":[]},{"start":1053.7478541666667,"say":"The whole lecture is now one geometric story. Data points create vertical residuals. Least squares chooses the shortest residual vector in observation space. Perpendicularity gives the normal equations.","live":null,"does":[]},{"start":1068.6858541666666,"say":"Pythagoras turns the same projection into R squared. And when predictor directions nearly align, the spanning coordinates become ill conditioned even if the projected fit barely changes.","live":null,"does":[[1080.7521666666667,"item_1 is hidden from the screen — left the board."],[1080.7521666666667,"item_2 is hidden from the screen — left the board."],[1080.7521666666667,"item_3 is hidden from the screen — left the board."],[1080.7521666666667,"item_4 is hidden from the screen — left the board."]]}]}]},"durationSeconds":1082,"chapters":[{"title":"Residuals and the Best Line","startSeconds":0,"narration":"We begin with five observed pairs. The horizontal coordinate is the predictor x, and the vertical coordinate is the response y. A regression line will summarize how the response changes as x changes. Here is one candidate line. For this example its height at the centre is fixed at three, and m controls its tilt. We have not yet chosen m. At each observed x value, the fitted line predicts a height. The vertical segment from that fitted height to the observed point is the residual. The sign records direction. A point above the line has a positive residual, and a point below it has a negative residual. But a good fit should penalize a large miss in either direction. So square every residual and add the squares. This sum of squared errors is nonnegative, it treats equal misses above and below equally, and large misses count much more than small ones. Notice what we are minimizing. These are vertical differences in the response coordinate. Ordinary regression of y on x does not use the shortest distance to the line; it holds each observed x fixed. With a horizontal line, m is zero. The five residuals have squared sum sixteen. The line is clearly missing the upward pattern in the data. To choose the tilt without calculus, let us watch every candidate and record its squared error. The same live value of m will drive both pictures. On the right, the red curve records the sum of squared errors for every slope in view. The yellow dot begins at slope zero and error sixteen. Tilt the fitted line upward to slope zero point six. The residual pattern tightens, and the squared error falls from sixteen to five point two. Continue to slope one point two. Several vertical misses are now small, and the dot reaches the bottom of the error curve. The squared error is one point six. Go past that slope to one point eight. Some residuals change sign, but their squares grow again. The error rises back to five point two. Return to the bottom. The line with slope one point two has the smallest sum of squared vertical residuals among this whole family. That is the least-squares fit. The curve on the right is not an appeal to differentiation. It is a geometric record of what the yellow segments on the left are doing. Tilting away from the best line makes their combined squared length grow. For this centred predictor, the intercept stayed at the mean response, three, while the slope changed. In a general fit, both intercept and slope are chosen together to minimize the same squared residual total. This data picture explains what least squares measures. To understand why the optimum has a clean algebraic form, we now change the geometry completely."},{"title":"The Fit as a Shadow","startSeconds":195.44677083333335,"narration":"Now replace the data plot by observation space. To draw it, we use a miniature data set with three observations. Three observations give three coordinates, so the whole construction fits in ordinary three-dimensional space. The first row records response one, the second response two, and the third response four. Read down the response column, not across a plotted axis. Stack those three responses into one vector y: one, two, four. Each coordinate belongs to one observation, so this single arrow represents the entire response data set. The design matrix also has one coordinate per observation in each column. Its intercept column is one, one, one, because the intercept contributes equally to every fitted response. The predictor column is minus one, zero, one. It records the three predictor values after centring. This green arrow is that whole column. A coefficient beta zero scales the blue column. A coefficient beta one scales the green column. Adding the two scaled arrows produces every fitted response vector this model is capable of making. Those linear combinations fill a plane through the origin. This is the column space of X. Changing the regression coefficients moves the fitted vector somewhere inside this plane, but never outside it. The picture is three-dimensional only because this example has three observations. With n observations, y lives in n-dimensional observation space. With p predictors and an intercept, the columns span a subspace of dimension at most p plus one. The yellow response vector usually misses the predictor plane. An exact fit would put y inside the column space, but these three observations do not lie on one exact regression line. Among all points in the plane, choose the one closest to y. The green arrow reaches that point. It is the fitted-value vector, hat y. Its three coordinates are the three fitted responses. Returning to the original data plot would place those fitted values on the regression line at the three observed predictor values. What remains is the red vector from hat y to y. Coordinate by coordinate, it contains observed response minus fitted response, so it is exactly the residual vector e. The closest route from a point to a plane is perpendicular to the plane. Therefore the residual stands at a right angle to every direction the predictor columns span. This is least squares in one sentence. The fitted vector is the orthogonal projection of y onto the column space of X. The word shadow is useful. Imagine light travelling perpendicular to the predictor plane. The response arrow casts its endpoint onto the plane, and the shadow point is hat y. The red leftover is perpendicular to the shadow plane. Pythagoras then says no other point in the plane can be closer to y. Moving away from hat y adds an in-plane displacement at right angles to the residual. That geometric fact is the minimization. We did not differentiate a sum of squares. We identified the unique shortest vector from y to the model subspace. The same picture survives with many observations and many predictors. Only the dimension changes. The response is one point, the possible fits form a column space, and least squares drops a perpendicular to it."},{"title":"Perpendicularity and the Normal Equations","startSeconds":421.9515,"narration":"Now use the right angle to calculate the coefficients. The fitted vector is X times beta hat, because it is a linear combination of the columns of X. The residual is observed y minus fitted y. Substituting the matrix form gives e equals y minus X beta hat. Perpendicular vectors have dot product zero. Since e is perpendicular to the entire column space, it is perpendicular to the intercept column, the first predictor column, and every other predictor column. For one column c sub j, perpendicularity reads c sub j transpose e equals zero. Write one such equation for every column. Stack all those dot products. The transpose X transpose places the predictor columns into the rows of one multiplication, so the whole collection becomes X transpose e equals zero. This compact zero is the algebraic form of the visible right angle. Every coordinate of X transpose e is one column's dot product with the residual. Carry that perpendicularity forward, and replace e by y minus X beta hat. It becomes X transpose times y minus X beta hat equals zero. Distribute X transpose. The first term is X transpose y. The second is X transpose X beta hat. Move the second term to the other side. We obtain X transpose X beta hat equals X transpose y. These are the normal equations. They are called normal because the residual is normal, meaning perpendicular, to the predictor subspace. The name is geometric before it is algebraic. The intercept equation has a familiar consequence. Its column is the all-ones vector, so its dot product with e is the sum of the residuals. With an intercept, that sum must be zero. Each predictor equation says something similar: the residual has zero dot product with that predictor column. No linear pattern available to the model remains inside the residual. If the columns are linearly independent, X transpose X is invertible. Multiplying by its inverse gives the familiar coefficient formula. But the inverse formula is not the fundamental fact. The fundamental fact is X transpose e equals zero. Numerical methods usually solve the projection problem without explicitly forming an inverse. If the columns are dependent, the fitted projection can still be defined even though the coefficient vector need not be unique. That distinction will matter when we discuss multicollinearity. So the normal equations come straight from one right angle: fitted values lie in the column space, residuals leave it perpendicularly, and every predictor column has zero residual dot product."},{"title":"R Squared as a Length Ratio","startSeconds":607.6031666666667,"narration":"R squared uses the same projection picture, but first we must identify what counts as total variation. With an intercept in the model, the baseline prediction is the sample mean of y. Subtract that mean from every observed response. The centred vector y sub c measures how far the data sit from the mean-only fit. Centre the fitted vector in the same way. Because the regression model includes the intercept direction, the centred fitted vector remains inside the predictor subspace. The centred response splits into two pieces. The green piece is the variation explained by the predictors. The red piece is the residual variation left unexplained. Projection makes those pieces perpendicular. The three vectors therefore form a right triangle: explained component, residual component, and total centred response. Pythagoras applies to their squared lengths. Total sum of squares equals regression sum of squares plus residual sum of squares. R squared is the explained squared length divided by the total squared length. Equivalently, it is one minus the unexplained squared length divided by the total. In the three-observation example, the total squared length is fourteen thirds. The explained squared length is nine halves, and the residual squared length is one sixth. Nine halves plus one sixth is fourteen thirds, exactly as the right triangle requires. The resulting R squared is twenty-seven over twenty-eight, about zero point nine six four. There is also an angle interpretation. Let theta be the angle between the centred response and its fitted projection. Then R squared is cosine squared theta. A small angle means the response lies close to the model subspace, so the projection keeps nearly all its length. A large angle means a larger perpendicular residual. With ordinary least squares and an intercept, this training-sample ratio lies between zero and one. Zero means the predictors improve nothing beyond the mean. One means every residual is zero. The intercept condition matters. Without the mean direction in the model, this familiar centred Pythagorean decomposition need not hold, and the usual interpretation of R squared needs care. R squared measures alignment with the fitted subspace, not causation, and not whether the model will predict new data well. It is a geometric summary of in-sample explained squared length. The projection picture has now given us residual minimization, the normal equations, and R squared. One final geometric question remains: what if the predictor directions themselves almost collapse together?"},{"title":"When Predictor Directions Nearly Align","startSeconds":789.1013541666666,"narration":"To see multicollinearity, first remove the intercept direction by centring, and scale two predictor columns to unit length. Then their relationship is summarized by the angle theta between their vectors. At sixty degrees, the blue and green predictors provide clearly different directions. Together they span the plane, and a fitted vector has coordinates beta one and beta two in this predictor basis. Draw the coefficient contributions tip to tail. The blue contribution first moves along x one. The green contribution then moves along x two, and their sum lands exactly at the red fitted vector. These coefficients are coordinates, not intrinsic lengths. Their values depend on how well the chosen predictor directions form a coordinate system. Now make the green predictor more like the blue one. At thirty degrees, the span is still the whole plane, and the fitted vector has not moved. But the two coefficient contributions must grow and cancel more strongly. Bring the angle down to ten degrees. The predictor arrows are now nearly parallel. To reach the same modest red vector, one long component runs left and the other almost cancels it while climbing to the target. This is multicollinearity. Two columns of X supply nearly the same direction in observation space, so the model has difficulty deciding how much of the fit to assign to each one. The fitted vector itself can still be perfectly clear. What becomes unstable is its description in this nearly collapsed coordinate system. Here is a small perturbation. Raise the fitted target's second coordinate from one point zero zero to one point zero two, only two hundredths. The red endpoint barely moves, but each coefficient changes by about zero point one one. The coordinate change is almost six times the size of the data perturbation, with opposite signs. The algebra reports the same geometry. For standardized columns, X transpose X has ones on the diagonal and cosine theta off the diagonal. Its determinant is one minus cosine squared theta, which is sine squared theta. As the predictor angle shrinks, this determinant approaches zero. At exactly zero degrees the two columns coincide. The span collapses from a plane to a line, X loses rank, and infinitely many coefficient pairs can describe the same fitted vector. Near zero, the inverse still exists in exact arithmetic, but it must magnify the thin direction enormously. That sensitivity is called ill conditioning. For two standardized predictors, this condition number is the square root of one plus cosine theta over one minus cosine theta. At ten degrees, the Gram determinant is about zero point zero three and the condition number is already about eleven point four. A condition number near one means the predictor directions are well separated. A large value means small changes in y, rounding, or sampling can create much larger changes in the estimated coefficients. This does not automatically make the fitted values unstable. Many different large, cancelling coefficient pairs can land at nearly the same projection point. Prediction within the observed predictor pattern may therefore look steady while individual coefficient interpretations do not. The first lesson is geometric: nearly parallel columns provide almost the same direction. The second is about coordinates: reaching one moderate fit may require large contributions with opposite signs. The third is statistical: a small change in the sample can move those large coefficients substantially, increasing their uncertainty. And the fourth separates two questions. The model may predict a combined effect reasonably while the separate effects of the correlated predictors remain poorly determined. Possible responses include collecting data where the predictors vary more independently, combining redundant predictors, choosing one of them, or using a regularized fit. Each response changes how the thin direction is handled, so it should follow the scientific purpose of the model. The whole lecture is now one geometric story. Data points create vertical residuals. Least squares chooses the shortest residual vector in observation space. Perpendicularity gives the normal equations. Pythagoras turns the same projection into R squared. And when predictor directions nearly align, the spanning coordinates become ill conditioned even if the projected fit barely changes."}]}}