# Linear Regression as Geometry > Linear regression developed as a geometric projection. The lecture begins with observed data, vertical residuals, and a fitted line whose squared error visibly falls to a minimum as its slope changes. It then moves into observation space, where predictor columns span a subspace, fitted values are the orthogonal shadow of the response, and the residual is the perpendicular component. That right angle yields the normal equations directly. The same decomposition gives R squared as a ratio of squared lengths, while nearly parallel predictor columns reveal multicollinearity, unstable coefficients, and ill conditioning. - Canonical watch page: [Linear Regression as Geometry](https://academa.ai/lectures/least-squares-is-a-projection) - Publisher: [Academa, Inc.](https://academa.ai) - Subject: Statistics - Published: 2026-08-29T00:59:10.000Z - Updated: 2026-08-29T00:59:10.000Z - Duration: PT1082S (18 minutes 2 seconds) - Chapters: 5 - Views: 2 - Language: en-US - Access: Free - Video stream: [HLS content](https://academa.ai/media/l/01M14V05J1Q2H560M850WHHDZA/1/dark/master.m3u8) - Audiovisual record: [Semantic JSON](https://academa.ai/media/l/01M14V05J1Q2H560M850WHHDZA/1/semantic.json) - Thumbnail: [Image](https://academa.ai/media/l/01M14V05J1Q2H560M850WHHDZA/1/dark/poster.jpg) ## Description Learn linear regression through residual geometry, orthogonal projection, normal equations, R squared, and nearly parallel predictors. ## Chapters - [00:00–03:15.447 · Residuals and the Best Line](https://academa.ai/lectures/least-squares-is-a-projection?t=0) - [03:15.447–07:1.952 · The Fit as a Shadow](https://academa.ai/lectures/least-squares-is-a-projection?t=195.44677083333335) - [07:1.952–10:7.603 · Perpendicularity and the Normal Equations](https://academa.ai/lectures/least-squares-is-a-projection?t=421.9515) - [10:7.603–13:9.101 · R Squared as a Length Ratio](https://academa.ai/lectures/least-squares-is-a-projection?t=607.6031666666667) - [13:9.101–18:02 · When Predictor Directions Nearly Align](https://academa.ai/lectures/least-squares-is-a-projection?t=789.1013541666666) ## Transcript ### [00:00 · Residuals and the Best Line](https://academa.ai/lectures/least-squares-is-a-projection?t=0) We begin with five observed pairs. The horizontal coordinate is the predictor x, and the vertical coordinate is the response y. A regression line will summarize how the response changes as x changes. Here is one candidate line. For this example its height at the centre is fixed at three, and m controls its tilt. We have not yet chosen m. At each observed x value, the fitted line predicts a height. The vertical segment from that fitted height to the observed point is the residual. The sign records direction. A point above the line has a positive residual, and a point below it has a negative residual. But a good fit should penalize a large miss in either direction. So square every residual and add the squares. This sum of squared errors is nonnegative, it treats equal misses above and below equally, and large misses count much more than small ones. Notice what we are minimizing. These are vertical differences in the response coordinate. Ordinary regression of y on x does not use the shortest distance to the line; it holds each observed x fixed. With a horizontal line, m is zero. The five residuals have squared sum sixteen. The line is clearly missing the upward pattern in the data. To choose the tilt without calculus, let us watch every candidate and record its squared error. The same live value of m will drive both pictures. On the right, the red curve records the sum of squared errors for every slope in view. The yellow dot begins at slope zero and error sixteen. Tilt the fitted line upward to slope zero point six. The residual pattern tightens, and the squared error falls from sixteen to five point two. Continue to slope one point two. Several vertical misses are now small, and the dot reaches the bottom of the error curve. The squared error is one point six. Go past that slope to one point eight. Some residuals change sign, but their squares grow again. The error rises back to five point two. Return to the bottom. The line with slope one point two has the smallest sum of squared vertical residuals among this whole family. That is the least-squares fit. The curve on the right is not an appeal to differentiation. It is a geometric record of what the yellow segments on the left are doing. Tilting away from the best line makes their combined squared length grow. For this centred predictor, the intercept stayed at the mean response, three, while the slope changed. In a general fit, both intercept and slope are chosen together to minimize the same squared residual total. This data picture explains what least squares measures. To understand why the optimum has a clean algebraic form, we now change the geometry completely. ### [03:15.447 · The Fit as a Shadow](https://academa.ai/lectures/least-squares-is-a-projection?t=195.44677083333335) Now replace the data plot by observation space. To draw it, we use a miniature data set with three observations. Three observations give three coordinates, so the whole construction fits in ordinary three-dimensional space. The first row records response one, the second response two, and the third response four. Read down the response column, not across a plotted axis. Stack those three responses into one vector y: one, two, four. Each coordinate belongs to one observation, so this single arrow represents the entire response data set. The design matrix also has one coordinate per observation in each column. Its intercept column is one, one, one, because the intercept contributes equally to every fitted response. The predictor column is minus one, zero, one. It records the three predictor values after centring. This green arrow is that whole column. A coefficient beta zero scales the blue column. A coefficient beta one scales the green column. Adding the two scaled arrows produces every fitted response vector this model is capable of making. Those linear combinations fill a plane through the origin. This is the column space of X. Changing the regression coefficients moves the fitted vector somewhere inside this plane, but never outside it. The picture is three-dimensional only because this example has three observations. With n observations, y lives in n-dimensional observation space. With p predictors and an intercept, the columns span a subspace of dimension at most p plus one. The yellow response vector usually misses the predictor plane. An exact fit would put y inside the column space, but these three observations do not lie on one exact regression line. Among all points in the plane, choose the one closest to y. The green arrow reaches that point. It is the fitted-value vector, hat y. Its three coordinates are the three fitted responses. Returning to the original data plot would place those fitted values on the regression line at the three observed predictor values. What remains is the red vector from hat y to y. Coordinate by coordinate, it contains observed response minus fitted response, so it is exactly the residual vector e. The closest route from a point to a plane is perpendicular to the plane. Therefore the residual stands at a right angle to every direction the predictor columns span. This is least squares in one sentence. The fitted vector is the orthogonal projection of y onto the column space of X. The word shadow is useful. Imagine light travelling perpendicular to the predictor plane. The response arrow casts its endpoint onto the plane, and the shadow point is hat y. The red leftover is perpendicular to the shadow plane. Pythagoras then says no other point in the plane can be closer to y. Moving away from hat y adds an in-plane displacement at right angles to the residual. That geometric fact is the minimization. We did not differentiate a sum of squares. We identified the unique shortest vector from y to the model subspace. The same picture survives with many observations and many predictors. Only the dimension changes. The response is one point, the possible fits form a column space, and least squares drops a perpendicular to it. ### [07:1.952 · Perpendicularity and the Normal Equations](https://academa.ai/lectures/least-squares-is-a-projection?t=421.9515) Now use the right angle to calculate the coefficients. The fitted vector is X times beta hat, because it is a linear combination of the columns of X. The residual is observed y minus fitted y. Substituting the matrix form gives e equals y minus X beta hat. Perpendicular vectors have dot product zero. Since e is perpendicular to the entire column space, it is perpendicular to the intercept column, the first predictor column, and every other predictor column. For one column c sub j, perpendicularity reads c sub j transpose e equals zero. Write one such equation for every column. Stack all those dot products. The transpose X transpose places the predictor columns into the rows of one multiplication, so the whole collection becomes X transpose e equals zero. This compact zero is the algebraic form of the visible right angle. Every coordinate of X transpose e is one column's dot product with the residual. Carry that perpendicularity forward, and replace e by y minus X beta hat. It becomes X transpose times y minus X beta hat equals zero. Distribute X transpose. The first term is X transpose y. The second is X transpose X beta hat. Move the second term to the other side. We obtain X transpose X beta hat equals X transpose y. These are the normal equations. They are called normal because the residual is normal, meaning perpendicular, to the predictor subspace. The name is geometric before it is algebraic. The intercept equation has a familiar consequence. Its column is the all-ones vector, so its dot product with e is the sum of the residuals. With an intercept, that sum must be zero. Each predictor equation says something similar: the residual has zero dot product with that predictor column. No linear pattern available to the model remains inside the residual. If the columns are linearly independent, X transpose X is invertible. Multiplying by its inverse gives the familiar coefficient formula. But the inverse formula is not the fundamental fact. The fundamental fact is X transpose e equals zero. Numerical methods usually solve the projection problem without explicitly forming an inverse. If the columns are dependent, the fitted projection can still be defined even though the coefficient vector need not be unique. That distinction will matter when we discuss multicollinearity. So the normal equations come straight from one right angle: fitted values lie in the column space, residuals leave it perpendicularly, and every predictor column has zero residual dot product. ### [10:7.603 · R Squared as a Length Ratio](https://academa.ai/lectures/least-squares-is-a-projection?t=607.6031666666667) R squared uses the same projection picture, but first we must identify what counts as total variation. With an intercept in the model, the baseline prediction is the sample mean of y. Subtract that mean from every observed response. The centred vector y sub c measures how far the data sit from the mean-only fit. Centre the fitted vector in the same way. Because the regression model includes the intercept direction, the centred fitted vector remains inside the predictor subspace. The centred response splits into two pieces. The green piece is the variation explained by the predictors. The red piece is the residual variation left unexplained. Projection makes those pieces perpendicular. The three vectors therefore form a right triangle: explained component, residual component, and total centred response. Pythagoras applies to their squared lengths. Total sum of squares equals regression sum of squares plus residual sum of squares. R squared is the explained squared length divided by the total squared length. Equivalently, it is one minus the unexplained squared length divided by the total. In the three-observation example, the total squared length is fourteen thirds. The explained squared length is nine halves, and the residual squared length is one sixth. Nine halves plus one sixth is fourteen thirds, exactly as the right triangle requires. The resulting R squared is twenty-seven over twenty-eight, about zero point nine six four. There is also an angle interpretation. Let theta be the angle between the centred response and its fitted projection. Then R squared is cosine squared theta. A small angle means the response lies close to the model subspace, so the projection keeps nearly all its length. A large angle means a larger perpendicular residual. With ordinary least squares and an intercept, this training-sample ratio lies between zero and one. Zero means the predictors improve nothing beyond the mean. One means every residual is zero. The intercept condition matters. Without the mean direction in the model, this familiar centred Pythagorean decomposition need not hold, and the usual interpretation of R squared needs care. R squared measures alignment with the fitted subspace, not causation, and not whether the model will predict new data well. It is a geometric summary of in-sample explained squared length. The projection picture has now given us residual minimization, the normal equations, and R squared. One final geometric question remains: what if the predictor directions themselves almost collapse together? ### [13:9.101 · When Predictor Directions Nearly Align](https://academa.ai/lectures/least-squares-is-a-projection?t=789.1013541666666) To see multicollinearity, first remove the intercept direction by centring, and scale two predictor columns to unit length. Then their relationship is summarized by the angle theta between their vectors. At sixty degrees, the blue and green predictors provide clearly different directions. Together they span the plane, and a fitted vector has coordinates beta one and beta two in this predictor basis. Draw the coefficient contributions tip to tail. The blue contribution first moves along x one. The green contribution then moves along x two, and their sum lands exactly at the red fitted vector. These coefficients are coordinates, not intrinsic lengths. Their values depend on how well the chosen predictor directions form a coordinate system. Now make the green predictor more like the blue one. At thirty degrees, the span is still the whole plane, and the fitted vector has not moved. But the two coefficient contributions must grow and cancel more strongly. Bring the angle down to ten degrees. The predictor arrows are now nearly parallel. To reach the same modest red vector, one long component runs left and the other almost cancels it while climbing to the target. This is multicollinearity. Two columns of X supply nearly the same direction in observation space, so the model has difficulty deciding how much of the fit to assign to each one. The fitted vector itself can still be perfectly clear. What becomes unstable is its description in this nearly collapsed coordinate system. Here is a small perturbation. Raise the fitted target's second coordinate from one point zero zero to one point zero two, only two hundredths. The red endpoint barely moves, but each coefficient changes by about zero point one one. The coordinate change is almost six times the size of the data perturbation, with opposite signs. The algebra reports the same geometry. For standardized columns, X transpose X has ones on the diagonal and cosine theta off the diagonal. Its determinant is one minus cosine squared theta, which is sine squared theta. As the predictor angle shrinks, this determinant approaches zero. At exactly zero degrees the two columns coincide. The span collapses from a plane to a line, X loses rank, and infinitely many coefficient pairs can describe the same fitted vector. Near zero, the inverse still exists in exact arithmetic, but it must magnify the thin direction enormously. That sensitivity is called ill conditioning. For two standardized predictors, this condition number is the square root of one plus cosine theta over one minus cosine theta. At ten degrees, the Gram determinant is about zero point zero three and the condition number is already about eleven point four. A condition number near one means the predictor directions are well separated. A large value means small changes in y, rounding, or sampling can create much larger changes in the estimated coefficients. This does not automatically make the fitted values unstable. Many different large, cancelling coefficient pairs can land at nearly the same projection point. Prediction within the observed predictor pattern may therefore look steady while individual coefficient interpretations do not. The first lesson is geometric: nearly parallel columns provide almost the same direction. The second is about coordinates: reaching one moderate fit may require large contributions with opposite signs. The third is statistical: a small change in the sample can move those large coefficients substantially, increasing their uncertainty. And the fourth separates two questions. The model may predict a combined effect reasonably while the separate effects of the correlated predictors remain poorly determined. Possible responses include collecting data where the predictors vary more independently, combining redundant predictors, choosing one of them, or using a regularized fit. Each response changes how the thin direction is handled, so it should follow the scientific purpose of the model. The whole lecture is now one geometric story. Data points create vertical residuals. Least squares chooses the shortest residual vector in observation space. Perpendicularity gives the normal equations. Pythagoras turns the same projection into R squared. And when predictor directions nearly align, the spanning coordinates become ill conditioned even if the projected fit barely changes. ## About Academa, Inc. Academa makes technical knowledge easier to understand through visual lectures and lets learners request new lecture videos on the topics they need. ## Complete audiovisual record Immutable source: [semantic.json](https://academa.ai/media/l/01M14V05J1Q2H560M850WHHDZA/1/semantic.json) Record version: 1. Render attempt: 1. ### How to read this timeline Each scene owns its object identifiers. A beat's board is the complete board when listed, empty when marked empty, and unchanged from the nearest earlier listed board in the same scene when marked unchanged. Action times are absolute positions in the published video. ### Scene 1: [Residuals and the Best Line](https://academa.ai/lectures/least-squares-is-a-projection?t=0) Span: 00:00–03:15.447 (0s–195.44677083333335s). #### Objects - candidate: a Line \[green\] labelled "hat(y)=3+m x" drawn in data\_axes (start=(-2.0, (3.0 - (2.0 \* slope))), end=(2.0, (3.0 + (2.0 \* slope)))) - data\_axes: an Axes (x\_range=(-2.5, 2.5), y\_range=(-1.0, 7.0), x\_ticks\_every=1.0) - heading: a Heading that says "Observed Data and a Candidate Line" - heading\_2: a Heading that says "Tilt the Line and Watch the Error" - left\_label: a Tex \[text\] that says "Data and residuals" - line\_family: a Math \[text\] that says "$hat(y)\_i = 3 + m x\_i$" - minimum\_point: a PlotPoint \[green\] labelled "upright("minimum")" drawn in sse\_axes (target='sse\_curve', x=1.2) - point\_1: a Point \[blue\] drawn in data\_axes (location=(-2.0, 1.0)) - point\_2: a Point \[blue\] drawn in data\_axes (location=(-1.0, 2.0)) - point\_3: a Point \[blue\] drawn in data\_axes (location=(0.0, 2.0)) - point\_4: a Point \[blue\] drawn in data\_axes (location=(1.0, 4.0)) - point\_5: a Point \[blue\] drawn in data\_axes (location=(2.0, 6.0)) - residual\_1: a Line \[yellow\] drawn in data\_axes (start=(-2.0, (3.0 - (2.0 \* slope))), end=(-2.0, 1.0)) - residual\_2: a Line \[yellow\] drawn in data\_axes (start=(-1.0, (3.0 - slope)), end=(-1.0, 2.0)) - residual\_3: a Line \[yellow\] drawn in data\_axes (start=(0.0, 3.0), end=(0.0, 2.0)) - residual\_4: a Line \[yellow\] drawn in data\_axes (start=(1.0, (3.0 + slope)), end=(1.0, 4.0)) - residual\_5: a Line \[yellow\] drawn in data\_axes (start=(2.0, (3.0 + (2.0 \* slope))), end=(2.0, 6.0)) - residual\_definition: a Math \[text\] that says "$e\_i = y\_i - hat(y)\_i$" - right\_label: a Tex \[text\] that says "Squared error versus slope" - slope: a VariableNumber (format\_spec='.1f') - squared\_error: a Math \[text\] that says "$upright("SSE")(m) = sum\_i e\_i^2$" - sse\_axes: an Axes (x\_range=(-0.2, 2.4), y\_range=(0.0, 20.0), x\_ticks\_every=0.5) - sse\_curve: a FunctionPlot \[red\] drawn in sse\_axes (function=\, x\_range=(-0.1, 2.3)) - sse\_tracker: a PlotPoint \[yellow\] labelled "0.0" drawn in sse\_axes (target='sse\_curve', x=\) #### Beats ##### [00:00](https://academa.ai/lectures/least-squares-is-a-projection?t=0) Narration: We begin with five observed pairs. The horizontal coordinate is the predictor x, and the vertical coordinate is the response y. A regression line will summarize how the response changes as x changes. Board: Empty. Actions: - [00:00](https://academa.ai/lectures/least-squares-is-a-projection?t=0): heading is shown on the screen, written out. - [00:1.207](https://academa.ai/lectures/least-squares-is-a-projection?t=1.207): data\_axes is shown on the screen, written out. - [00:1.649](https://academa.ai/lectures/least-squares-is-a-projection?t=1.649): point\_1 is shown on the screen, written out. - [00:1.769](https://academa.ai/lectures/least-squares-is-a-projection?t=1.7690000000000001): point\_2 is shown on the screen, written out. - [00:1.889](https://academa.ai/lectures/least-squares-is-a-projection?t=1.889): point\_3 is shown on the screen, written out. - [00:2.009](https://academa.ai/lectures/least-squares-is-a-projection?t=2.009): point\_4 is shown on the screen, written out. - [00:2.129](https://academa.ai/lectures/least-squares-is-a-projection?t=2.129): point\_5 is shown on the screen, written out. ##### [00:14.253](https://academa.ai/lectures/least-squares-is-a-projection?t=14.253499999999999) Narration: Here is one candidate line. For this example its height at the centre is fixed at three, and m controls its tilt. We have not yet chosen m. Board: data\_axes — an Axes (x\_range=(-2.5, 2.5), y\_range=(-1.0, 7.0), x\_ticks\_every=1.0); heading — a Heading that says "Observed Data and a Candidate Line"; point\_1 — a Point \[blue\] drawn in data\_axes (location=(-2.0, 1.0)); point\_2 — a Point \[blue\] drawn in data\_axes (location=(-1.0, 2.0)); point\_3 — a Point \[blue\] drawn in data\_axes (location=(0.0, 2.0)); point\_4 — a Point \[blue\] drawn in data\_axes (location=(1.0, 4.0)); point\_5 — a Point \[blue\] drawn in data\_axes (location=(2.0, 6.0)) Actions: - [00:15.438](https://academa.ai/lectures/least-squares-is-a-projection?t=15.437999999999999): candidate is shown on the screen, written out. - [00:21.091](https://academa.ai/lectures/least-squares-is-a-projection?t=21.091): data\_axes moves to a new place on the board. - [00:21.091](https://academa.ai/lectures/least-squares-is-a-projection?t=21.091): line\_family is shown on the screen, written out. ##### [00:25.407](https://academa.ai/lectures/least-squares-is-a-projection?t=25.4065) Narration: At each observed x value, the fitted line predicts a height. The vertical segment from that fitted height to the observed point is the residual. Board: line\_family — a Math \[text\] that says "$hat(y)\_i = 3 + m x\_i$"; data\_axes — an Axes (x\_range=(-2.5, 2.5), y\_range=(-1.0, 7.0), x\_ticks\_every=1.0); heading — a Heading that says "Observed Data and a Candidate Line"; point\_1 — a Point \[blue\] drawn in data\_axes (location=(-2.0, 1.0)); point\_2 — a Point \[blue\] drawn in data\_axes (location=(-1.0, 2.0)); point\_3 — a Point \[blue\] drawn in data\_axes (location=(0.0, 2.0)); point\_4 — a Point \[blue\] drawn in data\_axes (location=(1.0, 4.0)); point\_5 — a Point \[blue\] drawn in data\_axes (location=(2.0, 6.0)); candidate — a Line \[green\] labelled "hat(y)=3+m x" drawn in data\_axes (start=(-2.0, (3.0 - (2.0 \* slope))), end=(2.0, (3.0 + (2.0 \* slope)))) Actions: - [00:30.515](https://academa.ai/lectures/least-squares-is-a-projection?t=30.515): residual\_1 is shown on the screen, written out. - [00:30.635](https://academa.ai/lectures/least-squares-is-a-projection?t=30.635): residual\_2 is shown on the screen, written out. - [00:30.755](https://academa.ai/lectures/least-squares-is-a-projection?t=30.755): residual\_3 is shown on the screen, written out. - [00:30.875](https://academa.ai/lectures/least-squares-is-a-projection?t=30.875): residual\_4 is shown on the screen, written out. - [00:30.995](https://academa.ai/lectures/least-squares-is-a-projection?t=30.995): residual\_5 is shown on the screen, written out. - [00:33.812](https://academa.ai/lectures/least-squares-is-a-projection?t=33.812000000000005): residual\_definition is shown on the screen, written out. ##### [00:35.492](https://academa.ai/lectures/least-squares-is-a-projection?t=35.4915) Narration: The sign records direction. A point above the line has a positive residual, and a point below it has a negative residual. But a good fit should penalize a large miss in either direction. Board: line\_family — a Math \[text\] that says "$hat(y)\_i = 3 + m x\_i$"; residual\_definition — a Math \[text\] that says "$e\_i = y\_i - hat(y)\_i$"; data\_axes — an Axes (x\_range=(-2.5, 2.5), y\_range=(-1.0, 7.0), x\_ticks\_every=1.0); heading — a Heading that says "Observed Data and a Candidate Line"; point\_1 — a Point \[blue\] drawn in data\_axes (location=(-2.0, 1.0)); point\_2 — a Point \[blue\] drawn in data\_axes (location=(-1.0, 2.0)); point\_3 — a Point \[blue\] drawn in data\_axes (location=(0.0, 2.0)); point\_4 — a Point \[blue\] drawn in data\_axes (location=(1.0, 4.0)); point\_5 — a Point \[blue\] drawn in data\_axes (location=(2.0, 6.0)); candidate — a Line \[green\] labelled "hat(y)=3+m x" drawn in data\_axes (start=(-2.0, (3.0 - (2.0 \* slope))), end=(2.0, (3.0 + (2.0 \* slope)))); residual\_1 — a Line \[yellow\] drawn in data\_axes (start=(-2.0, (3.0 - (2.0 \* slope))), end=(-2.0, 1.0)); residual\_2 — a Line \[yellow\] drawn in data\_axes (start=(-1.0, (3.0 - slope)), end=(-1.0, 2.0)); residual\_3 — a Line \[yellow\] drawn in data\_axes (start=(0.0, 3.0), end=(0.0, 2.0)); residual\_4 — a Line \[yellow\] drawn in data\_axes (start=(1.0, (3.0 + slope)), end=(1.0, 4.0)); residual\_5 — a Line \[yellow\] drawn in data\_axes (start=(2.0, (3.0 + (2.0 \* slope))), end=(2.0, 6.0)) Actions: - [00:38.475](https://academa.ai/lectures/least-squares-is-a-projection?t=38.475): residual\_definition (the "y\_i" part) is emphasized. - [00:39.195](https://academa.ai/lectures/least-squares-is-a-projection?t=39.19500000000001): residual\_definition (the "hat(y)\_i" part) is emphasized. - [00:39.195](https://academa.ai/lectures/least-squares-is-a-projection?t=39.19500000000001): residual\_definition (the "y\_i" part) is no longer emphasized. - [00:48.727](https://academa.ai/lectures/least-squares-is-a-projection?t=48.727000000000004): residual\_definition (the "hat(y)\_i" part) is no longer emphasized. ##### [00:49.327](https://academa.ai/lectures/least-squares-is-a-projection?t=49.327) Narration: So square every residual and add the squares. This sum of squared errors is nonnegative, it treats equal misses above and below equally, and large misses count much more than small ones. Board: Unchanged from the preceding beat in this scene. Actions: - [00:50.093](https://academa.ai/lectures/least-squares-is-a-projection?t=50.092999999999996): squared\_error is shown on the screen, written out. - [00:52.403](https://academa.ai/lectures/least-squares-is-a-projection?t=52.403): squared\_error (the "e\_i^2" part) is emphasized. - [01:3.096](https://academa.ai/lectures/least-squares-is-a-projection?t=63.096): squared\_error (the "e\_i^2" part) is no longer emphasized. ##### [01:3.696](https://academa.ai/lectures/least-squares-is-a-projection?t=63.696) Narration: Notice what we are minimizing. These are vertical differences in the response coordinate. Ordinary regression of y on x does not use the shortest distance to the line; it holds each observed x fixed. Board: line\_family — a Math \[text\] that says "$hat(y)\_i = 3 + m x\_i$"; residual\_definition — a Math \[text\] that says "$e\_i = y\_i - hat(y)\_i$"; squared\_error — a Math \[text\] that says "$upright("SSE")(m) = sum\_i e\_i^2$"; data\_axes — an Axes (x\_range=(-2.5, 2.5), y\_range=(-1.0, 7.0), x\_ticks\_every=1.0); heading — a Heading that says "Observed Data and a Candidate Line"; point\_1 — a Point \[blue\] drawn in data\_axes (location=(-2.0, 1.0)); point\_2 — a Point \[blue\] drawn in data\_axes (location=(-1.0, 2.0)); point\_3 — a Point \[blue\] drawn in data\_axes (location=(0.0, 2.0)); point\_4 — a Point \[blue\] drawn in data\_axes (location=(1.0, 4.0)); point\_5 — a Point \[blue\] drawn in data\_axes (location=(2.0, 6.0)); candidate — a Line \[green\] labelled "hat(y)=3+m x" drawn in data\_axes (start=(-2.0, (3.0 - (2.0 \* slope))), end=(2.0, (3.0 + (2.0 \* slope)))); residual\_1 — a Line \[yellow\] drawn in data\_axes (start=(-2.0, (3.0 - (2.0 \* slope))), end=(-2.0, 1.0)); residual\_2 — a Line \[yellow\] drawn in data\_axes (start=(-1.0, (3.0 - slope)), end=(-1.0, 2.0)); residual\_3 — a Line \[yellow\] drawn in data\_axes (start=(0.0, 3.0), end=(0.0, 2.0)); residual\_4 — a Line \[yellow\] drawn in data\_axes (start=(1.0, (3.0 + slope)), end=(1.0, 4.0)); residual\_5 — a Line \[yellow\] drawn in data\_axes (start=(2.0, (3.0 + (2.0 \* slope))), end=(2.0, 6.0)) Actions: - [01:6.981](https://academa.ai/lectures/least-squares-is-a-projection?t=66.981): residual\_2 is indicated — a transient flash. - [01:17.094](https://academa.ai/lectures/least-squares-is-a-projection?t=77.094): residual\_4 is indicated — a transient flash. ##### [01:18.53](https://academa.ai/lectures/least-squares-is-a-projection?t=78.53) Narration: With a horizontal line, m is zero. The five residuals have squared sum sixteen. The line is clearly missing the upward pattern in the data. Board: Unchanged from the preceding beat in this scene. Actions: - [01:19.192](https://academa.ai/lectures/least-squares-is-a-projection?t=79.19199999999998): candidate is indicated — a transient flash. - [01:23.546](https://academa.ai/lectures/least-squares-is-a-projection?t=83.54599999999998): squared\_error is indicated — a transient flash. ##### [01:29.834](https://academa.ai/lectures/least-squares-is-a-projection?t=89.834) Narration: To choose the tilt without calculus, let us watch every candidate and record its squared error. The same live value of m will drive both pictures. Board: Unchanged from the preceding beat in this scene. Actions: - [01:39.621](https://academa.ai/lectures/least-squares-is-a-projection?t=99.6215): data\_axes moves to a new place on the board. - [01:39.621](https://academa.ai/lectures/least-squares-is-a-projection?t=99.6215): heading is hidden from the screen — left the board. - [01:39.621](https://academa.ai/lectures/least-squares-is-a-projection?t=99.6215): line\_family is hidden from the screen — left the board. - [01:39.621](https://academa.ai/lectures/least-squares-is-a-projection?t=99.6215): residual\_definition is hidden from the screen — left the board. - [01:39.621](https://academa.ai/lectures/least-squares-is-a-projection?t=99.6215): squared\_error is hidden from the screen — left the board. ##### [01:40.822](https://academa.ai/lectures/least-squares-is-a-projection?t=100.8215) Narration: On the right, the red curve records the sum of squared errors for every slope in view. The yellow dot begins at slope zero and error sixteen. Board: data\_axes — an Axes (x\_range=(-2.5, 2.5), y\_range=(-1.0, 7.0), x\_ticks\_every=1.0); point\_1 — a Point \[blue\] drawn in data\_axes (location=(-2.0, 1.0)); point\_2 — a Point \[blue\] drawn in data\_axes (location=(-1.0, 2.0)); point\_3 — a Point \[blue\] drawn in data\_axes (location=(0.0, 2.0)); point\_4 — a Point \[blue\] drawn in data\_axes (location=(1.0, 4.0)); point\_5 — a Point \[blue\] drawn in data\_axes (location=(2.0, 6.0)); candidate — a Line \[green\] labelled "hat(y)=3+m x" drawn in data\_axes (start=(-2.0, (3.0 - (2.0 \* slope))), end=(2.0, (3.0 + (2.0 \* slope)))); residual\_1 — a Line \[yellow\] drawn in data\_axes (start=(-2.0, (3.0 - (2.0 \* slope))), end=(-2.0, 1.0)); residual\_2 — a Line \[yellow\] drawn in data\_axes (start=(-1.0, (3.0 - slope)), end=(-1.0, 2.0)); residual\_3 — a Line \[yellow\] drawn in data\_axes (start=(0.0, 3.0), end=(0.0, 2.0)); residual\_4 — a Line \[yellow\] drawn in data\_axes (start=(1.0, (3.0 + slope)), end=(1.0, 4.0)); residual\_5 — a Line \[yellow\] drawn in data\_axes (start=(2.0, (3.0 + (2.0 \* slope))), end=(2.0, 6.0)) Actions: - [01:40.822](https://academa.ai/lectures/least-squares-is-a-projection?t=100.8215): sse\_axes is shown on the screen, written out. - [01:42.365](https://academa.ai/lectures/least-squares-is-a-projection?t=102.36500000000001): sse\_curve is shown on the screen, written out. - [01:47.055](https://academa.ai/lectures/least-squares-is-a-projection?t=107.055): sse\_tracker is shown on the screen, written out. ##### [01:51.556](https://academa.ai/lectures/least-squares-is-a-projection?t=111.5565) Narration: Tilt the fitted line upward to slope zero point six. The residual pattern tightens, and the squared error falls from sixteen to five point two. Board: data\_axes — an Axes (x\_range=(-2.5, 2.5), y\_range=(-1.0, 7.0), x\_ticks\_every=1.0); point\_1 — a Point \[blue\] drawn in data\_axes (location=(-2.0, 1.0)); point\_2 — a Point \[blue\] drawn in data\_axes (location=(-1.0, 2.0)); point\_3 — a Point \[blue\] drawn in data\_axes (location=(0.0, 2.0)); point\_4 — a Point \[blue\] drawn in data\_axes (location=(1.0, 4.0)); point\_5 — a Point \[blue\] drawn in data\_axes (location=(2.0, 6.0)); candidate — a Line \[green\] labelled "hat(y)=3+m x" drawn in data\_axes (start=(-2.0, (3.0 - (2.0 \* slope))), end=(2.0, (3.0 + (2.0 \* slope)))); residual\_1 — a Line \[yellow\] drawn in data\_axes (start=(-2.0, (3.0 - (2.0 \* slope))), end=(-2.0, 1.0)); residual\_2 — a Line \[yellow\] drawn in data\_axes (start=(-1.0, (3.0 - slope)), end=(-1.0, 2.0)); residual\_3 — a Line \[yellow\] drawn in data\_axes (start=(0.0, 3.0), end=(0.0, 2.0)); residual\_4 — a Line \[yellow\] drawn in data\_axes (start=(1.0, (3.0 + slope)), end=(1.0, 4.0)); residual\_5 — a Line \[yellow\] drawn in data\_axes (start=(2.0, (3.0 + (2.0 \* slope))), end=(2.0, 6.0)); sse\_axes — an Axes (x\_range=(-0.2, 2.4), y\_range=(0.0, 20.0), x\_ticks\_every=0.5); sse\_curve — a FunctionPlot \[red\] drawn in sse\_axes (function=\, x\_range=(-0.1, 2.3)); sse\_tracker — a PlotPoint \[yellow\] labelled "0.0" drawn in sse\_axes (target='sse\_curve', x=\) Actions: - [01:51.905](https://academa.ai/lectures/least-squares-is-a-projection?t=111.905): candidate is redrawn as the numbers it depends on change. - [01:51.905](https://academa.ai/lectures/least-squares-is-a-projection?t=111.905): residual\_1 is redrawn as the numbers it depends on change. - [01:51.905](https://academa.ai/lectures/least-squares-is-a-projection?t=111.905): residual\_2 is redrawn as the numbers it depends on change. - [01:51.905](https://academa.ai/lectures/least-squares-is-a-projection?t=111.905): residual\_4 is redrawn as the numbers it depends on change. - [01:51.905](https://academa.ai/lectures/least-squares-is-a-projection?t=111.905): residual\_5 is redrawn as the numbers it depends on change. - [01:51.905](https://academa.ai/lectures/least-squares-is-a-projection?t=111.905): sse\_tracker is redrawn as the numbers it depends on change. - [01:51.905](https://academa.ai/lectures/least-squares-is-a-projection?t=111.905): slope ticks to 0.6. ##### [02:1.573](https://academa.ai/lectures/least-squares-is-a-projection?t=121.5725) Narration: Continue to slope one point two. Several vertical misses are now small, and the dot reaches the bottom of the error curve. The squared error is one point six. Board: Unchanged from the preceding beat in this scene. Actions: - [02:1.828](https://academa.ai/lectures/least-squares-is-a-projection?t=121.828): candidate is redrawn as the numbers it depends on change. - [02:1.828](https://academa.ai/lectures/least-squares-is-a-projection?t=121.828): residual\_1 is redrawn as the numbers it depends on change. - [02:1.828](https://academa.ai/lectures/least-squares-is-a-projection?t=121.828): residual\_2 is redrawn as the numbers it depends on change. - [02:1.828](https://academa.ai/lectures/least-squares-is-a-projection?t=121.828): residual\_4 is redrawn as the numbers it depends on change. - [02:1.828](https://academa.ai/lectures/least-squares-is-a-projection?t=121.828): residual\_5 is redrawn as the numbers it depends on change. - [02:1.828](https://academa.ai/lectures/least-squares-is-a-projection?t=121.828): sse\_tracker is redrawn as the numbers it depends on change. - [02:1.828](https://academa.ai/lectures/least-squares-is-a-projection?t=121.828): slope ticks to 1.2. - [02:7.749](https://academa.ai/lectures/least-squares-is-a-projection?t=127.74900000000001): minimum\_point is shown on the screen, written out. ##### [02:12.993](https://academa.ai/lectures/least-squares-is-a-projection?t=132.993) Narration: Go past that slope to one point eight. Some residuals change sign, but their squares grow again. The error rises back to five point two. Board: data\_axes — an Axes (x\_range=(-2.5, 2.5), y\_range=(-1.0, 7.0), x\_ticks\_every=1.0); point\_1 — a Point \[blue\] drawn in data\_axes (location=(-2.0, 1.0)); point\_2 — a Point \[blue\] drawn in data\_axes (location=(-1.0, 2.0)); point\_3 — a Point \[blue\] drawn in data\_axes (location=(0.0, 2.0)); point\_4 — a Point \[blue\] drawn in data\_axes (location=(1.0, 4.0)); point\_5 — a Point \[blue\] drawn in data\_axes (location=(2.0, 6.0)); candidate — a Line \[green\] labelled "hat(y)=3+m x" drawn in data\_axes (start=(-2.0, (3.0 - (2.0 \* slope))), end=(2.0, (3.0 + (2.0 \* slope)))); residual\_1 — a Line \[yellow\] drawn in data\_axes (start=(-2.0, (3.0 - (2.0 \* slope))), end=(-2.0, 1.0)); residual\_2 — a Line \[yellow\] drawn in data\_axes (start=(-1.0, (3.0 - slope)), end=(-1.0, 2.0)); residual\_3 — a Line \[yellow\] drawn in data\_axes (start=(0.0, 3.0), end=(0.0, 2.0)); residual\_4 — a Line \[yellow\] drawn in data\_axes (start=(1.0, (3.0 + slope)), end=(1.0, 4.0)); residual\_5 — a Line \[yellow\] drawn in data\_axes (start=(2.0, (3.0 + (2.0 \* slope))), end=(2.0, 6.0)); sse\_axes — an Axes (x\_range=(-0.2, 2.4), y\_range=(0.0, 20.0), x\_ticks\_every=0.5); sse\_curve — a FunctionPlot \[red\] drawn in sse\_axes (function=\, x\_range=(-0.1, 2.3)); sse\_tracker — a PlotPoint \[yellow\] labelled "0.0" drawn in sse\_axes (target='sse\_curve', x=\); minimum\_point — a PlotPoint \[green\] labelled "upright("minimum")" drawn in sse\_axes (target='sse\_curve', x=1.2) Actions: - [02:13.585](https://academa.ai/lectures/least-squares-is-a-projection?t=133.58499999999998): candidate is redrawn as the numbers it depends on change. - [02:13.585](https://academa.ai/lectures/least-squares-is-a-projection?t=133.58499999999998): residual\_1 is redrawn as the numbers it depends on change. - [02:13.585](https://academa.ai/lectures/least-squares-is-a-projection?t=133.58499999999998): residual\_2 is redrawn as the numbers it depends on change. - [02:13.585](https://academa.ai/lectures/least-squares-is-a-projection?t=133.58499999999998): residual\_4 is redrawn as the numbers it depends on change. - [02:13.585](https://academa.ai/lectures/least-squares-is-a-projection?t=133.58499999999998): residual\_5 is redrawn as the numbers it depends on change. - [02:13.585](https://academa.ai/lectures/least-squares-is-a-projection?t=133.58499999999998): sse\_tracker is redrawn as the numbers it depends on change. - [02:13.585](https://academa.ai/lectures/least-squares-is-a-projection?t=133.58499999999998): slope ticks to 1.8. ##### [02:23.264](https://academa.ai/lectures/least-squares-is-a-projection?t=143.2645) Narration: Return to the bottom. The line with slope one point two has the smallest sum of squared vertical residuals among this whole family. That is the least-squares fit. Board: Unchanged from the preceding beat in this scene. Actions: - [02:23.613](https://academa.ai/lectures/least-squares-is-a-projection?t=143.613): candidate is redrawn as the numbers it depends on change. - [02:23.613](https://academa.ai/lectures/least-squares-is-a-projection?t=143.613): residual\_1 is redrawn as the numbers it depends on change. - [02:23.613](https://academa.ai/lectures/least-squares-is-a-projection?t=143.613): residual\_2 is redrawn as the numbers it depends on change. - [02:23.613](https://academa.ai/lectures/least-squares-is-a-projection?t=143.613): residual\_4 is redrawn as the numbers it depends on change. - [02:23.613](https://academa.ai/lectures/least-squares-is-a-projection?t=143.613): residual\_5 is redrawn as the numbers it depends on change. - [02:23.613](https://academa.ai/lectures/least-squares-is-a-projection?t=143.613): sse\_tracker is redrawn as the numbers it depends on change. - [02:23.613](https://academa.ai/lectures/least-squares-is-a-projection?t=143.613): slope ticks to 1.2. - [02:27.607](https://academa.ai/lectures/least-squares-is-a-projection?t=147.607): minimum\_point is indicated — a transient flash. ##### [02:35.323](https://academa.ai/lectures/least-squares-is-a-projection?t=155.323) Narration: The curve on the right is not an appeal to differentiation. It is a geometric record of what the yellow segments on the left are doing. Tilting away from the best line makes their combined squared length grow. Board: Unchanged from the preceding beat in this scene. Actions: - [02:41.709](https://academa.ai/lectures/least-squares-is-a-projection?t=161.709): residual\_1 is indicated — a transient flash. - [02:41.709](https://academa.ai/lectures/least-squares-is-a-projection?t=161.709): residual\_5 is indicated — a transient flash. - [02:47.839](https://academa.ai/lectures/least-squares-is-a-projection?t=167.839): sse\_curve is indicated — a transient flash. ##### [02:49.263](https://academa.ai/lectures/least-squares-is-a-projection?t=169.263) Narration: For this centred predictor, the intercept stayed at the mean response, three, while the slope changed. In a general fit, both intercept and slope are chosen together to minimize the same squared residual total. Board: Unchanged from the preceding beat in this scene. Actions: - [02:51.005](https://academa.ai/lectures/least-squares-is-a-projection?t=171.005): candidate is indicated — a transient flash. ##### [03:3.947](https://academa.ai/lectures/least-squares-is-a-projection?t=183.94650000000001) Narration: This data picture explains what least squares measures. To understand why the optimum has a clean algebraic form, we now change the geometry completely. Board: Unchanged from the preceding beat in this scene. Actions: - [03:14.405](https://academa.ai/lectures/least-squares-is-a-projection?t=194.4051041666667): data\_axes is hidden from the screen — left the board. - [03:14.405](https://academa.ai/lectures/least-squares-is-a-projection?t=194.4051041666667): point\_1 is hidden from the screen — data\_axes left the board. - [03:14.405](https://academa.ai/lectures/least-squares-is-a-projection?t=194.4051041666667): point\_2 is hidden from the screen — data\_axes left the board. - [03:14.405](https://academa.ai/lectures/least-squares-is-a-projection?t=194.4051041666667): point\_3 is hidden from the screen — data\_axes left the board. - [03:14.405](https://academa.ai/lectures/least-squares-is-a-projection?t=194.4051041666667): point\_4 is hidden from the screen — data\_axes left the board. - [03:14.405](https://academa.ai/lectures/least-squares-is-a-projection?t=194.4051041666667): point\_5 is hidden from the screen — data\_axes left the board. - [03:14.405](https://academa.ai/lectures/least-squares-is-a-projection?t=194.4051041666667): candidate is hidden from the screen — data\_axes left the board. - [03:14.405](https://academa.ai/lectures/least-squares-is-a-projection?t=194.4051041666667): residual\_1 is hidden from the screen — data\_axes left the board. - [03:14.405](https://academa.ai/lectures/least-squares-is-a-projection?t=194.4051041666667): residual\_2 is hidden from the screen — data\_axes left the board. - [03:14.405](https://academa.ai/lectures/least-squares-is-a-projection?t=194.4051041666667): residual\_3 is hidden from the screen — data\_axes left the board. - [03:14.405](https://academa.ai/lectures/least-squares-is-a-projection?t=194.4051041666667): residual\_4 is hidden from the screen — data\_axes left the board. - [03:14.405](https://academa.ai/lectures/least-squares-is-a-projection?t=194.4051041666667): residual\_5 is hidden from the screen — data\_axes left the board. - [03:14.405](https://academa.ai/lectures/least-squares-is-a-projection?t=194.4051041666667): sse\_axes is hidden from the screen — left the board. - [03:14.405](https://academa.ai/lectures/least-squares-is-a-projection?t=194.4051041666667): sse\_curve is hidden from the screen — sse\_axes left the board. - [03:14.405](https://academa.ai/lectures/least-squares-is-a-projection?t=194.4051041666667): sse\_tracker is hidden from the screen — sse\_axes left the board. - [03:14.405](https://academa.ai/lectures/least-squares-is-a-projection?t=194.4051041666667): minimum\_point is hidden from the screen — sse\_axes left the board. ### Scene 2: [The Fit as a Shadow](https://academa.ai/lectures/least-squares-is-a-projection?t=195.44677083333335) Span: 03:15.447–07:1.952 (195.44677083333335s–421.9515s). #### Objects - column\_plane: a Plane \[blue\] labelled "upright("col")(X)" drawn in space (normal=(1.0, -2.0, 1.0), edge\_direction=(1.0, 1.0, 1.0), opacity=0.16) - data\_table: a Table \[text\] that says "Observation $y$ $1$ $x$ 1 1 1 -1 2 2 1 0 3 4 1 1" (rows=(('Observation', '$y$', '$1$', '$x$'), ('1', '1', '1', '-1'), (…, header=True) - decomposition: a Math \[text\] that says "$y = hat(y) + e$" - fitted\_vector: a Vector \[green\] labelled "hat(y)" drawn in space (start=(0.0, 0.0, 0.0), end=(0.8333333333333334, 2.3333333333333335, 3.8333333333333335)) - heading: a Heading that says "Three Observations Become Three Coordinates" - heading\_2: a Heading that says "The Predictor Columns Span the Possible Fits" - heading\_3: a Heading that says "Least Squares Is Orthogonal Projection" - intercept\_vector: a Vector \[blue\] labelled "c\_0" drawn in space (start=(0.0, 0.0, 0.0), end=(1.0, 1.0, 1.0)) - linear\_combinations: a Math \[text\] that says "$hat(y) = beta\_0 c\_0 + beta\_1 c\_1$" - matrix\_form: a Math \[text\] that says "$hat(y) = X hat(beta)$" - predictor\_vector: a Vector \[green\] labelled "c\_1" drawn in space (start=(0.0, 0.0, 0.0), end=(-1.0, 0.0, 1.0)) - projection\_note: a Panel that says "The least-squares fitted vector is the point in the column space closest to the observed response vector." - projection\_rule: a Math \[text\] that says "$hat(y) = upright("proj")\_(upright("col")(X)) y$" - residual\_vector: a Vector \[red\] labelled "e" drawn in space (start=(0.8333333333333334, 2.3333333333333335, 3.8333333333333335), end=(1.0, 2.0, 4.0)) - response\_vector: a Vector \[yellow\] labelled "y" drawn in space (start=(0.0, 0.0, 0.0), end=(1.0, 2.0, 4.0)) - right\_angle: an Angle \[yellow\] drawn in space (vertex=(0.8333333333333334, 2.3333333333333335, 3.8333333333333335), sides=((0.0, 0.0, 0.0), (1.0, 2.0, 4.0)), right\_angle=True) - space: an Axes3D (x\_range=(-1.5, 2.0), y\_range=(-0.5, 3.0), z\_range=(-0.5, 4.5)) #### Beats ##### [03:15.447](https://academa.ai/lectures/least-squares-is-a-projection?t=195.44677083333335) Narration: Now replace the data plot by observation space. To draw it, we use a miniature data set with three observations. Three observations give three coordinates, so the whole construction fits in ordinary three-dimensional space. Board: Empty. Actions: - [03:15.447](https://academa.ai/lectures/least-squares-is-a-projection?t=195.44677083333335): heading is shown on the screen, written out. - [03:21.624](https://academa.ai/lectures/least-squares-is-a-projection?t=201.62377083333334): data\_table is shown on the screen, written out. - [03:28.067](https://academa.ai/lectures/least-squares-is-a-projection?t=208.06677083333335): space is shown on the screen, written out. ##### [03:30.211](https://academa.ai/lectures/least-squares-is-a-projection?t=210.21077083333336) Narration: The first row records response one, the second response two, and the third response four. Read down the response column, not across a plotted axis. Board: space — an Axes3D (x\_range=(-1.5, 2.0), y\_range=(-0.5, 3.0), z\_range=(-0.5, 4.5)); heading — a Heading that says "Three Observations Become Three Coordinates" Actions: - [03:30.756](https://academa.ai/lectures/least-squares-is-a-projection?t=210.75577083333334): data\_table is shown on the screen, written out. - [03:33.16](https://academa.ai/lectures/least-squares-is-a-projection?t=213.15977083333334): data\_table is shown on the screen, written out. - [03:34.808](https://academa.ai/lectures/least-squares-is-a-projection?t=214.80777083333334): data\_table is shown on the screen, written out. - [03:37.351](https://academa.ai/lectures/least-squares-is-a-projection?t=217.35077083333334): data\_table (the "column=2" part) is emphasized. ##### [03:41.144](https://academa.ai/lectures/least-squares-is-a-projection?t=221.14377083333335) Narration: Stack those three responses into one vector y: one, two, four. Each coordinate belongs to one observation, so this single arrow represents the entire response data set. Board: Unchanged from the preceding beat in this scene. Actions: - [03:43.605](https://academa.ai/lectures/least-squares-is-a-projection?t=223.60477083333333): response\_vector is shown on the screen, written out. - [03:53.345](https://academa.ai/lectures/least-squares-is-a-projection?t=233.34527083333336): data\_table (the "column=2" part) is no longer emphasized. ##### [03:53.945](https://academa.ai/lectures/least-squares-is-a-projection?t=233.94527083333335) Narration: The design matrix also has one coordinate per observation in each column. Its intercept column is one, one, one, because the intercept contributes equally to every fitted response. Board: space — an Axes3D (x\_range=(-1.5, 2.0), y\_range=(-0.5, 3.0), z\_range=(-0.5, 4.5)); heading — a Heading that says "Three Observations Become Three Coordinates"; response\_vector — a Vector \[yellow\] labelled "y" drawn in space (start=(0.0, 0.0, 0.0), end=(1.0, 2.0, 4.0)) Actions: - [03:59.332](https://academa.ai/lectures/least-squares-is-a-projection?t=239.33177083333334): intercept\_vector is shown on the screen, written out. ##### [04:6.817](https://academa.ai/lectures/least-squares-is-a-projection?t=246.81727083333334) Narration: The predictor column is minus one, zero, one. It records the three predictor values after centring. This green arrow is that whole column. Board: space — an Axes3D (x\_range=(-1.5, 2.0), y\_range=(-0.5, 3.0), z\_range=(-0.5, 4.5)); heading — a Heading that says "Three Observations Become Three Coordinates"; response\_vector — a Vector \[yellow\] labelled "y" drawn in space (start=(0.0, 0.0, 0.0), end=(1.0, 2.0, 4.0)); intercept\_vector — a Vector \[blue\] labelled "c\_0" drawn in space (start=(0.0, 0.0, 0.0), end=(1.0, 1.0, 1.0)) Actions: - [04:14.201](https://academa.ai/lectures/least-squares-is-a-projection?t=254.20077083333334): predictor\_vector is shown on the screen, written out. - [04:16.198](https://academa.ai/lectures/least-squares-is-a-projection?t=256.19827083333337): data\_table is hidden from the screen — left the board. - [04:16.198](https://academa.ai/lectures/least-squares-is-a-projection?t=256.19827083333337): heading is hidden from the screen — left the board. ##### [04:16.798](https://academa.ai/lectures/least-squares-is-a-projection?t=256.79827083333333) Narration: A coefficient beta zero scales the blue column. A coefficient beta one scales the green column. Adding the two scaled arrows produces every fitted response vector this model is capable of making. Board: space — an Axes3D (x\_range=(-1.5, 2.0), y\_range=(-0.5, 3.0), z\_range=(-0.5, 4.5)); response\_vector — a Vector \[yellow\] labelled "y" drawn in space (start=(0.0, 0.0, 0.0), end=(1.0, 2.0, 4.0)); intercept\_vector — a Vector \[blue\] labelled "c\_0" drawn in space (start=(0.0, 0.0, 0.0), end=(1.0, 1.0, 1.0)); predictor\_vector — a Vector \[green\] labelled "c\_1" drawn in space (start=(0.0, 0.0, 0.0), end=(-1.0, 0.0, 1.0)) Actions: - [04:16.798](https://academa.ai/lectures/least-squares-is-a-projection?t=256.79827083333333): linear\_combinations is shown on the screen, written out. - [04:19.039](https://academa.ai/lectures/least-squares-is-a-projection?t=259.03877083333333): intercept\_vector is indicated — a transient flash. - [04:22.603](https://academa.ai/lectures/least-squares-is-a-projection?t=262.60277083333335): predictor\_vector is indicated — a transient flash. ##### [04:30.621](https://academa.ai/lectures/least-squares-is-a-projection?t=270.62127083333337) Narration: Those linear combinations fill a plane through the origin. This is the column space of X. Changing the regression coefficients moves the fitted vector somewhere inside this plane, but never outside it. Board: space — an Axes3D (x\_range=(-1.5, 2.0), y\_range=(-0.5, 3.0), z\_range=(-0.5, 4.5)); response\_vector — a Vector \[yellow\] labelled "y" drawn in space (start=(0.0, 0.0, 0.0), end=(1.0, 2.0, 4.0)); intercept\_vector — a Vector \[blue\] labelled "c\_0" drawn in space (start=(0.0, 0.0, 0.0), end=(1.0, 1.0, 1.0)); predictor\_vector — a Vector \[green\] labelled "c\_1" drawn in space (start=(0.0, 0.0, 0.0), end=(-1.0, 0.0, 1.0)); linear\_combinations — a Math \[text\] that says "$hat(y) = beta\_0 c\_0 + beta\_1 c\_1$" Actions: - [04:31.295](https://academa.ai/lectures/least-squares-is-a-projection?t=271.29477083333336): linear\_combinations is indicated — a transient flash. - [04:32.758](https://academa.ai/lectures/least-squares-is-a-projection?t=272.7577708333333): column\_plane is shown on the screen, written out. ##### [04:44.213](https://academa.ai/lectures/least-squares-is-a-projection?t=284.21327083333335) Narration: The picture is three-dimensional only because this example has three observations. With n observations, y lives in n-dimensional observation space. With p predictors and an intercept, the columns span a subspace of dimension at most p plus one. Board: space — an Axes3D (x\_range=(-1.5, 2.0), y\_range=(-0.5, 3.0), z\_range=(-0.5, 4.5)); response\_vector — a Vector \[yellow\] labelled "y" drawn in space (start=(0.0, 0.0, 0.0), end=(1.0, 2.0, 4.0)); intercept\_vector — a Vector \[blue\] labelled "c\_0" drawn in space (start=(0.0, 0.0, 0.0), end=(1.0, 1.0, 1.0)); predictor\_vector — a Vector \[green\] labelled "c\_1" drawn in space (start=(0.0, 0.0, 0.0), end=(-1.0, 0.0, 1.0)); linear\_combinations — a Math \[text\] that says "$hat(y) = beta\_0 c\_0 + beta\_1 c\_1$"; column\_plane — a Plane \[blue\] labelled "upright("col")(X)" drawn in space (normal=(1.0, -2.0, 1.0), edge\_direction=(1.0, 1.0, 1.0), opacity=0.16) Actions: - [04:45.433](https://academa.ai/lectures/least-squares-is-a-projection?t=285.43277083333334): space turns in its own slot. ##### [05:1.543](https://academa.ai/lectures/least-squares-is-a-projection?t=301.54327083333334) Narration: The yellow response vector usually misses the predictor plane. An exact fit would put y inside the column space, but these three observations do not lie on one exact regression line. Board: Unchanged from the preceding beat in this scene. Actions: - [05:2.066](https://academa.ai/lectures/least-squares-is-a-projection?t=302.0657708333333): response\_vector is indicated — a transient flash. - [05:4.829](https://academa.ai/lectures/least-squares-is-a-projection?t=304.82877083333335): column\_plane is indicated — a transient flash. ##### [05:13.672](https://academa.ai/lectures/least-squares-is-a-projection?t=313.67227083333336) Narration: Among all points in the plane, choose the one closest to y. The green arrow reaches that point. It is the fitted-value vector, hat y. Board: Unchanged from the preceding beat in this scene. Actions: - [05:18.537](https://academa.ai/lectures/least-squares-is-a-projection?t=318.5367708333333): fitted\_vector is shown on the screen, written out. - [05:21.312](https://academa.ai/lectures/least-squares-is-a-projection?t=321.31177083333336): matrix\_form is shown on the screen, written out. - [05:23.657](https://academa.ai/lectures/least-squares-is-a-projection?t=323.6567708333333): linear\_combinations is hidden from the screen. ##### [05:24.257](https://academa.ai/lectures/least-squares-is-a-projection?t=324.25677083333335) Narration: Its three coordinates are the three fitted responses. Returning to the original data plot would place those fitted values on the regression line at the three observed predictor values. Board: space — an Axes3D (x\_range=(-1.5, 2.0), y\_range=(-0.5, 3.0), z\_range=(-0.5, 4.5)); response\_vector — a Vector \[yellow\] labelled "y" drawn in space (start=(0.0, 0.0, 0.0), end=(1.0, 2.0, 4.0)); intercept\_vector — a Vector \[blue\] labelled "c\_0" drawn in space (start=(0.0, 0.0, 0.0), end=(1.0, 1.0, 1.0)); predictor\_vector — a Vector \[green\] labelled "c\_1" drawn in space (start=(0.0, 0.0, 0.0), end=(-1.0, 0.0, 1.0)); matrix\_form — a Math \[text\] that says "$hat(y) = X hat(beta)$"; column\_plane — a Plane \[blue\] labelled "upright("col")(X)" drawn in space (normal=(1.0, -2.0, 1.0), edge\_direction=(1.0, 1.0, 1.0), opacity=0.16); fitted\_vector — a Vector \[green\] labelled "hat(y)" drawn in space (start=(0.0, 0.0, 0.0), end=(0.8333333333333334, 2.3333333333333335, 3.8333333333333335)) Actions: - [05:26.242](https://academa.ai/lectures/least-squares-is-a-projection?t=326.24177083333336): fitted\_vector is indicated — a transient flash. ##### [05:35.573](https://academa.ai/lectures/least-squares-is-a-projection?t=335.5727708333334) Narration: What remains is the red vector from hat y to y. Coordinate by coordinate, it contains observed response minus fitted response, so it is exactly the residual vector e. Board: Unchanged from the preceding beat in this scene. Actions: - [05:36.967](https://academa.ai/lectures/least-squares-is-a-projection?t=336.9667708333334): residual\_vector is shown on the screen, written out. - [05:44.908](https://academa.ai/lectures/least-squares-is-a-projection?t=344.90777083333336): decomposition is shown on the screen, written out. ##### [05:47.771](https://academa.ai/lectures/least-squares-is-a-projection?t=347.77127083333335) Narration: The closest route from a point to a plane is perpendicular to the plane. Therefore the residual stands at a right angle to every direction the predictor columns span. Board: space — an Axes3D (x\_range=(-1.5, 2.0), y\_range=(-0.5, 3.0), z\_range=(-0.5, 4.5)); response\_vector — a Vector \[yellow\] labelled "y" drawn in space (start=(0.0, 0.0, 0.0), end=(1.0, 2.0, 4.0)); intercept\_vector — a Vector \[blue\] labelled "c\_0" drawn in space (start=(0.0, 0.0, 0.0), end=(1.0, 1.0, 1.0)); predictor\_vector — a Vector \[green\] labelled "c\_1" drawn in space (start=(0.0, 0.0, 0.0), end=(-1.0, 0.0, 1.0)); matrix\_form — a Math \[text\] that says "$hat(y) = X hat(beta)$"; decomposition — a Math \[text\] that says "$y = hat(y) + e$"; column\_plane — a Plane \[blue\] labelled "upright("col")(X)" drawn in space (normal=(1.0, -2.0, 1.0), edge\_direction=(1.0, 1.0, 1.0), opacity=0.16); fitted\_vector — a Vector \[green\] labelled "hat(y)" drawn in space (start=(0.0, 0.0, 0.0), end=(0.8333333333333334, 2.3333333333333335, 3.8333333333333335)); residual\_vector — a Vector \[red\] labelled "e" drawn in space (start=(0.8333333333333334, 2.3333333333333335, 3.8333333333333335), end=(1.0, 2.0, 4.0)) Actions: - [05:49.78](https://academa.ai/lectures/least-squares-is-a-projection?t=349.7797708333334): column\_plane is indicated — a transient flash. - [05:54.599](https://academa.ai/lectures/least-squares-is-a-projection?t=354.59877083333333): right\_angle is shown on the screen, written out. - [05:57.942](https://academa.ai/lectures/least-squares-is-a-projection?t=357.94177083333335): decomposition moves to a new place on the board. - [05:57.942](https://academa.ai/lectures/least-squares-is-a-projection?t=357.94177083333335): matrix\_form is hidden from the screen. ##### [05:59.142](https://academa.ai/lectures/least-squares-is-a-projection?t=359.14177083333334) Narration: This is least squares in one sentence. The fitted vector is the orthogonal projection of y onto the column space of X. Board: space — an Axes3D (x\_range=(-1.5, 2.0), y\_range=(-0.5, 3.0), z\_range=(-0.5, 4.5)); response\_vector — a Vector \[yellow\] labelled "y" drawn in space (start=(0.0, 0.0, 0.0), end=(1.0, 2.0, 4.0)); intercept\_vector — a Vector \[blue\] labelled "c\_0" drawn in space (start=(0.0, 0.0, 0.0), end=(1.0, 1.0, 1.0)); predictor\_vector — a Vector \[green\] labelled "c\_1" drawn in space (start=(0.0, 0.0, 0.0), end=(-1.0, 0.0, 1.0)); decomposition — a Math \[text\] that says "$y = hat(y) + e$"; column\_plane — a Plane \[blue\] labelled "upright("col")(X)" drawn in space (normal=(1.0, -2.0, 1.0), edge\_direction=(1.0, 1.0, 1.0), opacity=0.16); fitted\_vector — a Vector \[green\] labelled "hat(y)" drawn in space (start=(0.0, 0.0, 0.0), end=(0.8333333333333334, 2.3333333333333335, 3.8333333333333335)); residual\_vector — a Vector \[red\] labelled "e" drawn in space (start=(0.8333333333333334, 2.3333333333333335, 3.8333333333333335), end=(1.0, 2.0, 4.0)); right\_angle — an Angle \[yellow\] drawn in space (vertex=(0.8333333333333334, 2.3333333333333335, 3.8333333333333335), sides=((0.0, 0.0, 0.0), (1.0, 2.0, 4.0)), right\_angle=True) Actions: - [05:59.142](https://academa.ai/lectures/least-squares-is-a-projection?t=359.14177083333334): projection\_note is shown on the screen, written out. - [06:2.521](https://academa.ai/lectures/least-squares-is-a-projection?t=362.52077083333336): fitted\_vector is indicated — a transient flash. - [06:4.169](https://academa.ai/lectures/least-squares-is-a-projection?t=364.1687708333334): projection\_rule is shown on the screen, written out. ##### [06:8.368](https://academa.ai/lectures/least-squares-is-a-projection?t=368.3682708333333) Narration: The word shadow is useful. Imagine light travelling perpendicular to the predictor plane. The response arrow casts its endpoint onto the plane, and the shadow point is hat y. Board: space — an Axes3D (x\_range=(-1.5, 2.0), y\_range=(-0.5, 3.0), z\_range=(-0.5, 4.5)); response\_vector — a Vector \[yellow\] labelled "y" drawn in space (start=(0.0, 0.0, 0.0), end=(1.0, 2.0, 4.0)); intercept\_vector — a Vector \[blue\] labelled "c\_0" drawn in space (start=(0.0, 0.0, 0.0), end=(1.0, 1.0, 1.0)); predictor\_vector — a Vector \[green\] labelled "c\_1" drawn in space (start=(0.0, 0.0, 0.0), end=(-1.0, 0.0, 1.0)); decomposition — a Math \[text\] that says "$y = hat(y) + e$"; column\_plane — a Plane \[blue\] labelled "upright("col")(X)" drawn in space (normal=(1.0, -2.0, 1.0), edge\_direction=(1.0, 1.0, 1.0), opacity=0.16); fitted\_vector — a Vector \[green\] labelled "hat(y)" drawn in space (start=(0.0, 0.0, 0.0), end=(0.8333333333333334, 2.3333333333333335, 3.8333333333333335)); residual\_vector — a Vector \[red\] labelled "e" drawn in space (start=(0.8333333333333334, 2.3333333333333335, 3.8333333333333335), end=(1.0, 2.0, 4.0)); right\_angle — an Angle \[yellow\] drawn in space (vertex=(0.8333333333333334, 2.3333333333333335, 3.8333333333333335), sides=((0.0, 0.0, 0.0), (1.0, 2.0, 4.0)), right\_angle=True); projection\_note — a Panel that says "The least-squares fitted vector is the point in the column space closest to the observed response vector."; projection\_rule — a Math \[text\] that says "$hat(y) = upright("proj")\_(upright("col")(X)) y$" Actions: - [06:9.193](https://academa.ai/lectures/least-squares-is-a-projection?t=369.1927708333333): fitted\_vector is indicated — a transient flash. - [06:14.824](https://academa.ai/lectures/least-squares-is-a-projection?t=374.82377083333336): response\_vector is indicated — a transient flash. ##### [06:20.532](https://academa.ai/lectures/least-squares-is-a-projection?t=380.5317708333333) Narration: The red leftover is perpendicular to the shadow plane. Pythagoras then says no other point in the plane can be closer to y. Moving away from hat y adds an in-plane displacement at right angles to the residual. Board: Unchanged from the preceding beat in this scene. Actions: - [06:21.031](https://academa.ai/lectures/least-squares-is-a-projection?t=381.03077083333335): residual\_vector is indicated — a transient flash. - [06:32.095](https://academa.ai/lectures/least-squares-is-a-projection?t=392.0947708333333): right\_angle is indicated — a transient flash. ##### [06:34.402](https://academa.ai/lectures/least-squares-is-a-projection?t=394.40177083333333) Narration: That geometric fact is the minimization. We did not differentiate a sum of squares. We identified the unique shortest vector from y to the model subspace. Board: Unchanged from the preceding beat in this scene. Actions: - [06:42.494](https://academa.ai/lectures/least-squares-is-a-projection?t=402.4937708333334): projection\_note (the "closest" part) is emphasized. - [06:45.64](https://academa.ai/lectures/least-squares-is-a-projection?t=405.6402708333333): projection\_note (the "closest" part) is no longer emphasized. ##### [06:46.24](https://academa.ai/lectures/least-squares-is-a-projection?t=406.24027083333334) Narration: The same picture survives with many observations and many predictors. Only the dimension changes. The response is one point, the possible fits form a column space, and least squares drops a perpendicular to it. Board: Unchanged from the preceding beat in this scene. Actions: - [07:0.91](https://academa.ai/lectures/least-squares-is-a-projection?t=420.9098333333334): decomposition is hidden from the screen — left the board. - [07:0.91](https://academa.ai/lectures/least-squares-is-a-projection?t=420.9098333333334): projection\_note is hidden from the screen — left the board. - [07:0.91](https://academa.ai/lectures/least-squares-is-a-projection?t=420.9098333333334): projection\_rule is hidden from the screen — left the board. - [07:0.91](https://academa.ai/lectures/least-squares-is-a-projection?t=420.9098333333334): space is hidden from the screen — left the board. - [07:0.91](https://academa.ai/lectures/least-squares-is-a-projection?t=420.9098333333334): response\_vector is hidden from the screen — space left the board. - [07:0.91](https://academa.ai/lectures/least-squares-is-a-projection?t=420.9098333333334): intercept\_vector is hidden from the screen — space left the board. - [07:0.91](https://academa.ai/lectures/least-squares-is-a-projection?t=420.9098333333334): predictor\_vector is hidden from the screen — space left the board. - [07:0.91](https://academa.ai/lectures/least-squares-is-a-projection?t=420.9098333333334): column\_plane is hidden from the screen — space left the board. - [07:0.91](https://academa.ai/lectures/least-squares-is-a-projection?t=420.9098333333334): fitted\_vector is hidden from the screen — space left the board. - [07:0.91](https://academa.ai/lectures/least-squares-is-a-projection?t=420.9098333333334): residual\_vector is hidden from the screen — space left the board. - [07:0.91](https://academa.ai/lectures/least-squares-is-a-projection?t=420.9098333333334): right\_angle is hidden from the screen — space left the board. ### Scene 3: [Perpendicularity and the Normal Equations](https://academa.ai/lectures/least-squares-is-a-projection?t=421.9515) Span: 07:1.952–10:7.603 (421.9515s–607.6031666666667s). #### Objects - column\_0: a Vector \[blue\] labelled "c\_0" drawn in space (start=(0.0, 0.0, 0.0), end=(1.0, 1.0, 1.0)) - column\_1: a Vector \[green\] labelled "c\_1" drawn in space (start=(0.0, 0.0, 0.0), end=(-1.0, 0.0, 1.0)) - condition: a Text \[text\] that says "This inverse formula requires linearly independent columns. The normal equations themselves remain the central geometric statement." - fit: a Vector \[green\] labelled "hat(y)" drawn in space (start=(0.0, 0.0, 0.0), end=(0.8333333333333334, 2.3333333333333335, 3.8333333333333335)) - heading: a Heading that says "Turn the Right Angle into Equations" - heading\_2: a Heading that says "The Normal Equations" - normal\_work: a Derivation \[text\] that says "$X^T (y - X hat(beta)) &= 0 \\ X^T y - X^T X hat(beta) &= 0 \\ X^T X hat(beta) &= X^T y$" - orthogonality\_fact: a Math \[text\] that says "$X^T e = 0$" - orthogonality\_work: a Derivation \[text\] that says "$hat(y) &= X hat(beta) \\ e &= y - X hat(beta) \\ c\_j^T e &= 0, quad j=0,1,dots,p \\ X^T e &= 0$" - plane: a Plane \[blue\] labelled "upright("col")(X)" drawn in space (normal=(1.0, -2.0, 1.0), edge\_direction=(1.0, 1.0, 1.0), opacity=0.16) - residual: a Vector \[red\] labelled "e" drawn in space (start=(0.8333333333333334, 2.3333333333333335, 3.8333333333333335), end=(1.0, 2.0, 4.0)) - right\_angle: an Angle \[yellow\] drawn in space (vertex=(0.8333333333333334, 2.3333333333333335, 3.8333333333333335), sides=((0.0, 0.0, 0.0), (1.0, 2.0, 4.0)), right\_angle=True) - solution: a Math \[text\] that says "$hat(beta) = (X^T X)^(-1) X^T y$" - space: an Axes3D (x\_range=(-1.5, 2.0), y\_range=(-0.5, 3.0), z\_range=(-0.5, 4.5)) #### Beats ##### [07:1.952](https://academa.ai/lectures/least-squares-is-a-projection?t=421.9515) Narration: Now use the right angle to calculate the coefficients. The fitted vector is X times beta hat, because it is a linear combination of the columns of X. Board: Empty. Actions: - [07:1.952](https://academa.ai/lectures/least-squares-is-a-projection?t=421.9515): heading is shown on the screen, written out. - [07:1.952](https://academa.ai/lectures/least-squares-is-a-projection?t=421.9515): space is shown on the screen, written out. - [07:5.783](https://academa.ai/lectures/least-squares-is-a-projection?t=425.7825): fit is shown on the screen, written out. - [07:9.046](https://academa.ai/lectures/least-squares-is-a-projection?t=429.0455): space moves to a new place on the board. - [07:9.046](https://academa.ai/lectures/least-squares-is-a-projection?t=429.0455): orthogonality\_work is shown on the screen, written out. - [07:10.288](https://academa.ai/lectures/least-squares-is-a-projection?t=430.2875): plane is shown on the screen, written out. - [07:10.288](https://academa.ai/lectures/least-squares-is-a-projection?t=430.2875): column\_0 is shown on the screen, written out. - [07:10.288](https://academa.ai/lectures/least-squares-is-a-projection?t=430.2875): column\_1 is shown on the screen, written out. ##### [07:12.281](https://academa.ai/lectures/least-squares-is-a-projection?t=432.2805) Narration: The residual is observed y minus fitted y. Substituting the matrix form gives e equals y minus X beta hat. Board: space — an Axes3D (x\_range=(-1.5, 2.0), y\_range=(-0.5, 3.0), z\_range=(-0.5, 4.5)); heading — a Heading that says "Turn the Right Angle into Equations"; plane — a Plane \[blue\] labelled "upright("col")(X)" drawn in space (normal=(1.0, -2.0, 1.0), edge\_direction=(1.0, 1.0, 1.0), opacity=0.16); column\_0 — a Vector \[blue\] labelled "c\_0" drawn in space (start=(0.0, 0.0, 0.0), end=(1.0, 1.0, 1.0)); column\_1 — a Vector \[green\] labelled "c\_1" drawn in space (start=(0.0, 0.0, 0.0), end=(-1.0, 0.0, 1.0)); fit — a Vector \[green\] labelled "hat(y)" drawn in space (start=(0.0, 0.0, 0.0), end=(0.8333333333333334, 2.3333333333333335, 3.8333333333333335)) Actions: - [07:12.815](https://academa.ai/lectures/least-squares-is-a-projection?t=432.8145): residual is shown on the screen, written out. - [07:12.815](https://academa.ai/lectures/least-squares-is-a-projection?t=432.8145): right\_angle is shown on the screen, written out. - [07:14.567](https://academa.ai/lectures/least-squares-is-a-projection?t=434.5675): orthogonality\_work is shown on the screen, written out. ##### [07:22.226](https://academa.ai/lectures/least-squares-is-a-projection?t=442.226) Narration: Perpendicular vectors have dot product zero. Since e is perpendicular to the entire column space, it is perpendicular to the intercept column, the first predictor column, and every other predictor column. Board: space — an Axes3D (x\_range=(-1.5, 2.0), y\_range=(-0.5, 3.0), z\_range=(-0.5, 4.5)); heading — a Heading that says "Turn the Right Angle into Equations"; plane — a Plane \[blue\] labelled "upright("col")(X)" drawn in space (normal=(1.0, -2.0, 1.0), edge\_direction=(1.0, 1.0, 1.0), opacity=0.16); column\_0 — a Vector \[blue\] labelled "c\_0" drawn in space (start=(0.0, 0.0, 0.0), end=(1.0, 1.0, 1.0)); column\_1 — a Vector \[green\] labelled "c\_1" drawn in space (start=(0.0, 0.0, 0.0), end=(-1.0, 0.0, 1.0)); fit — a Vector \[green\] labelled "hat(y)" drawn in space (start=(0.0, 0.0, 0.0), end=(0.8333333333333334, 2.3333333333333335, 3.8333333333333335)); residual — a Vector \[red\] labelled "e" drawn in space (start=(0.8333333333333334, 2.3333333333333335, 3.8333333333333335), end=(1.0, 2.0, 4.0)); right\_angle — an Angle \[yellow\] drawn in space (vertex=(0.8333333333333334, 2.3333333333333335, 3.8333333333333335), sides=((0.0, 0.0, 0.0), (1.0, 2.0, 4.0)), right\_angle=True) Actions: - [07:23.957](https://academa.ai/lectures/least-squares-is-a-projection?t=443.9565): orthogonality\_work is shown on the screen, written out. - [07:30.608](https://academa.ai/lectures/least-squares-is-a-projection?t=450.6085): column\_0 is indicated — a transient flash. - [07:31.898](https://academa.ai/lectures/least-squares-is-a-projection?t=451.89750000000004): column\_1 is indicated — a transient flash. ##### [07:36.132](https://academa.ai/lectures/least-squares-is-a-projection?t=456.1315) Narration: For one column c sub j, perpendicularity reads c sub j transpose e equals zero. Write one such equation for every column. Board: Unchanged from the preceding beat in this scene. Actions: - [07:39.719](https://academa.ai/lectures/least-squares-is-a-projection?t=459.7185): orthogonality\_work (the "c\_j^T e" part) is emphasized. - [07:47.161](https://academa.ai/lectures/least-squares-is-a-projection?t=467.1605): orthogonality\_work (the "c\_j^T e" part) is no longer emphasized. ##### [07:47.76](https://academa.ai/lectures/least-squares-is-a-projection?t=467.7605) Narration: Stack all those dot products. The transpose X transpose places the predictor columns into the rows of one multiplication, so the whole collection becomes X transpose e equals zero. Board: Unchanged from the preceding beat in this scene. Actions: - [07:48.063](https://academa.ai/lectures/least-squares-is-a-projection?t=468.0625): orthogonality\_work is shown on the screen, written out. - [07:56.481](https://academa.ai/lectures/least-squares-is-a-projection?t=476.4805): orthogonality\_work (the "X^T e" part) is emphasized. - [08:0.149](https://academa.ai/lectures/least-squares-is-a-projection?t=480.149): orthogonality\_work (the "X^T e" part) is no longer emphasized. ##### [08:0.749](https://academa.ai/lectures/least-squares-is-a-projection?t=480.749) Narration: This compact zero is the algebraic form of the visible right angle. Every coordinate of X transpose e is one column's dot product with the residual. Board: Unchanged from the preceding beat in this scene. Actions: - [08:1.969](https://academa.ai/lectures/least-squares-is-a-projection?t=481.9685): orthogonality\_work is indicated — a transient flash. - [08:4.197](https://academa.ai/lectures/least-squares-is-a-projection?t=484.1975): right\_angle is indicated — a transient flash. - [08:10.839](https://academa.ai/lectures/least-squares-is-a-projection?t=490.8385): heading is hidden from the screen — left the board. - [08:10.839](https://academa.ai/lectures/least-squares-is-a-projection?t=490.8385): orthogonality\_work is hidden from the screen — left the board. ##### [08:11.438](https://academa.ai/lectures/least-squares-is-a-projection?t=491.4385) Narration: Carry that perpendicularity forward, and replace e by y minus X beta hat. It becomes X transpose times y minus X beta hat equals zero. Board: space — an Axes3D (x\_range=(-1.5, 2.0), y\_range=(-0.5, 3.0), z\_range=(-0.5, 4.5)); plane — a Plane \[blue\] labelled "upright("col")(X)" drawn in space (normal=(1.0, -2.0, 1.0), edge\_direction=(1.0, 1.0, 1.0), opacity=0.16); column\_0 — a Vector \[blue\] labelled "c\_0" drawn in space (start=(0.0, 0.0, 0.0), end=(1.0, 1.0, 1.0)); column\_1 — a Vector \[green\] labelled "c\_1" drawn in space (start=(0.0, 0.0, 0.0), end=(-1.0, 0.0, 1.0)); fit — a Vector \[green\] labelled "hat(y)" drawn in space (start=(0.0, 0.0, 0.0), end=(0.8333333333333334, 2.3333333333333335, 3.8333333333333335)); residual — a Vector \[red\] labelled "e" drawn in space (start=(0.8333333333333334, 2.3333333333333335, 3.8333333333333335), end=(1.0, 2.0, 4.0)); right\_angle — an Angle \[yellow\] drawn in space (vertex=(0.8333333333333334, 2.3333333333333335, 3.8333333333333335), sides=((0.0, 0.0, 0.0), (1.0, 2.0, 4.0)), right\_angle=True) Actions: - [08:11.438](https://academa.ai/lectures/least-squares-is-a-projection?t=491.4385): orthogonality\_fact is shown on the screen, written out. - [08:14.063](https://academa.ai/lectures/least-squares-is-a-projection?t=494.0625): normal\_work is shown on the screen, written out. ##### [08:22.377](https://academa.ai/lectures/least-squares-is-a-projection?t=502.377) Narration: Distribute X transpose. The first term is X transpose y. The second is X transpose X beta hat. Board: space — an Axes3D (x\_range=(-1.5, 2.0), y\_range=(-0.5, 3.0), z\_range=(-0.5, 4.5)); plane — a Plane \[blue\] labelled "upright("col")(X)" drawn in space (normal=(1.0, -2.0, 1.0), edge\_direction=(1.0, 1.0, 1.0), opacity=0.16); column\_0 — a Vector \[blue\] labelled "c\_0" drawn in space (start=(0.0, 0.0, 0.0), end=(1.0, 1.0, 1.0)); column\_1 — a Vector \[green\] labelled "c\_1" drawn in space (start=(0.0, 0.0, 0.0), end=(-1.0, 0.0, 1.0)); fit — a Vector \[green\] labelled "hat(y)" drawn in space (start=(0.0, 0.0, 0.0), end=(0.8333333333333334, 2.3333333333333335, 3.8333333333333335)); residual — a Vector \[red\] labelled "e" drawn in space (start=(0.8333333333333334, 2.3333333333333335, 3.8333333333333335), end=(1.0, 2.0, 4.0)); right\_angle — an Angle \[yellow\] drawn in space (vertex=(0.8333333333333334, 2.3333333333333335, 3.8333333333333335), sides=((0.0, 0.0, 0.0), (1.0, 2.0, 4.0)), right\_angle=True); orthogonality\_fact — a Math \[text\] that says "$X^T e = 0$" Actions: - [08:22.72](https://academa.ai/lectures/least-squares-is-a-projection?t=502.71950000000004): normal\_work is shown on the screen, written out. - [08:25.182](https://academa.ai/lectures/least-squares-is-a-projection?t=505.1815): normal\_work (the "X^T y" part) is emphasized. - [08:28.2](https://academa.ai/lectures/least-squares-is-a-projection?t=508.1995): normal\_work (the "X^T X hat(beta)" part) is emphasized. - [08:28.2](https://academa.ai/lectures/least-squares-is-a-projection?t=508.1995): normal\_work (the "X^T y" part) is no longer emphasized. - [08:30.968](https://academa.ai/lectures/least-squares-is-a-projection?t=510.968): normal\_work (the "X^T X hat(beta)" part) is no longer emphasized. ##### [08:31.568](https://academa.ai/lectures/least-squares-is-a-projection?t=511.568) Narration: Move the second term to the other side. We obtain X transpose X beta hat equals X transpose y. These are the normal equations. Board: Unchanged from the preceding beat in this scene. Actions: - [08:31.911](https://academa.ai/lectures/least-squares-is-a-projection?t=511.9105): normal\_work is shown on the screen, written out. - [08:40.212](https://academa.ai/lectures/least-squares-is-a-projection?t=520.2115): A box is drawn around normal\_work. ##### [08:42.287](https://academa.ai/lectures/least-squares-is-a-projection?t=522.2865) Narration: They are called normal because the residual is normal, meaning perpendicular, to the predictor subspace. The name is geometric before it is algebraic. Board: Unchanged from the preceding beat in this scene. Actions: - [08:44.236](https://academa.ai/lectures/least-squares-is-a-projection?t=524.2365): residual is indicated — a transient flash. - [08:46.095](https://academa.ai/lectures/least-squares-is-a-projection?t=526.0945): right\_angle is indicated — a transient flash. ##### [08:52.825](https://academa.ai/lectures/least-squares-is-a-projection?t=532.8245000000001) Narration: The intercept equation has a familiar consequence. Its column is the all-ones vector, so its dot product with e is the sum of the residuals. With an intercept, that sum must be zero. Board: Unchanged from the preceding beat in this scene. Actions: - [08:57.202](https://academa.ai/lectures/least-squares-is-a-projection?t=537.2015): column\_0 is indicated — a transient flash. - [09:4.656](https://academa.ai/lectures/least-squares-is-a-projection?t=544.6555000000001): orthogonality\_fact is indicated — a transient flash. ##### [09:6.126](https://academa.ai/lectures/least-squares-is-a-projection?t=546.126) Narration: Each predictor equation says something similar: the residual has zero dot product with that predictor column. No linear pattern available to the model remains inside the residual. Board: Unchanged from the preceding beat in this scene. Actions: - [09:6.753](https://academa.ai/lectures/least-squares-is-a-projection?t=546.7525): column\_1 is indicated — a transient flash. - [09:9.412](https://academa.ai/lectures/least-squares-is-a-projection?t=549.4115): residual is indicated — a transient flash. ##### [09:18.255](https://academa.ai/lectures/least-squares-is-a-projection?t=558.2545) Narration: If the columns are linearly independent, X transpose X is invertible. Multiplying by its inverse gives the familiar coefficient formula. Board: Unchanged from the preceding beat in this scene. Actions: - [09:22.341](https://academa.ai/lectures/least-squares-is-a-projection?t=562.3405): solution is shown on the screen, written out. ##### [09:28.119](https://academa.ai/lectures/least-squares-is-a-projection?t=568.119) Narration: But the inverse formula is not the fundamental fact. The fundamental fact is X transpose e equals zero. Numerical methods usually solve the projection problem without explicitly forming an inverse. Board: space — an Axes3D (x\_range=(-1.5, 2.0), y\_range=(-0.5, 3.0), z\_range=(-0.5, 4.5)); plane — a Plane \[blue\] labelled "upright("col")(X)" drawn in space (normal=(1.0, -2.0, 1.0), edge\_direction=(1.0, 1.0, 1.0), opacity=0.16); column\_0 — a Vector \[blue\] labelled "c\_0" drawn in space (start=(0.0, 0.0, 0.0), end=(1.0, 1.0, 1.0)); column\_1 — a Vector \[green\] labelled "c\_1" drawn in space (start=(0.0, 0.0, 0.0), end=(-1.0, 0.0, 1.0)); fit — a Vector \[green\] labelled "hat(y)" drawn in space (start=(0.0, 0.0, 0.0), end=(0.8333333333333334, 2.3333333333333335, 3.8333333333333335)); residual — a Vector \[red\] labelled "e" drawn in space (start=(0.8333333333333334, 2.3333333333333335, 3.8333333333333335), end=(1.0, 2.0, 4.0)); right\_angle — an Angle \[yellow\] drawn in space (vertex=(0.8333333333333334, 2.3333333333333335, 3.8333333333333335), sides=((0.0, 0.0, 0.0), (1.0, 2.0, 4.0)), right\_angle=True); orthogonality\_fact — a Math \[text\] that says "$X^T e = 0$"; solution — a Math \[text\] that says "$hat(beta) = (X^T X)^(-1) X^T y$" Actions: - [09:29.861](https://academa.ai/lectures/least-squares-is-a-projection?t=569.8605): condition is shown on the screen, written out. - [09:35.131](https://academa.ai/lectures/least-squares-is-a-projection?t=575.1315): orthogonality\_fact is indicated — a transient flash. ##### [09:42.255](https://academa.ai/lectures/least-squares-is-a-projection?t=582.2555) Narration: If the columns are dependent, the fitted projection can still be defined even though the coefficient vector need not be unique. That distinction will matter when we discuss multicollinearity. Board: space — an Axes3D (x\_range=(-1.5, 2.0), y\_range=(-0.5, 3.0), z\_range=(-0.5, 4.5)); plane — a Plane \[blue\] labelled "upright("col")(X)" drawn in space (normal=(1.0, -2.0, 1.0), edge\_direction=(1.0, 1.0, 1.0), opacity=0.16); column\_0 — a Vector \[blue\] labelled "c\_0" drawn in space (start=(0.0, 0.0, 0.0), end=(1.0, 1.0, 1.0)); column\_1 — a Vector \[green\] labelled "c\_1" drawn in space (start=(0.0, 0.0, 0.0), end=(-1.0, 0.0, 1.0)); fit — a Vector \[green\] labelled "hat(y)" drawn in space (start=(0.0, 0.0, 0.0), end=(0.8333333333333334, 2.3333333333333335, 3.8333333333333335)); residual — a Vector \[red\] labelled "e" drawn in space (start=(0.8333333333333334, 2.3333333333333335, 3.8333333333333335), end=(1.0, 2.0, 4.0)); right\_angle — an Angle \[yellow\] drawn in space (vertex=(0.8333333333333334, 2.3333333333333335, 3.8333333333333335), sides=((0.0, 0.0, 0.0), (1.0, 2.0, 4.0)), right\_angle=True); orthogonality\_fact — a Math \[text\] that says "$X^T e = 0$"; solution — a Math \[text\] that says "$hat(beta) = (X^T X)^(-1) X^T y$"; condition — a Text \[text\] that says "This inverse formula requires linearly independent columns. The normal equations themselves remain the central geometric statement." Actions: - [09:44.659](https://academa.ai/lectures/least-squares-is-a-projection?t=584.6595): plane is indicated — a transient flash. - [09:46.841](https://academa.ai/lectures/least-squares-is-a-projection?t=586.8415): solution is indicated — a transient flash. ##### [09:54.117](https://academa.ai/lectures/least-squares-is-a-projection?t=594.117) Narration: So the normal equations come straight from one right angle: fitted values lie in the column space, residuals leave it perpendicularly, and every predictor column has zero residual dot product. Board: Unchanged from the preceding beat in this scene. Actions: - [09:56.637](https://academa.ai/lectures/least-squares-is-a-projection?t=596.6365000000001): right\_angle is indicated — a transient flash. - [09:59.191](https://academa.ai/lectures/least-squares-is-a-projection?t=599.1905): plane is indicated — a transient flash. - [10:6.562](https://academa.ai/lectures/least-squares-is-a-projection?t=606.5615): condition is hidden from the screen — left the board. - [10:6.562](https://academa.ai/lectures/least-squares-is-a-projection?t=606.5615): normal\_work is hidden from the screen — left the board. - [10:6.562](https://academa.ai/lectures/least-squares-is-a-projection?t=606.5615): orthogonality\_fact is hidden from the screen — left the board. - [10:6.562](https://academa.ai/lectures/least-squares-is-a-projection?t=606.5615): solution is hidden from the screen — left the board. - [10:6.562](https://academa.ai/lectures/least-squares-is-a-projection?t=606.5615): space is hidden from the screen — left the board. - [10:6.562](https://academa.ai/lectures/least-squares-is-a-projection?t=606.5615): plane is hidden from the screen — space left the board. - [10:6.562](https://academa.ai/lectures/least-squares-is-a-projection?t=606.5615): column\_0 is hidden from the screen — space left the board. - [10:6.562](https://academa.ai/lectures/least-squares-is-a-projection?t=606.5615): column\_1 is hidden from the screen — space left the board. - [10:6.562](https://academa.ai/lectures/least-squares-is-a-projection?t=606.5615): fit is hidden from the screen — space left the board. - [10:6.562](https://academa.ai/lectures/least-squares-is-a-projection?t=606.5615): residual is hidden from the screen — space left the board. - [10:6.562](https://academa.ai/lectures/least-squares-is-a-projection?t=606.5615): right\_angle is hidden from the screen — space left the board. ### Scene 4: [R Squared as a Length Ratio](https://academa.ai/lectures/least-squares-is-a-projection?t=607.6031666666667) Span: 10:7.603–13:9.101 (607.6031666666667s–789.1013541666666s). #### Objects - angle\_form: a Math \[text\] that says "$R^2 = cos^2 theta$" - decomposition: a Derivation \[text\] that says "$y\_c &= y - overline(y) bold(1) \\ hat(y)\_c &= hat(y) - overline(y) bold(1) \\ y\_c &= hat(y)\_c + e \\ y\_c dot y\_c &= hat(y)\_c dot hat(y)\_c + e dot e$" - explained\_arrow: a Vector \[green\] labelled "hat(y)\_c" drawn in triangle (end=(2.1213203436, 0.0)) - heading: a Heading that says "Centre the Response First" - heading\_2: a Heading that says "The Three-Observation Example" - numeric\_0: a Math \[text\] that says "$y\_c dot y\_c = frac(14, 3)$" - numeric\_1: a Math \[text\] that says "$hat(y)\_c dot hat(y)\_c = frac(9, 2)$" - numeric\_2: a Math \[text\] that says "$e dot e = frac(1, 6)$" - numeric\_3: a Math \[text\] that says "$R^2 = frac(9/2, 14/3) = frac(27, 28) approx 0.964$" - r\_squared: a Math \[text\] that says "$R^2 = frac(hat(y)\_c dot hat(y)\_c, y\_c dot y\_c) = 1 - frac(e dot e, y\_c dot y\_c)$" - residual\_arrow: a Vector \[red\] labelled "e" drawn in triangle (start=(2.1213203436, 0.0), end=(2.1213203436, 0.4082482905)) - right\_angle: an Angle \[yellow\] drawn in triangle (vertex=(2.1213203436, 0.0), sides=((0.0, 0.0), (2.1213203436, 0.4082482905)), right\_angle=True) - theta\_angle: an Angle \[blue\] labelled "theta" drawn in triangle (sides=((2.1213203436, 0.0), (2.1213203436, 0.4082482905)), radius=0.48) - total\_arrow: a Vector \[yellow\] labelled "y\_c" drawn in triangle (end=(2.1213203436, 0.4082482905)) - triangle: a Figure (x\_range=(-0.25, 2.75), y\_range=(-0.4, 1.4), aspect=(3.0, 1.8)) #### Beats ##### [10:7.603](https://academa.ai/lectures/least-squares-is-a-projection?t=607.6031666666667) Narration: R squared uses the same projection picture, but first we must identify what counts as total variation. With an intercept in the model, the baseline prediction is the sample mean of y. Board: Empty. Actions: - [10:7.603](https://academa.ai/lectures/least-squares-is-a-projection?t=607.6031666666667): heading is shown on the screen, written out. - [10:17.599](https://academa.ai/lectures/least-squares-is-a-projection?t=617.5991666666666): decomposition is shown on the screen, written out. ##### [10:19.744](https://academa.ai/lectures/least-squares-is-a-projection?t=619.7436666666666) Narration: Subtract that mean from every observed response. The centred vector y sub c measures how far the data sit from the mean-only fit. Board: heading — a Heading that says "Centre the Response First" Actions: - [10:21.229](https://academa.ai/lectures/least-squares-is-a-projection?t=621.2291666666666): decomposition (the "overline(y) bold(1)" part) is emphasized. - [10:24.12](https://academa.ai/lectures/least-squares-is-a-projection?t=624.1201666666667): total\_arrow is shown on the screen, written out. - [10:29.449](https://academa.ai/lectures/least-squares-is-a-projection?t=629.4491666666667): decomposition (the "overline(y) bold(1)" part) is no longer emphasized. ##### [10:30.049](https://academa.ai/lectures/least-squares-is-a-projection?t=630.0491666666667) Narration: Centre the fitted vector in the same way. Because the regression model includes the intercept direction, the centred fitted vector remains inside the predictor subspace. Board: heading — a Heading that says "Centre the Response First"; total\_arrow — a Vector \[yellow\] labelled "y\_c" drawn in triangle (end=(2.1213203436, 0.4082482905)) Actions: - [10:30.351](https://academa.ai/lectures/least-squares-is-a-projection?t=630.3511666666667): decomposition is shown on the screen, written out. - [10:30.815](https://academa.ai/lectures/least-squares-is-a-projection?t=630.8151666666666): explained\_arrow is shown on the screen, written out. ##### [10:41.086](https://academa.ai/lectures/least-squares-is-a-projection?t=641.0861666666666) Narration: The centred response splits into two pieces. The green piece is the variation explained by the predictors. The red piece is the residual variation left unexplained. Board: heading — a Heading that says "Centre the Response First"; total\_arrow — a Vector \[yellow\] labelled "y\_c" drawn in triangle (end=(2.1213203436, 0.4082482905)); explained\_arrow — a Vector \[green\] labelled "hat(y)\_c" drawn in triangle (end=(2.1213203436, 0.0)) Actions: - [10:42.619](https://academa.ai/lectures/least-squares-is-a-projection?t=642.6191666666666): decomposition is shown on the screen, written out. - [10:48.598](https://academa.ai/lectures/least-squares-is-a-projection?t=648.5981666666667): residual\_arrow is shown on the screen, written out. ##### [10:52.565](https://academa.ai/lectures/least-squares-is-a-projection?t=652.5646666666667) Narration: Projection makes those pieces perpendicular. The three vectors therefore form a right triangle: explained component, residual component, and total centred response. Board: heading — a Heading that says "Centre the Response First"; total\_arrow — a Vector \[yellow\] labelled "y\_c" drawn in triangle (end=(2.1213203436, 0.4082482905)); explained\_arrow — a Vector \[green\] labelled "hat(y)\_c" drawn in triangle (end=(2.1213203436, 0.0)); residual\_arrow — a Vector \[red\] labelled "e" drawn in triangle (start=(2.1213203436, 0.0), end=(2.1213203436, 0.4082482905)) Actions: - [10:57.94](https://academa.ai/lectures/least-squares-is-a-projection?t=657.9401666666666): right\_angle is shown on the screen, written out. ##### [11:4.763](https://academa.ai/lectures/least-squares-is-a-projection?t=664.7626666666666) Narration: Pythagoras applies to their squared lengths. Total sum of squares equals regression sum of squares plus residual sum of squares. Board: heading — a Heading that says "Centre the Response First"; total\_arrow — a Vector \[yellow\] labelled "y\_c" drawn in triangle (end=(2.1213203436, 0.4082482905)); explained\_arrow — a Vector \[green\] labelled "hat(y)\_c" drawn in triangle (end=(2.1213203436, 0.0)); residual\_arrow — a Vector \[red\] labelled "e" drawn in triangle (start=(2.1213203436, 0.0), end=(2.1213203436, 0.4082482905)); right\_angle — an Angle \[yellow\] drawn in triangle (vertex=(2.1213203436, 0.0), sides=((0.0, 0.0), (2.1213203436, 0.4082482905)), right\_angle=True) Actions: - [11:5.111](https://academa.ai/lectures/least-squares-is-a-projection?t=665.1111666666667): decomposition is shown on the screen, written out. - [11:8.222](https://academa.ai/lectures/least-squares-is-a-projection?t=668.2221666666667): decomposition (the "y\_c dot y\_c" part) is emphasized. - [11:9.871](https://academa.ai/lectures/least-squares-is-a-projection?t=669.8711666666667): decomposition (the "hat(y)\_c dot hat(y)\_c" part) is emphasized. - [11:9.871](https://academa.ai/lectures/least-squares-is-a-projection?t=669.8711666666667): decomposition (the "y\_c dot y\_c" part) is no longer emphasized. - [11:12.123](https://academa.ai/lectures/least-squares-is-a-projection?t=672.1231666666666): decomposition (the "e dot e" part) is emphasized. - [11:12.123](https://academa.ai/lectures/least-squares-is-a-projection?t=672.1231666666666): decomposition (the "hat(y)\_c dot hat(y)\_c" part) is no longer emphasized. - [11:13.946](https://academa.ai/lectures/least-squares-is-a-projection?t=673.9461666666666): decomposition (the "e dot e" part) is no longer emphasized. ##### [11:14.546](https://academa.ai/lectures/least-squares-is-a-projection?t=674.5461666666666) Narration: R squared is the explained squared length divided by the total squared length. Equivalently, it is one minus the unexplained squared length divided by the total. Board: Unchanged from the preceding beat in this scene. Actions: - [11:14.894](https://academa.ai/lectures/least-squares-is-a-projection?t=674.8941666666667): r\_squared is shown on the screen, written out. - [11:15.962](https://academa.ai/lectures/least-squares-is-a-projection?t=675.9621666666667): r\_squared (the "frac(hat(y)\_c dot hat(y)\_c, y\_c dot y\_c)" part) is emphasized. - [11:19.828](https://academa.ai/lectures/least-squares-is-a-projection?t=679.8281666666667): r\_squared (the "1 - frac(e dot e, y\_c dot y\_c)" part) is emphasized. - [11:19.828](https://academa.ai/lectures/least-squares-is-a-projection?t=679.8281666666667): r\_squared (the "frac(hat(y)\_c dot hat(y)\_c, y\_c dot y\_c)" part) is no longer emphasized. - [11:25.018](https://academa.ai/lectures/least-squares-is-a-projection?t=685.0181666666666): r\_squared moves to a new place on the board. - [11:25.018](https://academa.ai/lectures/least-squares-is-a-projection?t=685.0181666666666): decomposition is hidden from the screen — left the board. - [11:25.018](https://academa.ai/lectures/least-squares-is-a-projection?t=685.0181666666666): heading is hidden from the screen — left the board. - [11:25.018](https://academa.ai/lectures/least-squares-is-a-projection?t=685.0181666666666): r\_squared (the "1 - frac(e dot e, y\_c dot y\_c)" part) is no longer emphasized. ##### [11:26.218](https://academa.ai/lectures/least-squares-is-a-projection?t=686.2181666666667) Narration: In the three-observation example, the total squared length is fourteen thirds. The explained squared length is nine halves, and the residual squared length is one sixth. Board: r\_squared — a Math \[text\] that says "$R^2 = frac(hat(y)\_c dot hat(y)\_c, y\_c dot y\_c) = 1 - frac(e dot e, y\_c dot y\_c)$"; total\_arrow — a Vector \[yellow\] labelled "y\_c" drawn in triangle (end=(2.1213203436, 0.4082482905)); explained\_arrow — a Vector \[green\] labelled "hat(y)\_c" drawn in triangle (end=(2.1213203436, 0.0)); residual\_arrow — a Vector \[red\] labelled "e" drawn in triangle (start=(2.1213203436, 0.0), end=(2.1213203436, 0.4082482905)); right\_angle — an Angle \[yellow\] drawn in triangle (vertex=(2.1213203436, 0.0), sides=((0.0, 0.0), (2.1213203436, 0.4082482905)), right\_angle=True) Actions: - [11:29.828](https://academa.ai/lectures/least-squares-is-a-projection?t=689.8281666666667): numeric\_0 is shown on the screen, written out. - [11:32.87](https://academa.ai/lectures/least-squares-is-a-projection?t=692.8701666666667): numeric\_1 is shown on the screen, written out. - [11:35.494](https://academa.ai/lectures/least-squares-is-a-projection?t=695.4941666666666): numeric\_2 is shown on the screen, written out. ##### [11:37.244](https://academa.ai/lectures/least-squares-is-a-projection?t=697.2436666666666) Narration: Nine halves plus one sixth is fourteen thirds, exactly as the right triangle requires. The resulting R squared is twenty-seven over twenty-eight, about zero point nine six four. Board: r\_squared — a Math \[text\] that says "$R^2 = frac(hat(y)\_c dot hat(y)\_c, y\_c dot y\_c) = 1 - frac(e dot e, y\_c dot y\_c)$"; total\_arrow — a Vector \[yellow\] labelled "y\_c" drawn in triangle (end=(2.1213203436, 0.4082482905)); explained\_arrow — a Vector \[green\] labelled "hat(y)\_c" drawn in triangle (end=(2.1213203436, 0.0)); residual\_arrow — a Vector \[red\] labelled "e" drawn in triangle (start=(2.1213203436, 0.0), end=(2.1213203436, 0.4082482905)); right\_angle — an Angle \[yellow\] drawn in triangle (vertex=(2.1213203436, 0.0), sides=((0.0, 0.0), (2.1213203436, 0.4082482905)), right\_angle=True); numeric\_0 — a Math \[text\] that says "$y\_c dot y\_c = frac(14, 3)$"; numeric\_1 — a Math \[text\] that says "$hat(y)\_c dot hat(y)\_c = frac(9, 2)$"; numeric\_2 — a Math \[text\] that says "$e dot e = frac(1, 6)$" Actions: - [11:43.884](https://academa.ai/lectures/least-squares-is-a-projection?t=703.8841666666667): numeric\_3 is shown on the screen, written out. - [11:48.552](https://academa.ai/lectures/least-squares-is-a-projection?t=708.5521666666666): A box is drawn around numeric\_3. ##### [11:50](https://academa.ai/lectures/least-squares-is-a-projection?t=709.9996666666666) Narration: There is also an angle interpretation. Let theta be the angle between the centred response and its fitted projection. Then R squared is cosine squared theta. Board: r\_squared — a Math \[text\] that says "$R^2 = frac(hat(y)\_c dot hat(y)\_c, y\_c dot y\_c) = 1 - frac(e dot e, y\_c dot y\_c)$"; total\_arrow — a Vector \[yellow\] labelled "y\_c" drawn in triangle (end=(2.1213203436, 0.4082482905)); explained\_arrow — a Vector \[green\] labelled "hat(y)\_c" drawn in triangle (end=(2.1213203436, 0.0)); residual\_arrow — a Vector \[red\] labelled "e" drawn in triangle (start=(2.1213203436, 0.0), end=(2.1213203436, 0.4082482905)); right\_angle — an Angle \[yellow\] drawn in triangle (vertex=(2.1213203436, 0.0), sides=((0.0, 0.0), (2.1213203436, 0.4082482905)), right\_angle=True); numeric\_0 — a Math \[text\] that says "$y\_c dot y\_c = frac(14, 3)$"; numeric\_1 — a Math \[text\] that says "$hat(y)\_c dot hat(y)\_c = frac(9, 2)$"; numeric\_2 — a Math \[text\] that says "$e dot e = frac(1, 6)$"; numeric\_3 — a Math \[text\] that says "$R^2 = frac(9/2, 14/3) = frac(27, 28) approx 0.964$" Actions: - [11:51.114](https://academa.ai/lectures/least-squares-is-a-projection?t=711.1141666666666): theta\_angle is shown on the screen, written out. - [11:59.635](https://academa.ai/lectures/least-squares-is-a-projection?t=719.6351666666667): angle\_form is shown on the screen, written out. ##### [12:2.012](https://academa.ai/lectures/least-squares-is-a-projection?t=722.0116666666667) Narration: A small angle means the response lies close to the model subspace, so the projection keeps nearly all its length. A large angle means a larger perpendicular residual. Board: r\_squared — a Math \[text\] that says "$R^2 = frac(hat(y)\_c dot hat(y)\_c, y\_c dot y\_c) = 1 - frac(e dot e, y\_c dot y\_c)$"; angle\_form — a Math \[text\] that says "$R^2 = cos^2 theta$"; total\_arrow — a Vector \[yellow\] labelled "y\_c" drawn in triangle (end=(2.1213203436, 0.4082482905)); explained\_arrow — a Vector \[green\] labelled "hat(y)\_c" drawn in triangle (end=(2.1213203436, 0.0)); residual\_arrow — a Vector \[red\] labelled "e" drawn in triangle (start=(2.1213203436, 0.0), end=(2.1213203436, 0.4082482905)); right\_angle — an Angle \[yellow\] drawn in triangle (vertex=(2.1213203436, 0.0), sides=((0.0, 0.0), (2.1213203436, 0.4082482905)), right\_angle=True); numeric\_0 — a Math \[text\] that says "$y\_c dot y\_c = frac(14, 3)$"; numeric\_1 — a Math \[text\] that says "$hat(y)\_c dot hat(y)\_c = frac(9, 2)$"; numeric\_2 — a Math \[text\] that says "$e dot e = frac(1, 6)$"; numeric\_3 — a Math \[text\] that says "$R^2 = frac(9/2, 14/3) = frac(27, 28) approx 0.964$"; theta\_angle — an Angle \[blue\] labelled "theta" drawn in triangle (sides=((2.1213203436, 0.0), (2.1213203436, 0.4082482905)), radius=0.48) Actions: - [12:3.637](https://academa.ai/lectures/least-squares-is-a-projection?t=723.6371666666666): total\_arrow is indicated — a transient flash. - [12:11.601](https://academa.ai/lectures/least-squares-is-a-projection?t=731.6011666666667): residual\_arrow is indicated — a transient flash. ##### [12:13.235](https://academa.ai/lectures/least-squares-is-a-projection?t=733.2346666666666) Narration: With ordinary least squares and an intercept, this training-sample ratio lies between zero and one. Zero means the predictors improve nothing beyond the mean. One means every residual is zero. Board: Unchanged from the preceding beat in this scene. Actions: - [12:17.589](https://academa.ai/lectures/least-squares-is-a-projection?t=737.5891666666666): r\_squared is indicated — a transient flash. - [12:20.468](https://academa.ai/lectures/least-squares-is-a-projection?t=740.4681666666667): residual\_arrow is indicated — a transient flash. ##### [12:27.511](https://academa.ai/lectures/least-squares-is-a-projection?t=747.5111666666667) Narration: The intercept condition matters. Without the mean direction in the model, this familiar centred Pythagorean decomposition need not hold, and the usual interpretation of R squared needs care. Board: Unchanged from the preceding beat in this scene. Actions: - [12:30.785](https://academa.ai/lectures/least-squares-is-a-projection?t=750.7851666666667): total\_arrow is indicated — a transient flash. ##### [12:41.195](https://academa.ai/lectures/least-squares-is-a-projection?t=761.1951666666666) Narration: R squared measures alignment with the fitted subspace, not causation, and not whether the model will predict new data well. It is a geometric summary of in-sample explained squared length. Board: Unchanged from the preceding beat in this scene. Actions: - [12:42.809](https://academa.ai/lectures/least-squares-is-a-projection?t=762.8091666666667): angle\_form is indicated — a transient flash. ##### [12:54.323](https://academa.ai/lectures/least-squares-is-a-projection?t=774.3226666666667) Narration: The projection picture has now given us residual minimization, the normal equations, and R squared. One final geometric question remains: what if the predictor directions themselves almost collapse together? Board: Unchanged from the preceding beat in this scene. Actions: - [13:8.06](https://academa.ai/lectures/least-squares-is-a-projection?t=788.0596875): angle\_form is hidden from the screen — left the board. - [13:8.06](https://academa.ai/lectures/least-squares-is-a-projection?t=788.0596875): numeric\_0 is hidden from the screen — left the board. - [13:8.06](https://academa.ai/lectures/least-squares-is-a-projection?t=788.0596875): numeric\_1 is hidden from the screen — left the board. - [13:8.06](https://academa.ai/lectures/least-squares-is-a-projection?t=788.0596875): numeric\_2 is hidden from the screen — left the board. - [13:8.06](https://academa.ai/lectures/least-squares-is-a-projection?t=788.0596875): numeric\_3 is hidden from the screen — left the board. - [13:8.06](https://academa.ai/lectures/least-squares-is-a-projection?t=788.0596875): r\_squared is hidden from the screen — left the board. ### Scene 5: [When Predictor Directions Nearly Align](https://academa.ai/lectures/least-squares-is-a-projection?t=789.1013541666666) Span: 13:9.101–18:1.794 (789.1013541666666s–1081.7938333333332s). #### Objects - basis\_angle: an Angle \[yellow\] labelled "theta" drawn in figure (sides=((1.0, 0.0), (cos(theta), sin(theta))), radius=0.34) - coefficients: a Math \[text\] that says "$hat(beta) approx vec(-0.38, 1.15)$" - component\_1: a Vector \[blue\] labelled "hat(beta)\_1 x\_1" drawn in figure (end=((0.2 - ((target\_y / sin(theta)) \* cos(theta))), 0.0)) - component\_2: a Vector \[green\] labelled "hat(beta)\_2 x\_2" drawn in figure (start=((0.2 - ((target\_y / sin(theta)) \* cos(theta))), 0.0), end=(0.2, \)) - condition\_number: a Math \[text\] that says "$kappa(X) = sqrt(frac(1+cos theta, 1-cos theta))$" - determinant: a Math \[text\] that says "$d = 1 - cos^2 theta = sin^2 theta$" - exact\_dependence: a Math \[text\] that says "$theta=0 arrow.r x\_2=x\_1 arrow.r upright("rank")(X)\<2$" - figure: a Figure (x\_range=(-6.25, 1.25), y\_range=(-0.4, 1.7), aspect=(7.5, 2.1)) - fit\_arrow: a Vector \[red\] labelled "hat(y)" drawn in figure (end=(0.2, \)) - gram: a Math \[text\] that says "$X^T X = mat(1, cos theta; cos theta, 1)$" - heading: a Heading that says "Two Predictor Directions" - heading\_2: a Heading that says "A Small Change in the Target" - heading\_3: a Heading that says "Why the Coordinates Become Unstable" - heading\_4: a Heading that says "What Multicollinearity Means" - item\_1: a Text \[text\] that says "Nearly parallel columns provide almost the same direction." - item\_2: a Text \[text\] that says "Large opposite coefficients can produce one modest fitted vector." - item\_3: a Text \[text\] that says "Small data changes can cause much larger coefficient changes." - item\_4: a Text \[text\] that says "Predictions may remain stable while individual effects are unstable." - numeric\_det: a Math \[text\] that says "$theta=10 degree: quad d approx 0.030$" - numeric\_kappa: a Math \[text\] that says "$kappa(X) approx 11.4$" - perturb\_beta: a Math \[text\] that says "$hat(beta) approx vec(-5.47, 5.76)$" - perturb\_fit: a Math \[text\] that says "$hat(y) = vec(0.20, 1.00)$" - target\_y: a VariableNumber (initial\_value=1.0) - theta: a VariableNumber (initial\_value=1.0471975511965976) - x\_1: a Vector \[blue\] labelled "x\_1" drawn in figure - x\_2: a Vector \[green\] labelled "x\_2" drawn in figure (end=(cos(theta), sin(theta))) #### Beats ##### [13:9.101](https://academa.ai/lectures/least-squares-is-a-projection?t=789.1013541666666) Narration: To see multicollinearity, first remove the intercept direction by centring, and scale two predictor columns to unit length. Then their relationship is summarized by the angle theta between their vectors. Board: Empty. Actions: - [13:9.101](https://academa.ai/lectures/least-squares-is-a-projection?t=789.1013541666666): heading is shown on the screen, written out. - [13:13.873](https://academa.ai/lectures/least-squares-is-a-projection?t=793.8733541666667): x\_1 is shown on the screen, written out. - [13:13.873](https://academa.ai/lectures/least-squares-is-a-projection?t=793.8733541666667): x\_2 is shown on the screen, written out. - [13:18.61](https://academa.ai/lectures/least-squares-is-a-projection?t=798.6103541666666): basis\_angle is shown on the screen, written out. - [13:20.072](https://academa.ai/lectures/least-squares-is-a-projection?t=800.0723541666666): figure is shown on the screen, written out. ##### [13:21.752](https://academa.ai/lectures/least-squares-is-a-projection?t=801.7523541666666) Narration: At sixty degrees, the blue and green predictors provide clearly different directions. Together they span the plane, and a fitted vector has coordinates beta one and beta two in this predictor basis. Board: figure — a Figure (x\_range=(-6.25, 1.25), y\_range=(-0.4, 1.7), aspect=(7.5, 2.1)); heading — a Heading that says "Two Predictor Directions"; x\_1 — a Vector \[blue\] labelled "x\_1" drawn in figure; x\_2 — a Vector \[green\] labelled "x\_2" drawn in figure (end=(cos(theta), sin(theta))); basis\_angle — an Angle \[yellow\] labelled "theta" drawn in figure (sides=((1.0, 0.0), (cos(theta), sin(theta))), radius=0.34) Actions: - [13:29.252](https://academa.ai/lectures/least-squares-is-a-projection?t=809.2523541666666): fit\_arrow is shown on the screen, written out. ##### [13:34.206](https://academa.ai/lectures/least-squares-is-a-projection?t=814.2058541666667) Narration: Draw the coefficient contributions tip to tail. The blue contribution first moves along x one. The green contribution then moves along x two, and their sum lands exactly at the red fitted vector. Board: figure — a Figure (x\_range=(-6.25, 1.25), y\_range=(-0.4, 1.7), aspect=(7.5, 2.1)); heading — a Heading that says "Two Predictor Directions"; x\_1 — a Vector \[blue\] labelled "x\_1" drawn in figure; x\_2 — a Vector \[green\] labelled "x\_2" drawn in figure (end=(cos(theta), sin(theta))); basis\_angle — an Angle \[yellow\] labelled "theta" drawn in figure (sides=((1.0, 0.0), (cos(theta), sin(theta))), radius=0.34); fit\_arrow — a Vector \[red\] labelled "hat(y)" drawn in figure (end=(0.2, \)) Actions: - [13:37.99](https://academa.ai/lectures/least-squares-is-a-projection?t=817.9903541666666): component\_1 is shown on the screen, written out. - [13:41.078](https://academa.ai/lectures/least-squares-is-a-projection?t=821.0783541666666): component\_2 is shown on the screen, written out. - [13:45.606](https://academa.ai/lectures/least-squares-is-a-projection?t=825.6063541666666): coefficients is shown on the screen, written out. ##### [13:47.6](https://academa.ai/lectures/least-squares-is-a-projection?t=827.5998541666667) Narration: These coefficients are coordinates, not intrinsic lengths. Their values depend on how well the chosen predictor directions form a coordinate system. Board: figure — a Figure (x\_range=(-6.25, 1.25), y\_range=(-0.4, 1.7), aspect=(7.5, 2.1)); coefficients — a Math \[text\] that says "$hat(beta) approx vec(-0.38, 1.15)$"; heading — a Heading that says "Two Predictor Directions"; x\_1 — a Vector \[blue\] labelled "x\_1" drawn in figure; x\_2 — a Vector \[green\] labelled "x\_2" drawn in figure (end=(cos(theta), sin(theta))); basis\_angle — an Angle \[yellow\] labelled "theta" drawn in figure (sides=((1.0, 0.0), (cos(theta), sin(theta))), radius=0.34); fit\_arrow — a Vector \[red\] labelled "hat(y)" drawn in figure (end=(0.2, \)); component\_1 — a Vector \[blue\] labelled "hat(beta)\_1 x\_1" drawn in figure (end=((0.2 - ((target\_y / sin(theta)) \* cos(theta))), 0.0)); component\_2 — a Vector \[green\] labelled "hat(beta)\_2 x\_2" drawn in figure (start=((0.2 - ((target\_y / sin(theta)) \* cos(theta))), 0.0), end=(0.2, \)) Actions: - [13:49.097](https://academa.ai/lectures/least-squares-is-a-projection?t=829.0973541666666): component\_1 is indicated — a transient flash. - [13:49.097](https://academa.ai/lectures/least-squares-is-a-projection?t=829.0973541666666): component\_2 is indicated — a transient flash. ##### [13:57.023](https://academa.ai/lectures/least-squares-is-a-projection?t=837.0228541666667) Narration: Now make the green predictor more like the blue one. At thirty degrees, the span is still the whole plane, and the fitted vector has not moved. But the two coefficient contributions must grow and cancel more strongly. Board: Unchanged from the preceding beat in this scene. Actions: - [14:0.61](https://academa.ai/lectures/least-squares-is-a-projection?t=840.6103541666666): x\_2 is redrawn as the numbers it depends on change. - [14:0.61](https://academa.ai/lectures/least-squares-is-a-projection?t=840.6103541666666): basis\_angle is redrawn as the numbers it depends on change. - [14:0.61](https://academa.ai/lectures/least-squares-is-a-projection?t=840.6103541666666): component\_1 is redrawn as the numbers it depends on change. - [14:0.61](https://academa.ai/lectures/least-squares-is-a-projection?t=840.6103541666666): component\_2 is redrawn as the numbers it depends on change. - [14:0.61](https://academa.ai/lectures/least-squares-is-a-projection?t=840.6103541666666): theta ticks to 0.5235987755982988. - [14:9.666](https://academa.ai/lectures/least-squares-is-a-projection?t=849.6663541666667): coefficients becomes "$hat(beta) approx vec(-1.53, 2.00)$". ##### [14:10.266](https://academa.ai/lectures/least-squares-is-a-projection?t=850.2663541666666) Narration: Bring the angle down to ten degrees. The predictor arrows are now nearly parallel. To reach the same modest red vector, one long component runs left and the other almost cancels it while climbing to the target. Board: Unchanged from the preceding beat in this scene. Actions: - [14:10.521](https://academa.ai/lectures/least-squares-is-a-projection?t=850.5213541666666): x\_2 is redrawn as the numbers it depends on change. - [14:10.521](https://academa.ai/lectures/least-squares-is-a-projection?t=850.5213541666666): basis\_angle is redrawn as the numbers it depends on change. - [14:10.521](https://academa.ai/lectures/least-squares-is-a-projection?t=850.5213541666666): component\_1 is redrawn as the numbers it depends on change. - [14:10.521](https://academa.ai/lectures/least-squares-is-a-projection?t=850.5213541666666): component\_2 is redrawn as the numbers it depends on change. - [14:10.521](https://academa.ai/lectures/least-squares-is-a-projection?t=850.5213541666666): theta ticks to 0.17453292519943295. - [14:19.914](https://academa.ai/lectures/least-squares-is-a-projection?t=859.9143541666666): component\_1 is indicated — a transient flash. - [14:21.376](https://academa.ai/lectures/least-squares-is-a-projection?t=861.3763541666666): component\_2 is indicated — a transient flash. - [14:23.409](https://academa.ai/lectures/least-squares-is-a-projection?t=863.4088541666666): coefficients becomes "$hat(beta) approx vec(-5.47, 5.76)$". ##### [14:24.009](https://academa.ai/lectures/least-squares-is-a-projection?t=864.0088541666667) Narration: This is multicollinearity. Two columns of X supply nearly the same direction in observation space, so the model has difficulty deciding how much of the fit to assign to each one. Board: Unchanged from the preceding beat in this scene. Actions: - [14:26.319](https://academa.ai/lectures/least-squares-is-a-projection?t=866.3193541666666): x\_1 is indicated — a transient flash. - [14:28.315](https://academa.ai/lectures/least-squares-is-a-projection?t=868.3153541666666): x\_2 is indicated — a transient flash. ##### [14:35.51](https://academa.ai/lectures/least-squares-is-a-projection?t=875.5098541666666) Narration: The fitted vector itself can still be perfectly clear. What becomes unstable is its description in this nearly collapsed coordinate system. Board: Unchanged from the preceding beat in this scene. Actions: - [14:35.986](https://academa.ai/lectures/least-squares-is-a-projection?t=875.9863541666666): fit\_arrow is indicated — a transient flash. - [14:40.711](https://academa.ai/lectures/least-squares-is-a-projection?t=880.7113541666666): component\_1 is indicated — a transient flash. - [14:43.95](https://academa.ai/lectures/least-squares-is-a-projection?t=883.9503541666667): figure moves to a new place on the board. - [14:43.95](https://academa.ai/lectures/least-squares-is-a-projection?t=883.9503541666667): coefficients is hidden from the screen — left the board. - [14:43.95](https://academa.ai/lectures/least-squares-is-a-projection?t=883.9503541666667): heading is hidden from the screen — left the board. ##### [14:45.15](https://academa.ai/lectures/least-squares-is-a-projection?t=885.1503541666666) Narration: Here is a small perturbation. Raise the fitted target's second coordinate from one point zero zero to one point zero two, only two hundredths. Board: figure — a Figure (x\_range=(-6.25, 1.25), y\_range=(-0.4, 1.7), aspect=(7.5, 2.1)); x\_1 — a Vector \[blue\] labelled "x\_1" drawn in figure; x\_2 — a Vector \[green\] labelled "x\_2" drawn in figure (end=(cos(theta), sin(theta))); basis\_angle — an Angle \[yellow\] labelled "theta" drawn in figure (sides=((1.0, 0.0), (cos(theta), sin(theta))), radius=0.34); fit\_arrow — a Vector \[red\] labelled "hat(y)" drawn in figure (end=(0.2, \)); component\_1 — a Vector \[blue\] labelled "hat(beta)\_1 x\_1" drawn in figure (end=((0.2 - ((target\_y / sin(theta)) \* cos(theta))), 0.0)); component\_2 — a Vector \[green\] labelled "hat(beta)\_2 x\_2" drawn in figure (start=((0.2 - ((target\_y / sin(theta)) \* cos(theta))), 0.0), end=(0.2, \)) Actions: - [14:45.15](https://academa.ai/lectures/least-squares-is-a-projection?t=885.1503541666666): perturb\_fit is shown on the screen, written out. - [14:45.15](https://academa.ai/lectures/least-squares-is-a-projection?t=885.1503541666666): perturb\_beta is shown on the screen, written out. - [14:47.878](https://academa.ai/lectures/least-squares-is-a-projection?t=887.8783541666667): fit\_arrow is redrawn as the numbers it depends on change. - [14:47.878](https://academa.ai/lectures/least-squares-is-a-projection?t=887.8783541666667): component\_1 is redrawn as the numbers it depends on change. - [14:47.878](https://academa.ai/lectures/least-squares-is-a-projection?t=887.8783541666667): component\_2 is redrawn as the numbers it depends on change. - [14:47.878](https://academa.ai/lectures/least-squares-is-a-projection?t=887.8783541666667): target\_y ticks to 1.02. ##### [14:55.41](https://academa.ai/lectures/least-squares-is-a-projection?t=895.4098541666666) Narration: The red endpoint barely moves, but each coefficient changes by about zero point one one. The coordinate change is almost six times the size of the data perturbation, with opposite signs. Board: figure — a Figure (x\_range=(-6.25, 1.25), y\_range=(-0.4, 1.7), aspect=(7.5, 2.1)); x\_1 — a Vector \[blue\] labelled "x\_1" drawn in figure; x\_2 — a Vector \[green\] labelled "x\_2" drawn in figure (end=(cos(theta), sin(theta))); basis\_angle — an Angle \[yellow\] labelled "theta" drawn in figure (sides=((1.0, 0.0), (cos(theta), sin(theta))), radius=0.34); fit\_arrow — a Vector \[red\] labelled "hat(y)" drawn in figure (end=(0.2, \)); component\_1 — a Vector \[blue\] labelled "hat(beta)\_1 x\_1" drawn in figure (end=((0.2 - ((target\_y / sin(theta)) \* cos(theta))), 0.0)); component\_2 — a Vector \[green\] labelled "hat(beta)\_2 x\_2" drawn in figure (start=((0.2 - ((target\_y / sin(theta)) \* cos(theta))), 0.0), end=(0.2, \)); perturb\_fit — a Math \[text\] that says "$hat(y) = vec(0.20, 1.00)$"; perturb\_beta — a Math \[text\] that says "$hat(beta) approx vec(-5.47, 5.76)$" Actions: - [14:57.964](https://academa.ai/lectures/least-squares-is-a-projection?t=897.9643541666667): perturb\_beta becomes "$hat(beta) approx vec(-5.59, 5.87)$". - [15:5.325](https://academa.ai/lectures/least-squares-is-a-projection?t=905.3253541666667): component\_1 is indicated — a transient flash. - [15:5.812](https://academa.ai/lectures/least-squares-is-a-projection?t=905.8123541666666): component\_2 is indicated — a transient flash. - [15:6.683](https://academa.ai/lectures/least-squares-is-a-projection?t=906.6833541666666): figure moves to a new place on the board. - [15:6.683](https://academa.ai/lectures/least-squares-is-a-projection?t=906.6833541666666): perturb\_beta is hidden from the screen — left the board. - [15:6.683](https://academa.ai/lectures/least-squares-is-a-projection?t=906.6833541666666): perturb\_fit is hidden from the screen — left the board. ##### [15:7.883](https://academa.ai/lectures/least-squares-is-a-projection?t=907.8833541666667) Narration: The algebra reports the same geometry. For standardized columns, X transpose X has ones on the diagonal and cosine theta off the diagonal. Board: figure — a Figure (x\_range=(-6.25, 1.25), y\_range=(-0.4, 1.7), aspect=(7.5, 2.1)); x\_1 — a Vector \[blue\] labelled "x\_1" drawn in figure; x\_2 — a Vector \[green\] labelled "x\_2" drawn in figure (end=(cos(theta), sin(theta))); basis\_angle — an Angle \[yellow\] labelled "theta" drawn in figure (sides=((1.0, 0.0), (cos(theta), sin(theta))), radius=0.34); fit\_arrow — a Vector \[red\] labelled "hat(y)" drawn in figure (end=(0.2, \)); component\_1 — a Vector \[blue\] labelled "hat(beta)\_1 x\_1" drawn in figure (end=((0.2 - ((target\_y / sin(theta)) \* cos(theta))), 0.0)); component\_2 — a Vector \[green\] labelled "hat(beta)\_2 x\_2" drawn in figure (start=((0.2 - ((target\_y / sin(theta)) \* cos(theta))), 0.0), end=(0.2, \)) Actions: - [15:7.883](https://academa.ai/lectures/least-squares-is-a-projection?t=907.8833541666667): gram is shown on the screen, written out. - [15:16.045](https://academa.ai/lectures/least-squares-is-a-projection?t=916.0453541666667): gram (the "cos theta" part) is emphasized. - [15:18.437](https://academa.ai/lectures/least-squares-is-a-projection?t=918.4368541666666): gram (the "cos theta" part) is no longer emphasized. ##### [15:19.037](https://academa.ai/lectures/least-squares-is-a-projection?t=919.0368541666667) Narration: Its determinant is one minus cosine squared theta, which is sine squared theta. As the predictor angle shrinks, this determinant approaches zero. Board: figure — a Figure (x\_range=(-6.25, 1.25), y\_range=(-0.4, 1.7), aspect=(7.5, 2.1)); x\_1 — a Vector \[blue\] labelled "x\_1" drawn in figure; x\_2 — a Vector \[green\] labelled "x\_2" drawn in figure (end=(cos(theta), sin(theta))); basis\_angle — an Angle \[yellow\] labelled "theta" drawn in figure (sides=((1.0, 0.0), (cos(theta), sin(theta))), radius=0.34); fit\_arrow — a Vector \[red\] labelled "hat(y)" drawn in figure (end=(0.2, \)); component\_1 — a Vector \[blue\] labelled "hat(beta)\_1 x\_1" drawn in figure (end=((0.2 - ((target\_y / sin(theta)) \* cos(theta))), 0.0)); component\_2 — a Vector \[green\] labelled "hat(beta)\_2 x\_2" drawn in figure (start=((0.2 - ((target\_y / sin(theta)) \* cos(theta))), 0.0), end=(0.2, \)); gram — a Math \[text\] that says "$X^T X = mat(1, cos theta; cos theta, 1)$" Actions: - [15:19.629](https://academa.ai/lectures/least-squares-is-a-projection?t=919.6293541666666): determinant is shown on the screen, written out. - [15:22.682](https://academa.ai/lectures/least-squares-is-a-projection?t=922.6823541666666): determinant (the "sin^2 theta" part) is emphasized. - [15:28.777](https://academa.ai/lectures/least-squares-is-a-projection?t=928.7773541666667): determinant (the "sin^2 theta" part) is no longer emphasized. ##### [15:29.377](https://academa.ai/lectures/least-squares-is-a-projection?t=929.3773541666667) Narration: At exactly zero degrees the two columns coincide. The span collapses from a plane to a line, X loses rank, and infinitely many coefficient pairs can describe the same fitted vector. Board: figure — a Figure (x\_range=(-6.25, 1.25), y\_range=(-0.4, 1.7), aspect=(7.5, 2.1)); x\_1 — a Vector \[blue\] labelled "x\_1" drawn in figure; x\_2 — a Vector \[green\] labelled "x\_2" drawn in figure (end=(cos(theta), sin(theta))); basis\_angle — an Angle \[yellow\] labelled "theta" drawn in figure (sides=((1.0, 0.0), (cos(theta), sin(theta))), radius=0.34); fit\_arrow — a Vector \[red\] labelled "hat(y)" drawn in figure (end=(0.2, \)); component\_1 — a Vector \[blue\] labelled "hat(beta)\_1 x\_1" drawn in figure (end=((0.2 - ((target\_y / sin(theta)) \* cos(theta))), 0.0)); component\_2 — a Vector \[green\] labelled "hat(beta)\_2 x\_2" drawn in figure (start=((0.2 - ((target\_y / sin(theta)) \* cos(theta))), 0.0), end=(0.2, \)); gram — a Math \[text\] that says "$X^T X = mat(1, cos theta; cos theta, 1)$"; determinant — a Math \[text\] that says "$d = 1 - cos^2 theta = sin^2 theta$" Actions: - [15:30.41](https://academa.ai/lectures/least-squares-is-a-projection?t=930.4103541666666): exact\_dependence is shown on the screen, written out. - [15:31.873](https://academa.ai/lectures/least-squares-is-a-projection?t=931.8733541666667): x\_1 is indicated — a transient flash. - [15:31.873](https://academa.ai/lectures/least-squares-is-a-projection?t=931.8733541666667): x\_2 is indicated — a transient flash. ##### [15:42.609](https://academa.ai/lectures/least-squares-is-a-projection?t=942.6093541666667) Narration: Near zero, the inverse still exists in exact arithmetic, but it must magnify the thin direction enormously. That sensitivity is called ill conditioning. Board: figure — a Figure (x\_range=(-6.25, 1.25), y\_range=(-0.4, 1.7), aspect=(7.5, 2.1)); x\_1 — a Vector \[blue\] labelled "x\_1" drawn in figure; x\_2 — a Vector \[green\] labelled "x\_2" drawn in figure (end=(cos(theta), sin(theta))); basis\_angle — an Angle \[yellow\] labelled "theta" drawn in figure (sides=((1.0, 0.0), (cos(theta), sin(theta))), radius=0.34); fit\_arrow — a Vector \[red\] labelled "hat(y)" drawn in figure (end=(0.2, \)); component\_1 — a Vector \[blue\] labelled "hat(beta)\_1 x\_1" drawn in figure (end=((0.2 - ((target\_y / sin(theta)) \* cos(theta))), 0.0)); component\_2 — a Vector \[green\] labelled "hat(beta)\_2 x\_2" drawn in figure (start=((0.2 - ((target\_y / sin(theta)) \* cos(theta))), 0.0), end=(0.2, \)); exact\_dependence — a Math \[text\] that says "$theta=0 arrow.r x\_2=x\_1 arrow.r upright("rank")(X)\<2$"; gram — a Math \[text\] that says "$X^T X = mat(1, cos theta; cos theta, 1)$"; determinant — a Math \[text\] that says "$d = 1 - cos^2 theta = sin^2 theta$" Actions: - [15:50.492](https://academa.ai/lectures/least-squares-is-a-projection?t=950.4923541666666): condition\_number is shown on the screen, written out. ##### [15:53.531](https://academa.ai/lectures/least-squares-is-a-projection?t=953.5308541666666) Narration: For two standardized predictors, this condition number is the square root of one plus cosine theta over one minus cosine theta. At ten degrees, the Gram determinant is about zero point zero three and the condition number is already about eleven point four. Board: figure — a Figure (x\_range=(-6.25, 1.25), y\_range=(-0.4, 1.7), aspect=(7.5, 2.1)); x\_1 — a Vector \[blue\] labelled "x\_1" drawn in figure; x\_2 — a Vector \[green\] labelled "x\_2" drawn in figure (end=(cos(theta), sin(theta))); basis\_angle — an Angle \[yellow\] labelled "theta" drawn in figure (sides=((1.0, 0.0), (cos(theta), sin(theta))), radius=0.34); fit\_arrow — a Vector \[red\] labelled "hat(y)" drawn in figure (end=(0.2, \)); component\_1 — a Vector \[blue\] labelled "hat(beta)\_1 x\_1" drawn in figure (end=((0.2 - ((target\_y / sin(theta)) \* cos(theta))), 0.0)); component\_2 — a Vector \[green\] labelled "hat(beta)\_2 x\_2" drawn in figure (start=((0.2 - ((target\_y / sin(theta)) \* cos(theta))), 0.0), end=(0.2, \)); exact\_dependence — a Math \[text\] that says "$theta=0 arrow.r x\_2=x\_1 arrow.r upright("rank")(X)\<2$"; gram — a Math \[text\] that says "$X^T X = mat(1, cos theta; cos theta, 1)$"; determinant — a Math \[text\] that says "$d = 1 - cos^2 theta = sin^2 theta$"; condition\_number — a Math \[text\] that says "$kappa(X) = sqrt(frac(1+cos theta, 1-cos theta))$" Actions: - [15:56.037](https://academa.ai/lectures/least-squares-is-a-projection?t=956.0373541666667): numeric\_kappa is shown on the screen, written out. - [16:3.456](https://academa.ai/lectures/least-squares-is-a-projection?t=963.4563541666666): numeric\_det is shown on the screen, written out. - [16:8.936](https://academa.ai/lectures/least-squares-is-a-projection?t=968.9363541666667): numeric\_kappa (the "11.4" part) is indicated — a transient flash. ##### [16:10.338](https://academa.ai/lectures/least-squares-is-a-projection?t=970.3378541666666) Narration: A condition number near one means the predictor directions are well separated. A large value means small changes in y, rounding, or sampling can create much larger changes in the estimated coefficients. Board: figure — a Figure (x\_range=(-6.25, 1.25), y\_range=(-0.4, 1.7), aspect=(7.5, 2.1)); x\_1 — a Vector \[blue\] labelled "x\_1" drawn in figure; x\_2 — a Vector \[green\] labelled "x\_2" drawn in figure (end=(cos(theta), sin(theta))); basis\_angle — an Angle \[yellow\] labelled "theta" drawn in figure (sides=((1.0, 0.0), (cos(theta), sin(theta))), radius=0.34); fit\_arrow — a Vector \[red\] labelled "hat(y)" drawn in figure (end=(0.2, \)); component\_1 — a Vector \[blue\] labelled "hat(beta)\_1 x\_1" drawn in figure (end=((0.2 - ((target\_y / sin(theta)) \* cos(theta))), 0.0)); component\_2 — a Vector \[green\] labelled "hat(beta)\_2 x\_2" drawn in figure (start=((0.2 - ((target\_y / sin(theta)) \* cos(theta))), 0.0), end=(0.2, \)); exact\_dependence — a Math \[text\] that says "$theta=0 arrow.r x\_2=x\_1 arrow.r upright("rank")(X)\<2$"; gram — a Math \[text\] that says "$X^T X = mat(1, cos theta; cos theta, 1)$"; determinant — a Math \[text\] that says "$d = 1 - cos^2 theta = sin^2 theta$"; condition\_number — a Math \[text\] that says "$kappa(X) = sqrt(frac(1+cos theta, 1-cos theta))$"; numeric\_det — a Math \[text\] that says "$theta=10 degree: quad d approx 0.030$"; numeric\_kappa — a Math \[text\] that says "$kappa(X) approx 11.4$" Actions: - [16:15.609](https://academa.ai/lectures/least-squares-is-a-projection?t=975.6093541666667): condition\_number is indicated — a transient flash. ##### [16:23.895](https://academa.ai/lectures/least-squares-is-a-projection?t=983.8948541666666) Narration: This does not automatically make the fitted values unstable. Many different large, cancelling coefficient pairs can land at nearly the same projection point. Prediction within the observed predictor pattern may therefore look steady while individual coefficient interpretations do not. Board: Unchanged from the preceding beat in this scene. Actions: - [16:25.763](https://academa.ai/lectures/least-squares-is-a-projection?t=985.7633541666667): fit\_arrow is indicated — a transient flash. - [16:28.944](https://academa.ai/lectures/least-squares-is-a-projection?t=988.9443541666666): component\_1 is indicated — a transient flash. - [16:28.944](https://academa.ai/lectures/least-squares-is-a-projection?t=988.9443541666666): component\_2 is indicated — a transient flash. - [16:40.101](https://academa.ai/lectures/least-squares-is-a-projection?t=1000.1013541666666): condition\_number is hidden from the screen — left the board. - [16:40.101](https://academa.ai/lectures/least-squares-is-a-projection?t=1000.1013541666666): determinant is hidden from the screen — left the board. - [16:40.101](https://academa.ai/lectures/least-squares-is-a-projection?t=1000.1013541666666): exact\_dependence is hidden from the screen — left the board. - [16:40.101](https://academa.ai/lectures/least-squares-is-a-projection?t=1000.1013541666666): figure is hidden from the screen — left the board. - [16:40.101](https://academa.ai/lectures/least-squares-is-a-projection?t=1000.1013541666666): x\_1 is hidden from the screen — figure left the board. - [16:40.101](https://academa.ai/lectures/least-squares-is-a-projection?t=1000.1013541666666): x\_2 is hidden from the screen — figure left the board. - [16:40.101](https://academa.ai/lectures/least-squares-is-a-projection?t=1000.1013541666666): basis\_angle is hidden from the screen — figure left the board. - [16:40.101](https://academa.ai/lectures/least-squares-is-a-projection?t=1000.1013541666666): fit\_arrow is hidden from the screen — figure left the board. - [16:40.101](https://academa.ai/lectures/least-squares-is-a-projection?t=1000.1013541666666): component\_1 is hidden from the screen — figure left the board. - [16:40.101](https://academa.ai/lectures/least-squares-is-a-projection?t=1000.1013541666666): component\_2 is hidden from the screen — figure left the board. - [16:40.101](https://academa.ai/lectures/least-squares-is-a-projection?t=1000.1013541666666): gram is hidden from the screen — left the board. - [16:40.101](https://academa.ai/lectures/least-squares-is-a-projection?t=1000.1013541666666): numeric\_det is hidden from the screen — left the board. - [16:40.101](https://academa.ai/lectures/least-squares-is-a-projection?t=1000.1013541666666): numeric\_kappa is hidden from the screen — left the board. ##### [16:41.301](https://academa.ai/lectures/least-squares-is-a-projection?t=1001.3013541666667) Narration: The first lesson is geometric: nearly parallel columns provide almost the same direction. Board: Empty. Actions: - [16:41.835](https://academa.ai/lectures/least-squares-is-a-projection?t=1001.8353541666667): item\_1 is shown on the screen, written out. ##### [16:47.927](https://academa.ai/lectures/least-squares-is-a-projection?t=1007.9268541666667) Narration: The second is about coordinates: reaching one moderate fit may require large contributions with opposite signs. Board: item\_1 — a Text \[text\] that says "Nearly parallel columns provide almost the same direction." Actions: - [16:48.461](https://academa.ai/lectures/least-squares-is-a-projection?t=1008.4613541666666): item\_2 is shown on the screen, written out. ##### [16:55.69](https://academa.ai/lectures/least-squares-is-a-projection?t=1015.6903541666666) Narration: The third is statistical: a small change in the sample can move those large coefficients substantially, increasing their uncertainty. Board: item\_1 — a Text \[text\] that says "Nearly parallel columns provide almost the same direction."; item\_2 — a Text \[text\] that says "Large opposite coefficients can produce one modest fitted vector." Actions: - [16:56.189](https://academa.ai/lectures/least-squares-is-a-projection?t=1016.1893541666666): item\_3 is shown on the screen, written out. ##### [17:4.359](https://academa.ai/lectures/least-squares-is-a-projection?t=1024.3593541666667) Narration: And the fourth separates two questions. The model may predict a combined effect reasonably while the separate effects of the correlated predictors remain poorly determined. Board: item\_1 — a Text \[text\] that says "Nearly parallel columns provide almost the same direction."; item\_2 — a Text \[text\] that says "Large opposite coefficients can produce one modest fitted vector."; item\_3 — a Text \[text\] that says "Small data changes can cause much larger coefficient changes." Actions: - [17:4.986](https://academa.ai/lectures/least-squares-is-a-projection?t=1024.9863541666666): item\_4 is shown on the screen, written out. ##### [17:15.303](https://academa.ai/lectures/least-squares-is-a-projection?t=1035.3033541666666) Narration: Possible responses include collecting data where the predictors vary more independently, combining redundant predictors, choosing one of them, or using a regularized fit. Each response changes how the thin direction is handled, so it should follow the scientific purpose of the model. Board: item\_1 — a Text \[text\] that says "Nearly parallel columns provide almost the same direction."; item\_2 — a Text \[text\] that says "Large opposite coefficients can produce one modest fitted vector."; item\_3 — a Text \[text\] that says "Small data changes can cause much larger coefficient changes."; item\_4 — a Text \[text\] that says "Predictions may remain stable while individual effects are unstable." Actions: - None. ##### [17:33.748](https://academa.ai/lectures/least-squares-is-a-projection?t=1053.7478541666667) Narration: The whole lecture is now one geometric story. Data points create vertical residuals. Least squares chooses the shortest residual vector in observation space. Perpendicularity gives the normal equations. Board: Unchanged from the preceding beat in this scene. Actions: - None. ##### [17:48.686](https://academa.ai/lectures/least-squares-is-a-projection?t=1068.6858541666666) Narration: Pythagoras turns the same projection into R squared. And when predictor directions nearly align, the spanning coordinates become ill conditioned even if the projected fit barely changes. Board: Unchanged from the preceding beat in this scene. Actions: - [18:0.752](https://academa.ai/lectures/least-squares-is-a-projection?t=1080.7521666666667): item\_1 is hidden from the screen — left the board. - [18:0.752](https://academa.ai/lectures/least-squares-is-a-projection?t=1080.7521666666667): item\_2 is hidden from the screen — left the board. - [18:0.752](https://academa.ai/lectures/least-squares-is-a-projection?t=1080.7521666666667): item\_3 is hidden from the screen — left the board. - [18:0.752](https://academa.ai/lectures/least-squares-is-a-projection?t=1080.7521666666667): item\_4 is hidden from the screen — left the board.