WEBVTT 00:00:00.058 --> 00:00:02.776 We begin with five observed pairs. 00:00:03.088 --> 00:00:06.989 The horizontal coordinate is the predictor x, and the vertical coordinate 00:00:06.989 --> 00:00:08.766 is the response y. 00:00:08.963 --> 00:00:13.666 A regression line will summarize how the response changes as x changes. 00:00:14.462 --> 00:00:16.472 Here is one candidate line. 00:00:16.669 --> 00:00:19.894 For this example its height at the centre is fixed at three, 00:00:19.977 --> 00:00:22.138 and m controls its tilt. 00:00:22.334 --> 00:00:24.471 We have not yet chosen m. 00:00:24.850 --> 00:00:29.923 At each observed x value, the fitted line predicts a height. The vertical segment 00:00:29.923 --> 00:00:33.338 from that fitted height to the observed point is the residual. 00:00:33.821 --> 00:00:35.828 The sign records direction. 00:00:35.911 --> 00:00:38.881 A point above the line has a positive residual, 00:00:38.964 --> 00:00:41.575 and a point below it has a negative residual. 00:00:41.658 --> 00:00:45.548 But a good fit should penalize a large miss in either direction. 00:00:46.089 --> 00:00:49.619 So square every residual and add the squares. 00:00:50.071 --> 00:00:53.670 This sum of squared errors is nonnegative, it treats equal 00:00:53.670 --> 00:00:59.035 misses above and below equally, and large misses count much more than small ones. 00:00:59.947 --> 00:01:02.293 Notice what we are minimizing. 00:01:02.490 --> 00:01:05.672 These are vertical differences in the response coordinate. 00:01:05.868 --> 00:01:10.453 Ordinary regression of y on x does not use the shortest distance to the line; 00:01:10.536 --> 00:01:13.079 it holds each observed x fixed. 00:01:13.876 --> 00:01:16.918 With a horizontal line, m is zero. 00:01:17.115 --> 00:01:20.077 The five residuals have squared sum sixteen. 00:01:20.273 --> 00:01:23.734 The line is clearly missing the upward pattern in the data. 00:01:24.345 --> 00:01:26.282 To choose the tilt without calculus, 00:01:26.365 --> 00:01:29.721 let us watch every candidate and record its squared error. 00:01:29.917 --> 00:01:33.448 The same live value of m will drive both pictures. 00:01:34.844 --> 00:01:35.621 On the right, 00:01:35.704 --> 00:01:40.046 the red curve records the sum of squared errors for every slope in view. 00:01:40.359 --> 00:01:44.261 The yellow dot begins at slope zero and error sixteen. 00:01:44.872 --> 00:01:48.367 Tilt the fitted line upward to slope zero point six. 00:01:48.564 --> 00:01:50.350 The residual pattern tightens, 00:01:50.433 --> 00:01:54.126 and the squared error falls from sixteen to five point two. 00:01:54.829 --> 00:01:57.152 Continue to slope one point two. 00:01:57.349 --> 00:01:59.332 Several vertical misses are now small, 00:01:59.415 --> 00:02:02.179 and the dot reaches the bottom of the error curve. 00:02:02.283 --> 00:02:05.024 The squared error is one point six. 00:02:05.820 --> 00:02:08.572 Go past that slope to one point eight. 00:02:08.769 --> 00:02:12.775 Some residuals change sign, but their squares grow again. 00:02:13.088 --> 00:02:15.875 The error rises back to five point two. 00:02:16.486 --> 00:02:18.124 Return to the bottom. 00:02:18.227 --> 00:02:23.195 The line with slope one point two has the smallest sum of squared vertical residuals 00:02:23.278 --> 00:02:25.276 among this whole family. 00:02:25.588 --> 00:02:27.842 That is the least-squares fit. 00:02:28.638 --> 00:02:32.064 The curve on the right is not an appeal to differentiation. 00:02:32.168 --> 00:02:36.116 It is a geometric record of what the yellow segments on the left are doing. 00:02:36.312 --> 00:02:40.910 Tilting away from the best line makes their combined squared length grow. 00:02:41.707 --> 00:02:43.203 For this centred predictor, 00:02:43.286 --> 00:02:47.977 the intercept stayed at the mean response, three, while the slope changed. 00:02:48.081 --> 00:02:50.264 In a general fit, both intercept 00:02:50.264 --> 00:02:54.908 and slope are chosen together to minimize the same squared residual total. 00:02:55.705 --> 00:02:59.189 This data picture explains what least squares measures. 00:02:59.501 --> 00:03:02.797 To understand why the optimum has a clean algebraic form, 00:03:02.880 --> 00:03:05.609 we now change the geometry completely. 00:03:06.663 --> 00:03:10.054 Now replace the data plot by observation space. 00:03:10.251 --> 00:03:14.396 To draw it, we use a miniature data set with three observations. 00:03:14.593 --> 00:03:16.983 Three observations give three coordinates, 00:03:17.066 --> 00:03:21.072 so the whole construction fits in ordinary three-dimensional space. 00:03:21.869 --> 00:03:25.942 The first row records response one, the second response two, 00:03:26.025 --> 00:03:28.023 and the third response four. 00:03:28.475 --> 00:03:32.447 Read down the response column, not across a plotted axis. 00:03:32.988 --> 00:03:38.074 Stack those three responses into one vector y: one, two, four. 00:03:38.270 --> 00:03:39.547 Each coordinate belongs 00:03:39.547 --> 00:03:44.866 to one observation, so this single arrow represents the entire response data set. 00:03:45.662 --> 00:03:50.609 The design matrix also has one coordinate per observation in each column. 00:03:50.713 --> 00:03:53.985 Its intercept column is one, one, one, 00:03:54.068 --> 00:03:57.981 because the intercept contributes equally to every fitted response. 00:03:58.893 --> 00:04:02.702 The predictor column is minus one, zero, one. 00:04:02.806 --> 00:04:06.267 It records the three predictor values after centring. 00:04:06.719 --> 00:04:09.355 This green arrow is that whole column. 00:04:10.151 --> 00:04:13.450 A coefficient beta zero scales the blue column. 00:04:13.646 --> 00:04:16.977 A coefficient beta one scales the green column. 00:04:17.060 --> 00:04:18.174 Adding the two scaled 00:04:18.174 --> 00:04:23.225 arrows produces every fitted response vector this model is capable of making. 00:04:24.022 --> 00:04:27.540 Those linear combinations fill a plane through the origin. 00:04:27.737 --> 00:04:30.326 This is the column space of X. 00:04:30.430 --> 00:04:34.308 Changing the regression coefficients moves the fitted vector somewhere inside 00:04:34.308 --> 00:04:36.956 this plane, but never outside it. 00:04:37.752 --> 00:04:39.831 The picture is three-dimensional only 00:04:39.831 --> 00:04:42.757 because this example has three observations. 00:04:43.070 --> 00:04:45.102 With n observations, 00:04:45.206 --> 00:04:48.446 y lives in n-dimensional observation space. 00:04:48.898 --> 00:04:51.276 With p predictors and an intercept, 00:04:51.359 --> 00:04:55.679 the columns span a subspace of dimension at most p plus one. 00:04:56.476 --> 00:05:00.354 The yellow response vector usually misses the predictor plane. 00:05:00.551 --> 00:05:04.032 An exact fit would put y inside the column space, 00:05:04.115 --> 00:05:08.319 but these three observations do not lie on one exact regression line. 00:05:09.116 --> 00:05:13.145 Among all points in the plane, choose the one closest to y. 00:05:13.249 --> 00:05:15.595 The green arrow reaches that point. 00:05:15.791 --> 00:05:18.810 It is the fitted-value vector, hat y. 00:05:19.607 --> 00:05:22.847 Its three coordinates are the three fitted responses. 00:05:23.044 --> 00:05:26.467 Returning to the original data plot would place those fitted values 00:05:26.550 --> 00:05:30.405 on the regression line at the three observed predictor values. 00:05:31.202 --> 00:05:34.825 What remains is the red vector from hat y to y. 00:05:35.021 --> 00:05:39.559 Coordinate by coordinate, it contains observed response minus fitted response, 00:05:39.642 --> 00:05:42.836 so it is exactly the residual vector e. 00:05:43.539 --> 00:05:48.172 The closest route from a point to a plane is perpendicular to the plane. 00:05:48.276 --> 00:05:50.075 Therefore the residual stands 00:05:50.075 --> 00:05:53.698 at a right angle to every direction the predictor columns span. 00:05:55.095 --> 00:05:57.929 This is least squares in one sentence. 00:05:58.241 --> 00:06:00.865 The fitted vector is the orthogonal projection 00:06:00.865 --> 00:06:04.151 of y onto the column space of X. 00:06:04.948 --> 00:06:07.315 The word shadow is useful. 00:06:07.398 --> 00:06:11.033 Imagine light travelling perpendicular to the predictor plane. 00:06:11.229 --> 00:06:16.873 The response arrow casts its endpoint onto the plane, and the shadow point is hat y. 00:06:17.669 --> 00:06:21.118 The red leftover is perpendicular to the shadow plane. 00:06:21.431 --> 00:06:23.206 Pythagoras then says 00:06:23.289 --> 00:06:26.308 no other point in the plane can be closer to y. 00:06:26.621 --> 00:06:29.755 Moving away from hat y adds an in-plane displacement 00:06:29.755 --> 00:06:31.892 at right angles to the residual. 00:06:32.503 --> 00:06:35.592 That geometric fact is the minimization. 00:06:35.789 --> 00:06:38.843 We did not differentiate a sum of squares. 00:06:39.040 --> 00:06:43.731 We identified the unique shortest vector from y to the model subspace. 00:06:44.527 --> 00:06:48.754 The same picture survives with many observations and many predictors. 00:06:49.206 --> 00:06:51.285 Only the dimension changes. 00:06:51.482 --> 00:06:53.349 The response is one point, 00:06:53.432 --> 00:06:59.308 the possible fits form a column space, and least squares drops a perpendicular to it. 00:07:00.341 --> 00:07:03.616 Now use the right angle to calculate the coefficients. 00:07:03.812 --> 00:07:06.271 The fitted vector is X times beta hat, 00:07:06.355 --> 00:07:09.978 because it is a linear combination of the columns of X. 00:07:10.774 --> 00:07:14.862 The residual is observed y minus fitted y. 00:07:15.059 --> 00:07:21.143 Substituting the matrix form gives e equals y minus X beta hat. 00:07:21.940 --> 00:07:25.238 Perpendicular vectors have dot product zero. 00:07:25.434 --> 00:07:28.868 Since e is perpendicular to the entire column space, 00:07:28.952 --> 00:07:32.955 it is perpendicular to the intercept column, the first predictor column, 00:07:33.039 --> 00:07:35.210 and every other predictor column. 00:07:36.123 --> 00:07:43.055 For one column c sub j, perpendicularity reads c sub j transpose e equals zero. 00:07:43.252 --> 00:07:46.108 Write one such equation for every column. 00:07:46.812 --> 00:07:48.903 Stack all those dot products. 00:07:49.099 --> 00:07:53.383 The transpose X transpose places the predictor columns into the rows of one 00:07:53.383 --> 00:07:58.632 multiplication, so the whole collection becomes X transpose e equals zero. 00:07:59.545 --> 00:08:03.655 This compact zero is the algebraic form of the visible right angle. 00:08:03.852 --> 00:08:09.229 Every coordinate of X transpose e is one column's dot product with the residual. 00:08:10.025 --> 00:08:15.363 Carry that perpendicularity forward, and replace e by y minus X beta hat. 00:08:15.447 --> 00:08:20.324 It becomes X transpose times y minus X beta hat equals zero. 00:08:21.086 --> 00:08:23.234 Distribute X transpose. 00:08:23.396 --> 00:08:26.067 The first term is X transpose y. 00:08:26.229 --> 00:08:29.237 The second is X transpose X beta hat. 00:08:30.010 --> 00:08:32.414 Move the second term to the other side. 00:08:32.611 --> 00:08:37.859 We obtain X transpose X beta hat equals X transpose y. 00:08:38.056 --> 00:08:40.402 These are the normal equations. 00:08:41.198 --> 00:08:45.642 They are called normal because the residual is normal, meaning perpendicular, 00:08:45.726 --> 00:08:47.666 to the predictor subspace. 00:08:47.862 --> 00:08:51.010 The name is geometric before it is algebraic. 00:08:51.435 --> 00:08:54.605 The intercept equation has a familiar consequence. 00:08:54.709 --> 00:08:57.214 Its column is the all-ones vector, 00:08:57.298 --> 00:09:00.874 so its dot product with e is the sum of the residuals. 00:09:01.071 --> 00:09:04.172 With an intercept, that sum must be zero. 00:09:04.969 --> 00:09:07.961 Each predictor equation says something similar: 00:09:08.045 --> 00:09:11.796 the residual has zero dot product with that predictor column. 00:09:11.900 --> 00:09:16.486 No linear pattern available to the model remains inside the residual. 00:09:17.190 --> 00:09:22.195 If the columns are linearly independent, X transpose X is invertible. 00:09:22.391 --> 00:09:26.374 Multiplying by its inverse gives the familiar coefficient formula. 00:09:27.171 --> 00:09:30.457 But the inverse formula is not the fundamental fact. 00:09:30.654 --> 00:09:34.915 The fundamental fact is X transpose e equals zero. 00:09:35.112 --> 00:09:37.965 Numerical methods usually solve the projection problem 00:09:38.049 --> 00:09:40.720 without explicitly forming an inverse. 00:09:41.424 --> 00:09:45.127 If the columns are dependent, the fitted projection can still be defined 00:09:45.127 --> 00:09:48.275 even though the coefficient vector need not be unique. 00:09:48.378 --> 00:09:52.280 That distinction will matter when we discuss multicollinearity. 00:09:53.076 --> 00:09:56.363 So the normal equations come straight from one right angle: 00:09:56.467 --> 00:10:01.131 fitted values lie in the column space, residuals leave it perpendicularly, 00:10:01.215 --> 00:10:04.919 and every predictor column has zero residual dot product. 00:10:06.019 --> 00:10:08.477 R squared uses the same projection picture, 00:10:08.561 --> 00:10:12.158 but first we must identify what counts as total variation. 00:10:12.242 --> 00:10:13.853 With an intercept in the model, 00:10:13.937 --> 00:10:17.084 the baseline prediction is the sample mean of y. 00:10:17.695 --> 00:10:21.330 Subtract that mean from every observed response. 00:10:21.526 --> 00:10:26.844 The centred vector y sub c measures how far the data sit from the mean-only fit. 00:10:27.641 --> 00:10:30.440 Centre the fitted vector in the same way. 00:10:30.752 --> 00:10:33.954 Because the regression model includes the intercept direction, 00:10:34.038 --> 00:10:38.404 the centred fitted vector remains inside the predictor subspace. 00:10:39.108 --> 00:10:42.382 The centred response splits into two pieces. 00:10:42.695 --> 00:10:46.388 The green piece is the variation explained by the predictors. 00:10:46.584 --> 00:10:50.486 The red piece is the residual variation left unexplained. 00:10:51.538 --> 00:10:54.453 Projection makes those pieces perpendicular. 00:10:54.650 --> 00:10:57.735 The three vectors therefore form a right triangle: 00:10:57.819 --> 00:11:03.103 explained component, residual component, and total centred response. 00:11:03.899 --> 00:11:06.826 Pythagoras applies to their squared lengths. 00:11:06.929 --> 00:11:12.712 Total sum of squares equals regression sum of squares plus residual sum of squares. 00:11:13.926 --> 00:11:19.047 R squared is the explained squared length divided by the total squared length. 00:11:19.499 --> 00:11:23.284 Equivalently, it is one minus the unexplained squared length divided 00:11:23.284 --> 00:11:24.550 by the total. 00:11:25.947 --> 00:11:30.394 In the three-observation example, the total squared length is fourteen thirds. 00:11:30.707 --> 00:11:33.258 The explained squared length is nine halves, 00:11:33.342 --> 00:11:36.176 and the residual squared length is one sixth. 00:11:37.088 --> 00:11:40.128 Nine halves plus one sixth is fourteen thirds, 00:11:40.212 --> 00:11:42.929 exactly as the right triangle requires. 00:11:43.126 --> 00:11:46.490 The resulting R squared is twenty-seven over twenty-eight, 00:11:46.574 --> 00:11:49.199 about zero point nine six four. 00:11:50.112 --> 00:11:52.643 There is also an angle interpretation. 00:11:52.840 --> 00:11:57.612 Let theta be the angle between the centred response and its fitted projection. 00:11:57.925 --> 00:12:01.246 Then R squared is cosine squared theta. 00:12:02.042 --> 00:12:06.115 A small angle means the response lies close to the model subspace, 00:12:06.199 --> 00:12:08.974 so the projection keeps nearly all its length. 00:12:09.171 --> 00:12:12.748 A large angle means a larger perpendicular residual. 00:12:13.544 --> 00:12:16.201 With ordinary least squares and an intercept, 00:12:16.285 --> 00:12:20.163 this training-sample ratio lies between zero and one. 00:12:20.267 --> 00:12:23.867 Zero means the predictors improve nothing beyond the mean. 00:12:24.063 --> 00:12:26.920 One means every residual is zero. 00:12:27.717 --> 00:12:29.889 The intercept condition matters. 00:12:30.085 --> 00:12:33.963 Without the mean direction in the model, this familiar centred Pythagorean 00:12:33.963 --> 00:12:39.606 decomposition need not hold, and the usual interpretation of R squared needs care. 00:12:40.403 --> 00:12:44.974 R squared measures alignment with the fitted subspace, not causation, 00:12:45.058 --> 00:12:48.077 and not whether the model will predict new data well. 00:12:48.692 --> 00:12:53.453 It is a geometric summary of in-sample explained squared length. 00:12:54.249 --> 00:12:57.892 The projection picture has now given us residual minimization, 00:12:57.976 --> 00:13:00.914 the normal equations, and R squared. 00:13:01.529 --> 00:13:04.420 One final geometric question remains: 00:13:04.617 --> 00:13:08.414 what if the predictor directions themselves almost collapse together? 00:13:09.499 --> 00:13:13.944 To see multicollinearity, first remove the intercept direction by centring, 00:13:14.027 --> 00:13:17.081 and scale two predictor columns to unit length. 00:13:17.278 --> 00:13:21.818 Then their relationship is summarized by the angle theta between their vectors. 00:13:22.615 --> 00:13:23.971 At sixty degrees, 00:13:24.054 --> 00:13:27.608 the blue and green predictors provide clearly different directions. 00:13:27.805 --> 00:13:29.405 Together they span the plane, 00:13:29.488 --> 00:13:34.365 and a fitted vector has coordinates beta one and beta two in this predictor basis. 00:13:35.162 --> 00:13:38.320 Draw the coefficient contributions tip to tail. 00:13:38.424 --> 00:13:41.534 The blue contribution first moves along x one. 00:13:41.617 --> 00:13:44.390 The green contribution then moves along x two, 00:13:44.473 --> 00:13:47.736 and their sum lands exactly at the red fitted vector. 00:13:48.533 --> 00:13:52.179 These coefficients are coordinates, not intrinsic lengths. 00:13:52.492 --> 00:13:53.478 Their values depend 00:13:53.478 --> 00:13:57.577 on how well the chosen predictor directions form a coordinate system. 00:13:58.490 --> 00:14:01.263 Now make the green predictor more like the blue one. 00:14:01.346 --> 00:14:02.321 At thirty degrees, 00:14:02.321 --> 00:14:05.965 the span is still the whole plane, and the fitted vector has not moved. 00:14:06.048 --> 00:14:10.554 But the two coefficient contributions must grow and cancel more strongly. 00:14:11.350 --> 00:14:13.778 Bring the angle down to ten degrees. 00:14:13.881 --> 00:14:16.866 The predictor arrows are now nearly parallel. 00:14:17.062 --> 00:14:20.857 To reach the same modest red vector, one long component runs left 00:14:20.940 --> 00:14:24.528 and the other almost cancels it while climbing to the target. 00:14:25.325 --> 00:14:27.436 This is multicollinearity. 00:14:27.519 --> 00:14:31.975 Two columns of X supply nearly the same direction in observation space, 00:14:32.058 --> 00:14:36.541 so the model has difficulty deciding how much of the fit to assign to each one. 00:14:37.244 --> 00:14:40.635 The fitted vector itself can still be perfectly clear. 00:14:40.832 --> 00:14:43.142 What becomes unstable is its description 00:14:43.142 --> 00:14:45.837 in this nearly collapsed coordinate system. 00:14:47.233 --> 00:14:49.405 Here is a small perturbation. 00:14:49.602 --> 00:14:51.169 Raise the fitted target's second 00:14:51.169 --> 00:14:56.604 coordinate from one point zero zero to one point zero two, only two hundredths. 00:14:57.400 --> 00:14:58.991 The red endpoint barely moves, 00:14:58.991 --> 00:15:02.832 but each coefficient changes by about zero point one one. 00:15:02.915 --> 00:15:06.547 The coordinate change is almost six times the size of the data perturbation, 00:15:06.630 --> 00:15:08.407 with opposite signs. 00:15:09.711 --> 00:15:12.556 The algebra reports the same geometry. 00:15:12.752 --> 00:15:14.504 For standardized columns, 00:15:14.587 --> 00:15:20.230 X transpose X has ones on the diagonal and cosine theta off the diagonal. 00:15:21.027 --> 00:15:24.044 Its determinant is one minus cosine squared theta, 00:15:24.127 --> 00:15:26.194 which is sine squared theta. 00:15:26.506 --> 00:15:31.058 As the predictor angle shrinks, this determinant approaches zero. 00:15:31.855 --> 00:15:35.313 At exactly zero degrees the two columns coincide. 00:15:35.396 --> 00:15:39.400 The span collapses from a plane to a line, X loses rank, 00:15:39.483 --> 00:15:44.035 and infinitely many coefficient pairs can describe the same fitted vector. 00:15:44.831 --> 00:15:48.487 Near zero, the inverse still exists in exact arithmetic, 00:15:48.570 --> 00:15:51.891 but it must magnify the thin direction enormously. 00:15:52.087 --> 00:15:54.840 That sensitivity is called ill conditioning. 00:15:55.543 --> 00:15:59.537 For two standardized predictors, this condition number is the square root 00:15:59.537 --> 00:16:03.973 of one plus cosine theta over one minus cosine theta. 00:16:04.170 --> 00:16:07.409 At ten degrees, the Gram determinant is about zero point 00:16:07.409 --> 00:16:11.833 zero three and the condition number is already about eleven point four. 00:16:12.630 --> 00:16:17.112 A condition number near one means the predictor directions are well separated. 00:16:17.727 --> 00:16:21.221 A large value means small changes in y, rounding, 00:16:21.221 --> 00:16:26.249 or sampling can create much larger changes in the estimated coefficients. 00:16:27.046 --> 00:16:30.820 This does not automatically make the fitted values unstable. 00:16:31.017 --> 00:16:32.073 Many different large, 00:16:32.073 --> 00:16:36.579 cancelling coefficient pairs can land at nearly the same projection point. 00:16:36.891 --> 00:16:39.910 Prediction within the observed predictor pattern may therefore 00:16:39.910 --> 00:16:44.264 look steady while individual coefficient interpretations do not. 00:16:45.661 --> 00:16:47.914 The first lesson is geometric: 00:16:48.110 --> 00:16:51.559 nearly parallel columns provide almost the same direction. 00:16:52.043 --> 00:16:53.933 The second is about coordinates: 00:16:54.016 --> 00:16:58.707 reaching one moderate fit may require large contributions with opposite signs. 00:16:59.248 --> 00:17:03.776 The third is statistical: a small change in the sample can move those large 00:17:03.776 --> 00:17:07.539 coefficients substantially, increasing their uncertainty. 00:17:08.335 --> 00:17:10.821 And the fourth separates two questions. 00:17:10.924 --> 00:17:14.848 The model may predict a combined effect reasonably while the separate effects 00:17:14.848 --> 00:17:17.961 of the correlated predictors remain poorly determined. 00:17:19.013 --> 00:17:21.426 Possible responses include collecting data 00:17:21.509 --> 00:17:23.759 where the predictors vary more independently, 00:17:23.842 --> 00:17:29.288 combining redundant predictors, choosing one of them, or using a regularized fit. 00:17:29.740 --> 00:17:32.873 Each response changes how the thin direction is handled, 00:17:32.956 --> 00:17:36.069 so it should follow the scientific purpose of the model. 00:17:36.981 --> 00:17:39.978 The whole lecture is now one geometric story. 00:17:40.174 --> 00:17:43.170 Data points create vertical residuals. 00:17:43.274 --> 00:17:47.802 Least squares chooses the shortest residual vector in observation space. 00:17:47.999 --> 00:17:50.949 Perpendicularity gives the normal equations. 00:17:51.745 --> 00:17:55.206 Pythagoras turns the same projection into R squared. 00:17:55.402 --> 00:17:59.188 And when predictor directions nearly align, the spanning coordinates 00:17:59.188 --> 00:18:03.252 become ill conditioned even if the projected fit barely changes.