WEBVTT 00:00:00.035 --> 00:00:04.480 Overfitting is often used as a warning word: this model overfits, 00:00:04.563 --> 00:00:06.572 that model generalizes. 00:00:06.769 --> 00:00:09.774 I want to make the warning visible and measurable. 00:00:09.857 --> 00:00:14.348 We will give several polynomial models the same twenty noisy observations, 00:00:14.431 --> 00:00:18.577 then watch each model decide what story those observations tell. 00:00:19.974 --> 00:00:22.180 Here are the twenty observations. 00:00:22.377 --> 00:00:27.985 Each yellow dot contains two things: a curved underlying relationship and a small 00:00:27.985 --> 00:00:32.234 disturbance that pushed the measurement above or below that relationship. 00:00:33.031 --> 00:00:35.354 Suppose we know only the dots. 00:00:35.457 --> 00:00:38.393 Our first candidate is a polynomial of degree one, 00:00:38.476 --> 00:00:40.508 which is simply a straight line. 00:00:40.612 --> 00:00:44.642 It can tilt and it can move up or down, but it cannot bend. 00:00:45.438 --> 00:00:50.176 The line catches the broad upward trend, but look at the ends and the middle. 00:00:50.280 --> 00:00:53.473 The dots curve away from it in an organized pattern. 00:00:53.925 --> 00:00:56.561 Those are not isolated misses. 00:00:56.758 --> 00:01:00.102 The model lacks the shape needed to follow the relationship. 00:01:01.317 --> 00:01:03.094 Allow degree three. 00:01:03.198 --> 00:01:07.947 The line gives way to a curve, and the fit can now follow the large bend. 00:01:08.143 --> 00:01:11.418 It still refuses to chase every individual dot. 00:01:12.215 --> 00:01:14.166 That refusal is useful. 00:01:14.363 --> 00:01:17.800 A noisy observation is not the relationship itself. 00:01:18.113 --> 00:01:22.049 Some of the vertical displacement belongs to chance, so a model that leaves 00:01:22.049 --> 00:01:26.368 small residuals may be behaving more sensibly than one that erases them. 00:01:27.165 --> 00:01:29.186 Now allow degree seven. 00:01:29.290 --> 00:01:31.450 The curve gains more freedom. 00:01:31.553 --> 00:01:35.640 It bends toward several individual observations and begins to reproduce 00:01:35.640 --> 00:01:38.009 their alternating rises and falls. 00:01:38.806 --> 00:01:41.326 Finally allow degree nineteen. 00:01:41.638 --> 00:01:46.248 Twenty coefficients are enough to force a polynomial through twenty distinct points. 00:01:46.561 --> 00:01:49.743 The red curve touches every yellow dot exactly. 00:01:50.539 --> 00:01:53.477 On the training observations this looks perfect. 00:01:53.674 --> 00:01:57.980 Between the observations, however, the curve changes direction again and again. 00:01:58.063 --> 00:01:59.514 It has treated each upward 00:01:59.514 --> 00:02:03.601 or downward disturbance as if it were a real feature of the world. 00:02:04.653 --> 00:02:06.614 Because this is a controlled example, 00:02:06.697 --> 00:02:10.471 we may reveal the relationship that generated the observations. 00:02:10.667 --> 00:02:12.886 The blue curve is the signal. 00:02:13.082 --> 00:02:16.111 The red curve agrees with all twenty recorded values, 00:02:16.194 --> 00:02:19.597 yet it disagrees repeatedly with the relationship between them. 00:02:20.300 --> 00:02:22.925 So failure happens in two directions. 00:02:23.121 --> 00:02:26.245 The straight line is too rigid to represent the signal. 00:02:26.442 --> 00:02:28.021 The degree-nineteen polynomial 00:02:28.021 --> 00:02:32.538 is flexible enough to represent both the signal and the accidental noise. 00:02:32.735 --> 00:02:35.533 One underfits; the other overfits. 00:02:37.185 --> 00:02:39.368 The central question is therefore not whether 00:02:39.368 --> 00:02:41.899 a model can fit these observations. 00:02:42.351 --> 00:02:44.255 It is whether the fitted pattern continues 00:02:44.255 --> 00:02:47.611 to work at places the fitting process did not get to see. 00:02:48.641 --> 00:02:50.709 Start with the rigid failure. 00:02:51.161 --> 00:02:51.881 The blue curve 00:02:51.881 --> 00:02:56.929 is the underlying relationship, the yellow dots are our original twenty observations, 00:02:57.012 --> 00:03:00.484 and the green line is the best straight model available to us. 00:03:01.397 --> 00:03:04.846 Now add many more observations from the same relationship. 00:03:05.043 --> 00:03:07.955 Eighty gray points make the bend much easier to see, 00:03:08.038 --> 00:03:11.034 but they do not give a straight line the ability to bend. 00:03:11.947 --> 00:03:14.337 More data can reveal that this model is wrong, 00:03:14.420 --> 00:03:16.694 but it cannot repair the model family. 00:03:16.777 --> 00:03:19.100 These red gaps are systematic. 00:03:19.296 --> 00:03:23.778 On one stretch the line is too high; on another it is too low. 00:03:24.091 --> 00:03:26.588 This persistent miss is underfitting. 00:03:27.385 --> 00:03:29.707 Now look at the flexible failure. 00:03:30.020 --> 00:03:33.887 The right-hand graph uses the original twenty observations again. 00:03:34.502 --> 00:03:38.996 The red degree-nineteen polynomial passes through every one of them. 00:03:39.792 --> 00:03:42.206 If we measure only at those yellow locations, 00:03:42.289 --> 00:03:44.588 every vertical residual is zero. 00:03:44.692 --> 00:03:48.083 Its mean squared training error is therefore zero. 00:03:49.135 --> 00:03:53.628 But now place green observations at new locations between the training points. 00:03:53.825 --> 00:03:56.634 They follow the same blue relationship and were not available 00:03:56.634 --> 00:03:58.713 while the polynomial was fitted. 00:03:59.510 --> 00:04:03.679 The magenta gaps measure predictions at those unseen locations. 00:04:03.875 --> 00:04:04.804 Several are large 00:04:04.804 --> 00:04:08.741 because the red curve swings away between neighboring training points. 00:04:08.938 --> 00:04:13.048 Perfect recall of the training set did not produce reliable prediction. 00:04:13.845 --> 00:04:16.597 That gives us an operational definition. 00:04:16.794 --> 00:04:20.232 Overfitting is not simply a complicated-looking curve. 00:04:20.428 --> 00:04:22.414 It is a measurable gap: 00:04:22.518 --> 00:04:25.245 very small error on the data used for fitting, 00:04:25.328 --> 00:04:28.695 paired with substantially larger error on new data. 00:04:29.492 --> 00:04:33.948 The straight line and the degree-nineteen curve fail for opposite reasons. 00:04:34.031 --> 00:04:36.619 The line cannot express enough structure. 00:04:36.702 --> 00:04:40.581 The interpolating curve reacts to structure that was never really present. 00:04:41.977 --> 00:04:45.995 To compare models with one number, use mean squared error. 00:04:46.192 --> 00:04:49.986 For each observation, subtract prediction from observation, 00:04:50.069 --> 00:04:55.492 square that residual, add the squares, and divide by the number of observations. 00:04:56.544 --> 00:04:59.318 Training mean squared error uses the observations 00:04:59.318 --> 00:05:01.339 that selected the fitted curve. 00:05:01.652 --> 00:05:06.482 Test mean squared error repeats the same calculation on observations held aside 00:05:06.482 --> 00:05:08.061 from that selection. 00:05:08.858 --> 00:05:11.329 The formulas look nearly identical. 00:05:11.412 --> 00:05:13.468 Their difference is experimental: 00:05:13.664 --> 00:05:16.287 one asks how well the model remembers its lesson, 00:05:16.370 --> 00:05:19.435 and the other asks how well it handles a new question. 00:05:20.543 --> 00:05:24.848 Now repeat the fitting experiment for degree one, degree two, degree three, 00:05:24.931 --> 00:05:27.312 and so on up to degree nineteen. 00:05:27.509 --> 00:05:30.540 For every fitted model, record both errors. 00:05:31.336 --> 00:05:33.366 The blue curve is training error. 00:05:33.449 --> 00:05:37.815 It falls as degree increases because each larger polynomial family contains 00:05:37.815 --> 00:05:40.463 more ways to accommodate the observations. 00:05:41.166 --> 00:05:44.046 That decrease is built into the fitting process. 00:05:44.243 --> 00:05:47.817 A more flexible model can imitate a simpler one if that is best, 00:05:47.900 --> 00:05:51.349 or use its extra freedom to reduce the residuals further. 00:05:52.401 --> 00:05:54.524 The red curve is test error. 00:05:54.607 --> 00:05:56.546 At first it also falls. 00:05:56.743 --> 00:06:00.363 Moving beyond a straight line lets the model capture the real curvature, 00:06:00.446 --> 00:06:02.258 so predictions improve. 00:06:03.310 --> 00:06:06.817 Near degree five, test error reaches its minimum. 00:06:07.014 --> 00:06:10.509 In this experiment that is the useful region: enough flexibility 00:06:10.509 --> 00:06:14.573 to represent the signal, without enough freedom to chase every disturbance. 00:06:15.625 --> 00:06:20.479 Past that point, training error keeps falling but test error turns upward. 00:06:20.931 --> 00:06:24.286 The two measurements separate because extra complexity is helping 00:06:24.286 --> 00:06:28.048 on observations already seen while hurting predictions elsewhere. 00:06:28.961 --> 00:06:33.292 At degree nineteen the training point is almost on the horizontal axis. 00:06:33.489 --> 00:06:37.854 The test point is high. Their vertical separation is the observable signature 00:06:37.854 --> 00:06:40.514 of the interpolation failure we just watched. 00:06:41.217 --> 00:06:45.421 Low complexity and high complexity therefore have different diagnoses. 00:06:45.618 --> 00:06:50.111 On the left, both errors are large because the model misses the signal. 00:06:50.308 --> 00:06:52.895 On the right, only training error is tiny, 00:06:52.978 --> 00:06:56.508 because the model has learned details that do not transfer. 00:06:57.305 --> 00:06:58.234 Selecting the degree 00:06:58.234 --> 00:07:02.832 with the smallest training error would always reward the most flexible candidate. 00:07:03.029 --> 00:07:06.606 That is exactly the decision rule that invites overfitting. 00:07:07.147 --> 00:07:12.442 Instead, reserve data for validation or testing and look for low error there. 00:07:12.638 --> 00:07:16.329 The marked point is not a law that degree five is always right. 00:07:16.412 --> 00:07:21.312 It is the answer for this relationship, this noise level, and this amount of data. 00:07:22.364 --> 00:07:23.453 Change the noise, 00:07:23.536 --> 00:07:28.541 the sampling locations, or the amount of data, and the red curve can change. 00:07:28.738 --> 00:07:32.140 That dependence is the doorway to bias and variance. 00:07:33.185 --> 00:07:35.740 Imagine repeating the entire study. 00:07:36.052 --> 00:07:40.740 Draw twenty fresh noisy observations from the same relationship, fit the model, 00:07:40.824 --> 00:07:43.681 discard that sample, and do it again. 00:07:44.385 --> 00:07:49.447 On the left, each gray line is a straight model fitted to a different fresh sample. 00:07:49.644 --> 00:07:54.765 Here is the first fit, then the second, the third, the fourth, and the fifth. 00:07:55.561 --> 00:07:58.902 The fitted lines move a little because the observations change, 00:07:58.986 --> 00:08:01.112 but they remain a tight family. 00:08:01.308 --> 00:08:04.049 Their small spread means low variance. 00:08:04.846 --> 00:08:08.283 Now average the five fitted predictions at every x. 00:08:08.596 --> 00:08:10.709 The red line is that average. 00:08:10.813 --> 00:08:15.829 It still misses the blue curve in the same organized way, especially near the ends. 00:08:16.626 --> 00:08:19.309 That persistent difference is bias. 00:08:19.412 --> 00:08:22.068 Even if we repeated the study many more times, 00:08:22.152 --> 00:08:25.253 averaging straight lines would still produce a straight line. 00:08:25.356 --> 00:08:29.664 Repetition cannot create curvature that the model family forbids. 00:08:30.461 --> 00:08:34.537 Now repeat the same experiment with a highly flexible polynomial. 00:08:34.641 --> 00:08:37.660 The blue curve on the right is still the same relationship. 00:08:38.364 --> 00:08:42.684 Each colored curve is a refit on a new sample of twenty observations. 00:08:42.880 --> 00:08:45.200 One sample pulls the curve upward here, 00:08:45.284 --> 00:08:49.697 another pulls it downward, and another creates a different sequence of bends. 00:08:50.493 --> 00:08:53.834 The model family is capable of following the underlying curve, 00:08:53.918 --> 00:08:56.369 so its average can be close to the truth. 00:08:56.565 --> 00:08:59.503 But the individual fits spread widely. 00:08:59.607 --> 00:09:03.915 Which data set happened to arrive has a large effect on the prediction. 00:09:04.712 --> 00:09:06.884 That sensitivity is variance. 00:09:07.080 --> 00:09:11.097 High variance means that if we repeated the data collection, our fitted answer 00:09:11.097 --> 00:09:15.603 could change substantially even though the data-generating process did not. 00:09:16.399 --> 00:09:21.055 Bias and variance are therefore not labels attached permanently to one curve. 00:09:21.252 --> 00:09:24.364 They describe behavior across repeated samples. 00:09:24.561 --> 00:09:27.510 Bias asks where the average fit lands. 00:09:27.823 --> 00:09:32.804 Variance asks how widely the individual fits scatter around that average. 00:09:34.201 --> 00:09:37.113 Bias is the difference between the average fitted prediction 00:09:37.197 --> 00:09:39.160 and the true relationship. 00:09:39.612 --> 00:09:43.907 A rigid straight model has substantial bias here because its average prediction 00:09:43.907 --> 00:09:46.102 cannot reproduce the bend. 00:09:47.154 --> 00:09:49.291 Variance is the spread of fitted predictions 00:09:49.291 --> 00:09:52.020 at the same input across fresh samples. 00:09:52.635 --> 00:09:56.315 A flexible model can have low bias and still have high variance 00:09:56.315 --> 00:09:59.613 if each sample leads it toward a different detailed curve. 00:10:00.665 --> 00:10:04.126 Underfitting is the visible face of excessive bias. 00:10:04.322 --> 00:10:08.514 Overfitting is the visible face of excessive sensitivity to the sample. 00:10:08.711 --> 00:10:11.138 The useful model must control both. 00:10:12.242 --> 00:10:16.016 Return to repeated fitting, but change the amount of evidence. 00:10:16.468 --> 00:10:21.659 On the left, each colored flexible curve was fitted from only twenty observations. 00:10:22.362 --> 00:10:26.495 The fits disagree because a handful of noisy points can pull a flexible 00:10:26.495 --> 00:10:29.805 polynomial in noticeably different directions. 00:10:30.002 --> 00:10:33.787 The blue relationship is fixed; the fitted curves are not. 00:10:34.584 --> 00:10:35.372 On the right, 00:10:35.455 --> 00:10:40.297 fit the same degree of polynomial after collecting one hundred observations. 00:10:40.493 --> 00:10:45.173 Each individual observation now has less power to redirect the entire curve. 00:10:45.877 --> 00:10:49.952 The colored curves still differ, because noise has not vanished. 00:10:50.149 --> 00:10:52.576 But their spread is much narrower. 00:10:52.773 --> 00:10:56.930 More data has reduced variance without changing the basic model family. 00:10:57.982 --> 00:11:00.490 Notice what more data did not do. 00:11:00.687 --> 00:11:03.379 It did not make every complex model safe, 00:11:03.462 --> 00:11:05.912 and it would not make a straight line bend. 00:11:06.109 --> 00:11:10.812 It made the fitted details less dependent on a few particular observations. 00:11:12.208 --> 00:11:15.889 We can summarize that change with two error contributions. 00:11:16.086 --> 00:11:18.513 Blue is error from bias. 00:11:18.710 --> 00:11:22.553 It falls as complexity gives the model more shapes to choose from. 00:11:23.466 --> 00:11:25.893 Red is error from variance. 00:11:26.206 --> 00:11:27.762 With only twenty observations, 00:11:27.762 --> 00:11:32.592 it rises as flexible models become increasingly sensitive to the sample. 00:11:33.296 --> 00:11:36.533 The two contributions cross near this yellow point. 00:11:36.616 --> 00:11:39.636 To the left, missing structure dominates. 00:11:39.832 --> 00:11:43.304 To the right, sample sensitivity dominates. 00:11:43.501 --> 00:11:48.366 The crossing is a useful picture of balance, not a universal decision rule. 00:11:49.279 --> 00:11:54.446 Now keep the blue bias curve and increase the data to one hundred observations. 00:11:54.643 --> 00:11:58.707 The right-hand graph shows the same declining bias contribution. 00:11:59.503 --> 00:12:04.566 The red variance contribution is lower because repeated fits are more stable. 00:12:04.763 --> 00:12:09.744 Its crossing with bias moves to the right, toward a more flexible polynomial. 00:12:10.541 --> 00:12:15.090 This does not say that complexity is good by itself. It says complexity becomes 00:12:15.173 --> 00:12:19.202 easier to support when the data constrain it from many directions. 00:12:19.515 --> 00:12:23.405 The final choice still belongs to validation or test performance. 00:12:24.201 --> 00:12:27.197 The point of the whole experiment is now visible. 00:12:27.301 --> 00:12:31.888 Underfitting is a systematic miss that survives more observations. 00:12:32.084 --> 00:12:36.205 Overfitting is a gap between fitting performance and new-data performance, 00:12:36.288 --> 00:12:39.319 driven by excessive sensitivity to the sample. 00:12:40.831 --> 00:12:45.615 First, training error asks about observations the model already used. 00:12:45.812 --> 00:12:50.387 Second, test error asks about genuinely separate observations. 00:12:50.584 --> 00:12:54.613 The difference between those questions is what makes overfitting measurable. 00:12:55.665 --> 00:13:00.391 Third, bias is the persistent miss seen across fresh samples. 00:13:00.588 --> 00:13:04.162 Fourth, variance is the spread of the refitted predictions, 00:13:04.245 --> 00:13:07.009 and adding data can narrow that spread. 00:13:07.805 --> 00:13:11.043 A useful model has enough freedom to represent the relationship 00:13:11.126 --> 00:13:14.413 and enough evidence to keep that freedom under control. 00:13:14.609 --> 00:13:17.905 The goal is not a curve that remembers every point. 00:13:17.988 --> 00:13:22.203 It is a curve whose accuracy survives the arrival of the next point.