Overfitting Made Visible: Bias, Variance, and the Role of Data

Overfitting is often used as a warning word: this model overfits, that model generalizes. I want to make the warning visible and measurable. We will give several polynomial models the same twenty noisy observations, then watch each model decide what story those observations tell. Here are the twenty observations. Each yellow dot contains two things: a curved underlying relationship and a small disturbance that pushed the measurement above or below that relationship. Suppose we know only the dots. Our first candidate is a polynomial of degree one, which is simply a straight line. It can tilt and it can move up or down, but it cannot bend. The line catches the broad upward trend, but look at the ends and the middle. The dots curve away from it in an organized pattern. Those are not isolated misses. The model lacks the shape needed to follow the relationship. Allow degree three. The line gives way to a curve, and the fit can now follow the large bend. It still refuses to chase every individual dot. That refusal is useful. A noisy observation is not the relationship itself. Some of the vertical displacement belongs to chance, so a model that leaves small residuals may be behaving more sensibly than one that erases them. Now allow degree seven. The curve gains more freedom. It bends toward several individual observations and begins to reproduce their alternating rises and falls. Finally allow degree nineteen. Twenty coefficients are enough to force a polynomial through twenty distinct points. The red curve touches every yellow dot exactly. On the training observations this looks perfect. Between the observations, however, the curve changes direction again and again. It has treated each upward or downward disturbance as if it were a real feature of the world. Because this is a controlled example, we may reveal the relationship that generated the observations. The blue curve is the signal. The red curve agrees with all twenty recorded values, yet it disagrees repeatedly with the relationship between them. So failure happens in two directions. The straight line is too rigid to represent the signal. The degree-nineteen polynomial is flexible enough to represent both the signal and the accidental noise. One underfits; the other overfits. The central question is therefore not whether a model can fit these observations. It is whether the fitted pattern continues to work at places the fitting process did not get to see.

Start with the rigid failure. The blue curve is the underlying relationship, the yellow dots are our original twenty observations, and the green line is the best straight model available to us. Now add many more observations from the same relationship. Eighty gray points make the bend much easier to see, but they do not give a straight line the ability to bend. More data can reveal that this model is wrong, but it cannot repair the model family. These red gaps are systematic. On one stretch the line is too high; on another it is too low. This persistent miss is underfitting. Now look at the flexible failure. The right-hand graph uses the original twenty observations again. The red degree-nineteen polynomial passes through every one of them. If we measure only at those yellow locations, every vertical residual is zero. Its mean squared training error is therefore zero. But now place green observations at new locations between the training points. They follow the same blue relationship and were not available while the polynomial was fitted. The magenta gaps measure predictions at those unseen locations. Several are large because the red curve swings away between neighboring training points. Perfect recall of the training set did not produce reliable prediction. That gives us an operational definition. Overfitting is not simply a complicated-looking curve. It is a measurable gap: very small error on the data used for fitting, paired with substantially larger error on new data. The straight line and the degree-nineteen curve fail for opposite reasons. The line cannot express enough structure. The interpolating curve reacts to structure that was never really present. To compare models with one number, use mean squared error. For each observation, subtract prediction from observation, square that residual, add the squares, and divide by the number of observations. Training mean squared error uses the observations that selected the fitted curve. Test mean squared error repeats the same calculation on observations held aside from that selection. The formulas look nearly identical. Their difference is experimental: one asks how well the model remembers its lesson, and the other asks how well it handles a new question.

Now repeat the fitting experiment for degree one, degree two, degree three, and so on up to degree nineteen. For every fitted model, record both errors. The blue curve is training error. It falls as degree increases because each larger polynomial family contains more ways to accommodate the observations. That decrease is built into the fitting process. A more flexible model can imitate a simpler one if that is best, or use its extra freedom to reduce the residuals further. The red curve is test error. At first it also falls. Moving beyond a straight line lets the model capture the real curvature, so predictions improve. Near degree five, test error reaches its minimum. In this experiment that is the useful region: enough flexibility to represent the signal, without enough freedom to chase every disturbance. Past that point, training error keeps falling but test error turns upward. The two measurements separate because extra complexity is helping on observations already seen while hurting predictions elsewhere. At degree nineteen the training point is almost on the horizontal axis. The test point is high. Their vertical separation is the observable signature of the interpolation failure we just watched. Low complexity and high complexity therefore have different diagnoses. On the left, both errors are large because the model misses the signal. On the right, only training error is tiny, because the model has learned details that do not transfer. Selecting the degree with the smallest training error would always reward the most flexible candidate. That is exactly the decision rule that invites overfitting. Instead, reserve data for validation or testing and look for low error there. The marked point is not a law that degree five is always right. It is the answer for this relationship, this noise level, and this amount of data. Change the noise, the sampling locations, or the amount of data, and the red curve can change. That dependence is the doorway to bias and variance.

Imagine repeating the entire study. Draw twenty fresh noisy observations from the same relationship, fit the model, discard that sample, and do it again. On the left, each gray line is a straight model fitted to a different fresh sample. Here is the first fit, then the second, the third, the fourth, and the fifth. The fitted lines move a little because the observations change, but they remain a tight family. Their small spread means low variance. Now average the five fitted predictions at every x. The red line is that average. It still misses the blue curve in the same organized way, especially near the ends. That persistent difference is bias. Even if we repeated the study many more times, averaging straight lines would still produce a straight line. Repetition cannot create curvature that the model family forbids. Now repeat the same experiment with a highly flexible polynomial. The blue curve on the right is still the same relationship. Each colored curve is a refit on a new sample of twenty observations. One sample pulls the curve upward here, another pulls it downward, and another creates a different sequence of bends. The model family is capable of following the underlying curve, so its average can be close to the truth. But the individual fits spread widely. Which data set happened to arrive has a large effect on the prediction. That sensitivity is variance. High variance means that if we repeated the data collection, our fitted answer could change substantially even though the data-generating process did not. Bias and variance are therefore not labels attached permanently to one curve. They describe behavior across repeated samples. Bias asks where the average fit lands. Variance asks how widely the individual fits scatter around that average. Bias is the difference between the average fitted prediction and the true relationship. A rigid straight model has substantial bias here because its average prediction cannot reproduce the bend. Variance is the spread of fitted predictions at the same input across fresh samples. A flexible model can have low bias and still have high variance if each sample leads it toward a different detailed curve. Underfitting is the visible face of excessive bias. Overfitting is the visible face of excessive sensitivity to the sample. The useful model must control both.

Return to repeated fitting, but change the amount of evidence. On the left, each colored flexible curve was fitted from only twenty observations. The fits disagree because a handful of noisy points can pull a flexible polynomial in noticeably different directions. The blue relationship is fixed; the fitted curves are not. On the right, fit the same degree of polynomial after collecting one hundred observations. Each individual observation now has less power to redirect the entire curve. The colored curves still differ, because noise has not vanished. But their spread is much narrower. More data has reduced variance without changing the basic model family. Notice what more data did not do. It did not make every complex model safe, and it would not make a straight line bend. It made the fitted details less dependent on a few particular observations. We can summarize that change with two error contributions. Blue is error from bias. It falls as complexity gives the model more shapes to choose from. Red is error from variance. With only twenty observations, it rises as flexible models become increasingly sensitive to the sample. The two contributions cross near this yellow point. To the left, missing structure dominates. To the right, sample sensitivity dominates. The crossing is a useful picture of balance, not a universal decision rule. Now keep the blue bias curve and increase the data to one hundred observations. The right-hand graph shows the same declining bias contribution. The red variance contribution is lower because repeated fits are more stable. Its crossing with bias moves to the right, toward a more flexible polynomial. This does not say that complexity is good by itself. It says complexity becomes easier to support when the data constrain it from many directions. The final choice still belongs to validation or test performance. The point of the whole experiment is now visible. Underfitting is a systematic miss that survives more observations. Overfitting is a gap between fitting performance and new-data performance, driven by excessive sensitivity to the sample. First, training error asks about observations the model already used. Second, test error asks about genuinely separate observations. The difference between those questions is what makes overfitting measurable. Third, bias is the persistent miss seen across fresh samples. Fourth, variance is the spread of the refitted predictions, and adding data can narrow that spread. A useful model has enough freedom to represent the relationship and enough evidence to keep that freedom under control. The goal is not a curve that remembers every point. It is a curve whose accuracy survives the arrival of the next point.