Generalization

At the end of the last waypoint, the model could walk downhill and make its lossOne number for how wrong a guess is. Big is bad, small is good. smaller and smaller. That sounds like success. But there's a problem hiding in it.

The loss only measures how well the model does on the examples it trained on. And those aren't the houses we care about. We already know what they sold for. What we want is a good price for the next house, the one nobody has seen yet.

A low loss on the training examples feels like proof that the model learned. Is it?

Can a perfect score be a bad sign?

Think of a student preparing for an exam with last year's answer key. One student works through the problems until they understand the subject. Another just memorizes the answers. On last year's questions, both score 100%. On this year's exam, only one of them passes.

A model can do the same thing. Let's give our model a few more houses to learn from:

Now train two models on them. The first is our straight line. It fits the general trend, about $196 per square foot, but it doesn't pass exactly through every point. Each house is a few thousand dollars off.

The second model has more knobs, enough to bend into a curve that passes exactly through all six houses. Its loss is zero. Every prediction is perfect.

If we judged by loss alone, the curve would win easily. Which one would you trust with a new house?

What happens with a new house?

Here are two houses that sold but that neither model saw during training:

The line misses by $3,000 and $5,000, about as close as it got on its own training houses. The curve, the "perfect" model, misses by $18,000 and $22,000.

Here's what happened. The 1,700 sq ft house sold high, maybe because it had a great view. That's not part of the size-to-price pattern, it's just something particular about that house. The line mostly ignored it. The curve bent itself to hit that point exactly, and in doing so, it pushed its predictions for nearby houses up too high.

The curve didn't learn the pattern. It learned the examples, including their quirks. This is called overfittingFitting the training examples so closely that their quirks get learned as the pattern.: fitting the training examples so closely that the model learns their noise as if it were the pattern.

Is the opposite a problem too?

Yes. Remember the lazy guess from the Loss waypoint, the one that called every house $300,000? That model is too simple to capture the pattern at all. It does badly on the training houses and badly on new ones. This is called underfittingA model too simple to capture the pattern. Bad on the examples it trained on, and bad on new ones..

So the goal is neither the simplest model nor the most flexible one. It's a model that captures the real pattern and nothing else. A model that does well on examples it has never seen is said to generalizeDoing well on examples the model has never seen.. That's the whole point of learning, and it's what we actually want.

How can we tell if a model generalizes?

The training loss can't tell us. The curve and the line both did well on their training houses, and the curve did better. The only way to find out how a model handles new examples is to give it new examples.

So before training, we set some examples aside and hide them from the model. The model learns from the rest, called the trainingsetThe examples the model learns from. The knobs are adjusted using only these.. Then we check it on the hidden ones, called the testsetHeld-back examples used once, at the end, for an honest score.. Those two houses in the table above were exactly that.

Comparing the two scores tells the story. If the model does about as well on the test set as on the training set, it learned something real. If it does great on training and much worse on test, it memorized. A big gap between the two is the signature of overfitting.

Why not just use the test set over and over?

Because it stops being a fair test.

In practice, you don't train just one model. You try a straight line and a curve. You try different learning rates. You try training for longer or shorter. Each time, you check which version scores best on the test set, and you keep that one.

But notice what's happening. Every time you use the test set to make a choice, you're tuning your decisions to those particular examples. Do it enough times and you'll find a version that happens to do well on them, whether or not it generalizes. It's like a student retaking the same practice exam until they pass. The final score looks great, but it no longer tells you much.

The fix is to hold back two sets of examples:

The training set is what the model learns from. GradientdescentThe walk downhill: one small step against the gradient, then another. adjusts the knobs using only these examples.

The validationsetHeld-back examples used to compare versions and choose among them. is what you use to compare versions and make choices, like which model shape or learning rate works best.

The test set is locked away until the very end. You use it once, for a final, honest score of the model you chose.

A common split is something like 80% for training, 10% for validation, and 10% for test, though the exact numbers depend on how much data you have.

What counts as "unseen"?

This is trickier than it sounds. For the test to be honest, the test examples must be truly new to the model, and they must look like the examples it will face in real use.

Two common mistakes break that. The first is letting the same example slip into both sets. If a house appears twice in your data, once in training and once in test, the model has already seen the answer, and its test score is inflated. The second is ignoring time. If you want to predict next year's prices, testing on houses that sold before your training houses doesn't tell you much. The honest test is the one that looks like the future: train on older sales and test on newer ones.

The general rule is simple. Test the model the way it will be used. We'll come back to what happens when the real world stops looking like the test set at the Rare cases and Drift waypoint.

So, did it learn, or just memorize?

You can't tell from the training loss. A model that memorizes can score perfectly on the examples it has seen and still fail on the next one. The only honest answer comes from examples the model has never seen: a validation set to guide your choices and a test set for the final check. A model that holds up on those generalizes, and generalizing is what learning really means.

Now we can detect memorizing. But that's only half the problem. The curve overfit because nothing stopped it from bending to hit every point. Is there a way to stop a model from memorizing in the first place? That's the next waypoint: Regularization.