Regularization

In the last waypoint, we caught a model memorizing. The curve passed through all six training houses perfectly, then missed the new ones by $18,000 and $22,000. The straight line was less perfect on its training houses but much better on new ones.

Now we know how to detect memorizing: hold back examples and test on them. But detecting it isn't preventing it. Can we stop a model from memorizing in the first place?

Why did the curve memorize?

Because nothing told it not to. Remember what the model is trying to do: make the lossOne number for how wrong a guess is. Big is bad, small is good. as small as possible. Bending to hit every point, including the 1,700 sq ft house with the great view, made the loss smaller. As far as the loss was concerned, memorizing was the right move.

So the model didn't do anything wrong. It did exactly what we asked. The problem is in what we asked.

Why not just use a simpler model?

For our houses, that works. The straight line can't bend, so it can't memorize. But most real problems need flexibility. A straight line can't tell a dog from a cat, and the models that can have millions of knobs, plenty to memorize with.

There's also the question of how simple. Too simple, and we're back to the lazy $300,000 guess. Picking the exact right amount of flexibility in advance is very hard. What we'd like is a flexible model that only uses its flexibility when the data really calls for it.

Wouldn't more examples help?

Yes, and it's one of the best fixes there is. With six houses, one house with a great view pulls the curve noticeably. With six thousand houses, its quirk gets lost in the crowd, and the real pattern dominates.

But examples cost money and time to collect, and sometimes there just aren't more. So we need a way to keep a model from memorizing even when data is limited.

What does memorizing look like inside the model?

Let's look at the knobs. The straight line has two, and their values are modest: about 196 and about 7.

The curve that passed through every house has six knobs, and some of them are enormous. A few are in the tens of thousands, some positive and some negative, all pulling hard against each other. That tug-of-war is what makes the curve swing up and down to reach every point exactly.

This is typical. When a model memorizes, it usually needs extreme knob settings to twist itself around each example. A model that captures a smooth, general pattern can usually get by with modest ones.

What if big knobs had a price?

Here's the key idea. Right now, the loss only measures how wrong the predictions are. Let's add a second part: a charge for how big the knobs are.

new loss = how wrong the predictions are + a penalty for large knobs

Now the model has to balance two things. Big knobs might lower the prediction misses, but they raise the penalty. So the model only uses a large knob when it improves the predictions enough to be worth the cost. Bending to fit one unusual house isn't worth it. Following the main trend is.

Adding a cost for complexity like this is called regularizationA cost for complexity, so the model prefers the simplest rule that still fits.. It doesn't take any flexibility away from the model. It just makes flexibility cost something, so the model prefers the simplest rule that still fits.

There are two common ways to charge for knob size. One adds up the squares of the knobs, so large knobs get very expensive. This is the most widely used version, often called L2regularizationA penalty that adds up the squares of the knobs, so large knobs get very expensive. or weightdecayAnother name for L2 regularization: the knobs are pulled toward zero.. The other adds up the knob sizes directly, called L1regularizationA penalty that adds up the knob sizes, and tends to push unneeded knobs to zero., and it tends to push unneeded knobs all the way to zero, effectively switching them off.

Does it work?

Let's go back to the flexible curve, with all its knobs, and train it again with a penalty. Here's how it handles the two new houses at different penalty strengths:

With no penalty, it's the memorizing curve from the last waypoint. With the right penalty, the misses drop to about $6,000 each, nearly as good as the straight line, even though the model kept all its flexibility. It just stopped using it to chase the house with the view.

With too strong a penalty, the knobs get squeezed so hard that the curve flattens out. Every house gets nearly the same price. That's underfittingA model too simple to capture the pattern. Bad on the examples it trained on, and bad on new ones., the lazy guess all over again.

How strong should the penalty be?

That's a dial, and like the learningrateHow big each downhill step is., a person usually sets it. Too weak, and the model memorizes. Too strong, and it can't learn the pattern.

You've already seen how to choose it. Train a few versions with different penalty strengths, check each one on the validationsetHeld-back examples used to compare versions and choose among them., and keep the one that does best. This is exactly the kind of choice the validation set is for. The testsetHeld-back examples used once, at the end, for an honest score. stays locked away for the final score.

Are there other ways to stop memorizing?

The penalty is the classic method, but large models use a few more, all built on the same principle: make memorizing harder than learning the pattern.

Stop early. During training, watch the validation loss. At first it falls along with the training loss. At some point it starts to rise again, even as the training loss keeps falling. That's the moment the model starts memorizing, so you stop there. This is called earlystoppingStop training when the validation loss starts rising, even if the training loss is still falling..

Switch parts off at random. In a neural network, you can randomly switch off some of its small parts during each training step. No single part can rely on the others, so the knowledge has to be spread out instead of memorized in one spot. It's like a team where anyone might be absent on a given day, so everyone has to understand the work. This is called dropoutRandomly switch off parts of a neural network during training, so no single part can memorize on its own..

Vary the examples. For photos, you can flip, crop, or slightly recolor each image during training. A dog is still a dog when flipped, but the model can't memorize the exact pixels anymore. This is called dataaugmentationChange each example a little during training, so the model can't memorize the exact one it saw., and it's a cheap way to get some of the benefit of more examples.

So, how do we keep it from memorizing?

We make complexity cost something. A model left alone will do whatever lowers its loss, including memorizing. Regularization adds a price for complexity, so the model prefers the simplest rule that still fits, and that simpler rule tends to hold up on new examples. The validation set tells us how strong to make the penalty.

Now our model can learn a rule that generalizes. But look at what it gives us: a single number. "This house is worth $361,000." Is that a confident answer or a rough guess? A house just like the training houses and a house unlike any it has seen get answers that look exactly the same. And back in the Loss waypoint, we said the model can tell us how sure it is. What does that really mean? That's the next waypoint: Probability and uncertainty.