Loss

At the end of the last waypoint, our model had knobs and a set of examples, and its job was to find the knob settings that fit those examples. But there are endless possible settings. Before the computer can find a good one, it needs something more basic: a way to tell good from bad.

We can see at a glance when a line fits the points badly. The computer can't. It needs the answer as a number.

What does "wrong" look like for one example?

Let's go back to our houses and try a guess. Set the first knob to $150 per square foot and the second to zero:

For a single house, "how wrong" is easy: it's the gap between the prediction and the real price. We call that gap the errorThe gap between a prediction and the right answer, for one example..

But we don't want three numbers. We want one number for the whole guess, so we can compare it with other guesses. So how do we combine them?

Why not just add the misses up?

That's the obvious move, and it has a hidden trap. Some misses are too low and some are too high, and when you add them, they cancel out.

Look at a second guess, $200 per square foot:

Two misses too low and one too high: they add up to $10,000 off. Now try a lazy guess that ignores size entirely and calls every house $300,000:

Add those up and the overs and unders cancel, leaving the same $10,000 off. A guess that learned nothing gets the same score as a guess that fits nicely. Our measure is broken.

So what if we ignore the direction?

The fix is to count every miss as a positive amount, whether it was too high or too low. Now the totals tell the truth:

• $150 per sq ft: 60 + 65 + 110 = $235,000 off
• Flat $300,000: 90 + 10 + 110 = $210,000 off
• $200 per sq ft: 10 + 10 + 10 = $30,000 off

The good guess finally wins, and by a lot.

Are all misses equally bad?

Not usually. Imagine two models. One misses every house by $10,000. The other gets two houses exactly right but misses the third by $30,000. Both total $30,000 off. Which one would you trust?

Most of the time, you'd prefer the first. Small misses are a normal part of predicting, but a big miss can be costly. Being $30,000 off on a house is a real problem for whoever is buying it.

So the most common approach is to square each miss before adding it up (measured in thousands to keep the numbers readable):

• Three misses of 10: 100 + 100 + 100 = 300
• One miss of 30: 0 + 0 + 900 = 900

Squaring makes big misses count much more than small ones, so the model learns to avoid them. It also takes care of the direction problem for free, because squaring a negative number gives a positive one.

Does the number of examples change the score?

It would, if we only added things up. A model checked against a thousand houses would rack up a bigger total than one checked against three, even if it were better. So we take the average instead. That gives us a fair score no matter how many examples we use.

Put it all together, square each miss and then average them, and you get the most widely used measure of wrongness for predicting numbers. It's called meansquarederrorSquare each miss, then take the average..

What do we call this number?

This single number that says how wrong the model is across all its examples is called the lossOne number for how wrong a guess is across all the examples.. The rule that calculates it is the lossfunctionThe rule that turns every miss into one number.. Big loss, bad guess. Small loss, good guess.

Now "learning" has a precise meaning: find the knob settings that make the loss as small as possible.

What about answers that aren't numbers?

House prices are numbers, so measuring the miss was natural. But what about spam? The answer is "spam" or "not spam." There's no gap to measure. The model is either right or wrong.

The trick is to have the model say how sure it is, for example "90% sure this is spam." Now there's something to measure again. The loss here follows one simple idea: being confidently wrong should cost the most. A model that's 99% sure an important email is spam and is wrong gets a heavy penalty. A model that's 55% sure and is wrong gets a small one. A model that's confident and right pays almost nothing.

This kind of loss is called cross-entropyA loss for answers that are not numbers. Being confidently wrong costs the most.. We'll come back to what "how sure" really means at the probability waypoint. For now, the idea is the same as before: one number that measures how wrong the model is, shaped to punish the mistakes we care about most.

So, how does it know it was wrong?

The loss tells it. For every guess, the computer checks the model's predictions against the right answers and boils all the misses down into a single number. The choice of loss function decides which mistakes count most, so choosing it is really a decision about what "wrong" means for your problem.

But knowing how wrong you are doesn't tell you how to get better. With two knobs, we could try settings by hand and keep the best. Real models have millions or billions of knobs. How does the model find its way to a small loss? That's the next waypoint: Optimization.