Metrics

By now, our model learns a rule, generalizes to new examples, and says how sure it is. But is it any good?

So far, we've judged models by their lossOne number for how wrong a guess is. Big is bad, small is good. It's what the model learns from, not always what we care about.. The loss is perfect for training. It gives gradientdescentThe walk downhill: one small step against the gradient, then another. a smooth number to walk downhill on. But try telling someone that your spam filter reached a cross-entropyThe loss for answers that aren't numbers. It punishes confident mistakes the most. of 0.12. They'll ask the obvious question: so does it work? How much spam does it catch? Does it ever throw away real email?

The loss is the number the model learns from. What we need now is a number that matches what we actually care about. That number is called a metricA number that matches what we actually care about, as opposed to the loss the model learns from..

Why not just count how often it's right?

That's the obvious place to start. Run the model on the testsetHeld-back examples used once, at the end, for an honest score. and count the share of answers it got right. That's called accuracyThe share of answers the model got right., and it's easy to understand: "it's right 98% of the time."

Let's try it on a test set of 1,000 emails, 50 of which are spam. Our filter marks anything above 0.5 probability as spam:

It flags 60 emails as spam. 45 really are spam, and 15 are real emails.

It lets 940 emails through. 935 are real, and 5 are spam it missed.

It got 45 + 935 = 980 right, so its accuracy is 98%. Sounds great.

Can accuracy lie?

Now meet a lazy model. It ignores the email completely and says "not spam" every time. It catches nothing. But 950 of the 1,000 emails really aren't spam, so it's right 950 times. Its accuracy is 95%.

A model that does absolutely nothing scores 95%. Accuracy looks impressive here only because spam is rare, so being right about the common case is easy. When one answer is much more common than the other, accuracy can hide a model that's useless at the job we built it for. We'll come back to why rare cases cause trouble at the next waypoint.

What are the different ways to be wrong?

The real problem is that accuracy treats every mistake the same. But our filter can be wrong in two very different ways:

It can miss spam, letting junk into the inbox. Annoying, but you can delete it.

It can flag a real email, sending it to the spam folder. That might be a job offer you never see.

To see both at once, lay the results out in a small table:

This table is called a confusionmatrixA table of the ways the model was right and wrong: caught, missed, false alarms, and correct., because it shows exactly where the model gets confused. The two kinds of mistakes have names too. A false alarm, flagging something that isn't there, is a falsepositiveA false alarm. Flagging something that isn't there.. A miss, failing to flag something that is there, is a falsenegativeA miss. Failing to flag something that is there..

How do we measure each kind of mistake?

Two questions capture them.

When the model says "spam," how often is it right? It flagged 60 emails, and 45 were spam. That's 75%. This is called precisionWhen the model says "spam," how often it is right.. Low precision means lots of false alarms.

Of all the spam that arrived, how much did the model catch? There were 50 spam emails, and it caught 45. That's 90%. This is called recallOf all the spam that arrived, how much the model caught.. Low recall means lots of misses.

Now look at the lazy model again. It never flags anything, so its recall is 0%. It caught none of the spam. The number that accuracy hid is right there.

Can we have both?

Usually not completely, and this is where the thresholdThe cutoff that turns a probability into a decision, like marking spam above 0.9. from the last waypoint comes back. Our filter flags anything above 0.5. What if we raise the threshold to 0.9, so it only flags emails it's very sure about?

Raising the threshold makes the model more careful. It flags fewer real emails by mistake, so precision goes up. But it also lets more spam slip through, so recall goes down. Lowering the threshold does the opposite.

The model doesn't change at all. Only the cutoff moves. So the same model can be tuned to be cautious or aggressive, and someone has to decide which.

So which one matters more?

That depends on what each mistake costs, and that's a question about the job, not about the model.

For a spam filter, losing a real email is usually worse than seeing some junk, so precision matters more. For a test that screens for a serious disease, missing a sick patient is far worse than a false alarm that leads to a second test, so recall matters more. For fraud detection, a false alarm might mean a quick text to the customer, while a miss means lost money, so many banks lean toward recall.

This is why choosing a metric isn't just a technical step. It's a decision about which mistakes your organization can live with. If you want one number that balances precision and recall, there's a common one called the F1scoreOne number that balances precision and recall, assuming both kinds of mistakes matter equally.. Just remember that it assumes both kinds of mistakes matter equally, which is rarely true.

What about answers that are numbers?

For house prices, there's no "flagged" or "missed," but the same principle applies: pick a number people can understand and act on.

The loss we trained with, meansquarederrorSquare each miss, then take the average. Useful for training, awkward to explain., is measured in squared dollars, which means nothing to a buyer. A clearer metric is the average miss in dollars, called meanabsoluteerrorThe average miss, in the same units as the answer. For houses, dollars.. On the two new houses from the Generalization waypoint, the straight line was off by about $4,000 on average. The memorizing curve was off by about $20,000. Anyone can understand that difference.

Sometimes the size of the miss matters relative to the price. Being $20,000 off on a $200,000 house is a big error. On a $2 million house, it's a rounding error. In that case, measuring the miss as a percentage tells you more.

Is one number ever enough?

Rarely. A single average can hide a lot. A house-price model could be off by $4,000 on average and still be badly wrong in one neighborhood, every time. A spam filter could work well on English email and poorly on email in Spanish. So it's worth checking metrics separately for the groups you care about, not just overall.

And accuracy is only part of whether a model works in practice. Is it fast enough? Affordable to run? Are its probabilities calibratedWhen the model says 90%, it is right about 90% of the time.? Does it treat different groups fairly? We'll return to these questions later on the path, when models become products.

One last rule: measure on the test set, the examples the model has never seen. A metric measured on trainingdataThe examples the model learned from. A metric on these only shows how well it memorized. just tells you how well the model memorized.

So, how do we measure if it works?

Pick a number that matches what matters. Accuracy is a good start, but it treats every mistake the same and can look great while a model fails at its real job. When some mistakes cost more than others, precision and recall show what's really going on, and the threshold lets you trade one for the other. Choosing which mistakes matter most isn't a technical detail. It's a decision about the job the model is doing.

But two warning signs have come up on the way here. The lazy model scored 95% because spam was rare, and the model it ignored barely saw any spam to learn from. And back at the probability waypoint, the mansion got a confident price, because the model had never seen a house like it. Both come down to the same weakness: what happens when the data a model faces doesn't look like the data it learned from? That's the next waypoint: Rare cases and Drift.