Rare cases and Drift

Two warning signs came up on the way here. In Metrics, a lazy spam filter scored 95% accuracy while catching no spam at all, because spam was rare. In Probability and uncertainty, the model confidently priced a 5,000 sq ft mansion, even though it had never seen a house anywhere near that size.

Both point to the same weakness. A model only knows what its examples showed it. When it meets something its examples barely covered, it can fail, and often it fails without any warning.

Why are rare cases hard to learn?

Think back to how a model learns. GradientdescentThe walk downhill: one small step against the gradient, then another. walks downhill on the lossOne number for how wrong a guess is. Big is bad, small is good. Here it is an average, so common examples pull harder., and the loss is an average over all the examples. Every example pulls on the knobs a little.

Now picture our 1,000 emails: 950 real, 50 spam. The real emails pull on the knobs nineteen times as hard as the spam does, simply because there are nineteen times more of them. The model can get the loss quite low by getting very good at real emails and fairly sloppy at spam. Nothing in the loss tells it that spam was the point.

When one kind of example is much rarer than another, the data is called imbalancedWhen one kind of example is much rarer than another, so the common kind dominates the loss.. And imbalance is everywhere. Fraud might be 1 in 1,000 transactions. A rare disease might show up in 1 in 10,000 scans. A machine failure might happen once a year.

Why do rare cases matter so much?

Because they're often the reason the model exists. Nobody builds a fraud detector to recognize normal purchases, or a medical model to recognize healthy scans. The rare case is the one that costs money, or health, or trust.

So we have an uncomfortable combination: the cases that matter most are the ones the model has the least reason to learn.

How do we help a model learn rare cases?

There are a few common fixes, and they're often used together.

Measure what matters. First, make sure you can see the problem. Accuracy hid the lazy model completely. RecallOf all the cases that mattered, how many the model caught. exposed it right away. If the rare case matters, track a metricA number that matches what we actually care about, as opposed to the loss the model learns from. that watches it, as we saw in Metrics.

Give rare examples more weight. We can change the loss so each spam email counts as much as nineteen real ones. Now missing spam costs the model as much as it should, and gradient descent pays attention to it.

Rebalance the examples. Instead of changing the loss, we can change what the model sees: show it the rare examples more often, or show it fewer of the common ones. Either way, the model trains on a more even mix.

Collect more rare examples. This is often the most effective fix and the most expensive. Sometimes it means waiting for more fraud to happen. Sometimes it means paying experts to find and label rare cases.

Move the thresholdThe cutoff that turns a probability into a decision. Lowering it flags more cases.. Even without retraining, lowering the threshold makes the model flag more cases, raising recall at the cost of more false alarms, just as we saw in Metrics.

What about cases it has never seen?

Rare cases at least appear in the examples. Some cases don't appear at all. The mansion is one of these. The model learned from houses between 1,000 and 2,200 sq ft, and it has no idea how prices behave beyond that range. The straight line just keeps going, and it gives an answer with the same confidence as always.

This is the kind of uncertainty we met in Probability and uncertainty, where the model doesn't know what it doesn't know. One useful defense is to check whether a new input looks like the trainingdataThe examples the model learned from. A snapshot of the world at the time. before trusting the answer. If a house is far bigger than anything the model has seen, the system can flag it for a person instead of quietly giving a price.

What happens when the world changes?

So far, we've talked about cases that were rare from the start. But a case can also become rare, or new, after the model is trained.

A model learns from a snapshot of the world, the examples available at the time. The world doesn't stay still. Say our house model was trained on sales from two years ago, and prices have risen since. It keeps predicting about $300,000 for a 1,500 sq ft house, while similar houses now sell for $330,000. Nothing about the model changed. The world moved, and the model stayed where it was.

When the data a model meets in use slowly stops looking like the data it learned from, that's called driftWhen the data a model meets in use slowly stops looking like the data it learned from.. It comes in two forms. Sometimes the inputs change: buyers start looking at bigger houses, or in neighborhoods the model never saw. Sometimes the relationship itself changes: the same house is now worth more, so the rule the model learned is simply out of date. Spam is a classic case of both, because spammers keep changing their tactics specifically to get past filters.

Drift is easy to underestimate because it's gradual. A model doesn't break on a particular day. It just gets a little worse, month after month, until someone notices.

Why didn't the testsetHeld-back examples from the same snapshot. An honest score for the world as it was, not as it will be. catch it?

Because the test set came from the same snapshot. In Generalization, we held back examples to get an honest score, and that score really was honest, for the world as it was when the data was collected. A test set can tell you how a model will do on data like its examples. It can't tell you what the world will look like next year.

This is why "test the model the way it will be used" matters so much. But even a careful test only checks the past. Once a model is in use, someone has to keep watching it.

How do we catch it?

The idea is simple: keep comparing the present with the past.

Watch the inputs. Compare the data coming in with the data the model learned from. Are the houses getting bigger? Are new neighborhoods showing up? If the inputs drift away from the training data, that's an early warning, even before you know whether the predictions are wrong.

Watch the results. When the true answers arrive, like a house's actual sale price, or a person marking an email as spam, keep tracking the model's metrics over time. A slow decline is the signature of drift.

Retrain regularly. When the world has moved, the fix is usually to train the model again on recent examples, so its snapshot is up to date.

Watching models in use is a big part of keeping AI systems healthy, and we'll come back to it in the Evaluation & Ops chapter.

So, when does it stop working?

When the data it meets stops looking like the data it learned from. That happens with rare cases it barely saw, new cases it never saw, and a world that drifts away from its training data over time. The model rarely warns you. So we help it learn rare cases on purpose, flag inputs that look unfamiliar, and keep watching its results long after training ends.