Probability and uncertainty

Our model can now learn a rule that generalizes. Give it a house, and it gives back a price. But look at what it gives back: one number, delivered the same way every time.

Ask it about a 1,600 sq ft house, right in the middle of the houses it learned from, and it says about $321,000. Ask it about a 5,000 sq ft mansion, more than twice the size of anything it has seen, and it says about $988,000. Both answers look equally confident. But one rests on plenty of similar examples, and the other is a long stretch past anything the model knows.

A single answer hides something important: how much we should trust it.

Why does "how sure" matter?

Because real decisions depend on it. A bank using a model to value homes should treat a confident estimate differently from a rough guess. A spam filter that's nearly certain can delete an email. One that's unsure should put it in a folder for a person to check. A medical model that's unsure should hand the case to a doctor.

A model that knows when it's unsure is far more useful than one that sounds confident about everything. So how can a model tell us how sure it is?

What if the answer were a chance instead of a choice?

Start with the simplest case, a yes-or-no question: is this email spam?

So far we've imagined the model answering "spam" or "not spam." Instead, let it answer with a number between 0 and 1: the chance that the email is spam. An answer of 0.97 means "almost certainly spam." An answer of 0.55 means "slightly more likely spam than not, but I really don't know." This number is a probabilityA number between 0 and 1 for how likely an answer is. The certainty is inside the answer., and it carries the model's certainty right inside the answer.

The decision comes afterward. We choose a cutoff, called a thresholdThe cutoff that turns a probability into a decision, like marking spam above 0.9., for example: mark as spam anything above 0.9. Where to put the threshold depends on which mistakes are worse, and we'll come back to that at the Metrics waypoint.

What if there are more than two options?

Say the model looks at a photo and has to decide between dog, cat, and fox. Now it gives a probability to each option, and together they add up to 100%:

A set of probabilities covering every possible answer like this is called a probabilitydistributionA probability for every possible answer, all of them adding up to 100%..

Both photos have "dog" as the top answer. But the distributions tell very different stories. For photo A, nearly all the probability is piled on one option, so the model is sure. For photo B, it's spread across all three, so the model is barely leaning one way. The shape of the distribution is the model's certainty. Piled up means confident. Spread out means unsure.

This is also how models that write text work. At each step, they produce a distribution over every possible next word, and the shape of that distribution says how sure they are about what comes next.

How does a model produce probabilities?

Inside, the model first produces a raw score for each option. These scores can be any number, positive or negative, and they don't add up to anything in particular. A final step converts them into probabilities: it makes every score positive, keeps the highest score the highest, and scales them all so they add up to 100%. This step is called softmaxThe step that turns raw scores into probabilities: all positive, same ranking, adding up to 100%..

So the model's answer is no longer one option. It's a whole distribution, and the options it ranks highest get most of the probability.

How does it learn honest probabilities?

Back in the LossOne number for how wrong a guess is. Big is bad, small is good. waypoint, we met cross-entropyThe loss for answers that aren't numbers. It looks only at the probability on the right answer, and punishes confident mistakes the most., the loss for answers that aren't numbers, and its one simple idea: being confidently wrong should cost the most. Now we can see how it works.

Cross-entropy looks at only one thing: how much probability the model gave to the right answer. The more it gave, the smaller the penalty. And as that probability drops toward zero, the penalty shoots up:

Look at how the penalty grows. Going from 99% to 90% barely costs anything. Going from 10% to 1% costs more than everything above it combined. A model that says "1% chance this is a real email" about an important message from your boss gets hit hard.

That shapes how the model behaves. Confidence is cheap when it's right and very expensive when it's wrong. So, through gradientdescentThe walk downhill: one small step against the gradient, then another., the model learns to pile probability onto one answer only when the evidence really supports it, and to spread it out when it doesn't.

Can we trust the percentages?

Not automatically. Think of a weather forecaster. If they say "70% chance of rain" on a hundred different days, it should rain on about seventy of them. If it rains on all hundred, or on only ten, their percentages don't mean what they say, even if their forecasts are often right.

A model is the same. When it says 90%, it should be right about 90% of the time. A model whose probabilities match reality this way is called calibratedWhen the model says 90%, it is right about 90% of the time..

Many models aren't. Large neural networks in particular are often overconfidentSaying 99% when it is really right much less often.: they say 99% when they're really right only 85% of the time. Fortunately, this is easy to check. Take the validationsetHeld-back examples used to compare versions and choose among them., group the predictions by how confident they were, and compare each group's confidence with how often it was actually right. If they don't match, there are simple ways to adjust the probabilities afterward.

What about answers that are numbers?

The same idea works for house prices. Instead of one number, the model can give a range: "about $361,000, most likely between $345,000 and $377,000." A narrow range means the model is sure. A wide one means it isn't. That range is also a way of describing a distribution, just over prices instead of over options.

Can a model know what it doesn't know?

This is the hardest part, because uncertainty comes from two different places.

The first is the world itself. Even a perfect model can't know that a particular house has a great view if the data doesn't say so. Some things just vary, and no amount of training removes that. A good model learns to show this kind of uncertainty, because it sees it in its examples over and over.

The second is gaps in what the model has seen. The 5,000 sq ft mansion is a case like this. The problem isn't that mansions are unpredictable, it's that the model has never seen one. And this is where models often fail: they don't know that they don't know. The straight line happily extends itself to $988,000 with no warning, because nothing in its training ever showed it where its knowledge ends.

(In technical writing, you'll see these two called aleatoricUncertainty from the world itself. The data doesn't say everything, so even a perfect model can't remove it. and epistemicuncertaintyUncertainty from gaps in what the model has seen, like a house unlike any in its training..)

Catching that second kind takes extra work, and it's one of the main ways models fail in practice. We'll come back to it at the Rare cases and Drift waypoint.

So, how sure is it?

The model can tell us, if we let it answer with a distribution instead of a single choice. The shape of that distribution shows its certainty: piled up when it's sure, spread out when it isn't. Cross-entropy teaches it to be confident only when the evidence supports it, and checking calibration tells us whether its percentages mean what they say. What a model can't easily tell us is when it's facing something truly new.

Now we have a model that makes predictions and says how sure it is. But is it actually good? So far, we've judged models by their loss. The loss is what the model is trained on, but is it what we really care about? A spam filter with a low loss might still delete the one email that mattered. How do we measure whether a model works for the job it's meant to do? That's the next waypoint: Metrics.