Optimization

At the end of the last waypoint, the computer could finally tell how wrong a guess is. The lossOne number for how wrong a guess is. Big is bad, small is good. gives it a single number: big means bad, small means good. Learning now has a clear goal: find the knob settings that make the loss as small as possible.

But a score isn't a plan. Knowing you're wrong doesn't tell you what to change. So how does the model actually get better?

Why not just try every setting?

With our house example, that almost works. Keep the second knob at zero and try every value for the first one, from $100 to $300 per square foot, checking the loss each time. A few hundred tries, and you'd find the lowest point. It turns out to be about $202 per square foot.

Now add the second knob. For every value of the first, you have to try every value of the second. A few hundred tries become tens of thousands. Each new knob multiplies the work again.

Real models have millions or billions of knobs. Even if each knob could only take ten values, a model with just a hundred knobs would have more combinations than there are atoms in the universe. Trying everything is out of the question.

What about guessing randomly?

You could pick random settings and keep the best one you find. But almost every random change makes things worse, not better, and with billions of knobs you'd be guessing forever. We need something smarter than luck.

What if we just nudged one knob and checked?

That's a much better idea. Start where we left off at the last waypoint, at $150 per square foot. Try $151 and check the loss. It went down. So keep turning that way: $152, $153, and so on, checking each time.

Now the model isn't guessing anymore. It's following a direction. Each small change tells it whether it's getting warmer or colder.

The problem is speed. Checking one knob at a time means running the whole model once for every knob, just to find out which way to turn it. With billions of knobs, that's still far too slow.

Can it know the direction without trying?

Yes, and this is the key idea behind almost all of modern AI.

Picture the loss as a curve. For our first knob, plot every setting from $100 to $300 on one axis and its loss on the other. You get a valley shape: high on both sides, lowest around $202. At any point on that curve, the ground slopes one way or the other. If you know the slope where you're standing, you know which way is downhill, without taking a single test step.

Calculus gives the computer a way to calculate that slope directly. And it can do this for every knob at once, in a single pass through the model. The collection of all those slopes, one per knob, is called the gradientThe slope of the loss for every knob at once. It points uphill.. It's like a compass that points uphill for every knob, so the model just goes the opposite way.

For neural networks, the method that calculates all those slopes efficiently, working backward from the loss through each layer, is called backpropagationHow a neural network calculates those slopes, working backward from the loss.. You don't need its details to follow the idea. What matters is that a model with billions of knobs can still find out, in one go, which way to turn each of them.

Picture a hillside in the fog

Imagine standing on a hillside in thick fog, trying to reach the bottom of the valley. You can't see the valley, but you can feel which way the ground slopes under your feet. So you take a small step downhill. Then you feel again and take another step. Eventually you reach low ground.

That's exactly what the model does. Your position is the knob settings. The height of the ground is the loss. Feeling the slope is calculating the gradient. And the walk itself, one small step downhill after another, is called gradientdescentThe walk downhill: one small step against the gradient, then another.. It's how nearly every modern AI model learns.

How big should each step be?

This turns out to matter a lot. Say the model is at $150, and the slope says to go up. If it jumps all the way to $260, it overshoots the valley and ends up on the other side. Now every house is overpriced, and the loss is even worse than where it started:

Steps that are too big make the model bounce back and forth across the valley, never settling down. Steps that are too small are safe but painfully slow, like crossing a valley in tiny shuffles.

The size of each step is set by a number called the learningrateHow big each downhill step is.. It's one of the few things a person still chooses by hand, and choosing it well is part of the craft of training models.

What if there are millions of examples?

There's one more practical problem. To calculate the loss exactly, the model has to check every example. With three houses, that's nothing. With millions of photos or billions of sentences, checking all of them before every single step would take forever.

So instead, the model looks at a small random handful of examples, called a batchA small random handful of examples, used for one step., and takes a step based on those. Any one batch gives a slightly rough sense of which way is downhill, but the rough guesses average out over many steps, and the model moves far faster. This version is called stochasticgradientdescentGradient descent that steps from a batch instead of every example., where "stochastic" just means "with some randomness." It's how large models are actually trained.

When does it stop?

The loop repeats: make predictions, measure the loss, calculate the gradient, take a step. Each round, the loss drops a little. Early on it falls fast. Later the ground flattens out, the steps make less and less difference, and the loss stops improving. That's when training usually stops.

The model isn't guaranteed to find the very lowest point. For complex models, the landscape is full of hills and dips, and it may settle in a valley that isn't the deepest one. In practice, the valleys it finds are usually good enough, and they're what every model you've ever used was built on.

This is also how the vectors from the first waypoint get their places. Every number in them is a knob, and every one is moved by this same walk downhill, until similar things end up near each other because that's what made the loss smaller.

So, how does it improve?

One small step at a time. The model calculates which way each knob should turn to lower the loss, nudges all of them a little in that direction, and repeats, often millions of times. Finding the best settings this way is called optimizationFinding the knob settings that make the loss as small as possible., and gradient descent is the method almost every model uses for it.

But there's a catch we haven't faced yet. Everything so far has been about lowering the loss on the examples the model has seen. A flexible enough model could push that loss to zero just by memorizing every example, like a student who memorizes the answer key without understanding the subject. Then it would fail on the first new house it sees. How can we tell whether a model really learned, or just memorized? That's the next waypoint: Generalization.