Learn from examples

We ended the last waypoint with a promise: the computer learns where to place things. But what does "learn" mean for a computer?

Start with what we want from it. Almost every AI task has the same shape. Something goes in, and something should come out. A photo goes in, and "dog" or "cat" should come out. An email goes in, and "spam" or "not spam" should come out. A house's details go in, and a fair price should come out.

So what we really want is a rule that turns inputs into the right outputs. The question is where that rule comes from.

Why not just write the rules ourselves?

That was the original plan, and for many problems it works. It's called traditional programming: a person writes the rules, and the computer follows them. But try it with spam: if an email says "you've won," mark it as spam. Then a real email arrives from a friend who just won a raffle. You add an exception, and another, and another. The rules start to pile up and still miss cases.

Now try writing rules for spotting a dog in a photo. Pointy ears? Some dogs have floppy ears. Fur? So do cats. Four legs? Not if the dog is lying down. You can't even explain how you recognize a dog, so you certainly can't write it down as rules.

For most interesting problems, we can't write the rule. But we have something else: lots of examples.

What if the computer learned from examples?

Let's take a simple case where we can see everything. We want to predict house prices from their size, and we have a few houses that already sold:

Each row is an example: an input and the right answer. What we want is a rule that fits these examples well enough that, when a new house comes along, say 1,800 sq ft, it gives a sensible price.

A rule that turns an input into an output has a name in math: a functionA rule that turns an input into an output.. That's what the computer learns. Not facts, and not a list of answers, but a function.

What does a function look like to a computer?

Draw the examples as points, with size on one axis and price on the other. They roughly line up. So a natural first guess for our rule is a straight line through them:

price = (some amount) × size + (some starting amount)

Those two blanks are the interesting part. Change the first one and the line gets steeper or flatter. Change the second and the whole line shifts up or down. Every pair of values gives a different line, which means a different rule.

These adjustable values are called parametersKnobs on a rule. Turn them, and the rule changes.. Think of them as knobs. The shape of the rule is fixed (a straight line), and learning means finding the knob settings that make the line fit the examples. From here on, we'll call this rule and its knobs a "modelA rule with a fixed shape and knobs you can turn.".

We usually won't find the perfect rule. The real relationship between size and price depends on location, condition, and a hundred other things. What we find is a rule that comes close. That's why it's called functionapproximationA close rule, found from examples instead of written by hand.: approximating a rule we can't write down, using examples of what it should do.

What if the world isn't a straight line?

Our house example was kind to us. The points lined up, so a straight line with two knobs was enough. Most real problems aren't that tidy. The rule that separates dog photos from cat photos is far too twisted to draw with a straight line.

The answer is a more flexible model, one with many more knobs that can bend into almost any shape. The most successful kind is the neuralnetworkMany simple calculations stacked in layers, each with its own knobs.: many small, simple calculations connected in layers, each with its own knobs. One layer alone can't do much, but stacked together they can approximate astonishingly complex rules. Large models have millions or even billions of knobs.

What's important is that the idea doesn't change. There's a shape for the rule, knobs to adjust, and examples to fit. The house-price line is just the smallest version of it.

So, what does a computer learn?

The computer learns a function. The computer doesn't get the rule from us. It gets a shape with adjustable knobs and a set of examples, and learning means finding the knob settings that fit those examples.

Now, you might be wondering: of all the possible knob settings, how does the computer tell a good one from a bad one? Before it can find the right rule, it needs a way to measure how wrong a guess is. That's the next waypoint: Loss.