Tokens

In Foundations, we learned that a computer sees everything as numbers. Words become vectorsLists of numbers. When the numbers are chosen well, similar meanings sit close together., lists of numbers where similar meanings sit close together. But we skipped a question that turns out to matter a lot.

Text doesn't arrive as neat words with vectors attached. It arrives as a stream of characters: letters, spaces, punctuation, numbers, emoji, typos, names, and words in every language. Before a model can turn text into vectors, it has to decide something basic: which pieces of text get a vector?

Why not give every word its own vector?

That's the natural first idea. Make a list of every word in the language, give each one an ID number, and give each ID its own vector. To read a sentence, look up each word and grab its vector.

It works until you try it on real text. Then the problems pile up.

The list never ends. English alone has hundreds of thousands of words, and real text is full of names, product codes, slang, typos, and words that didn't exist last year. Any word missing from the list is invisible to the model. "Helo" isn't on the list, so the model sees nothing there.

Related words don't share anything. "Run," "runs," "running," and "runner" are four separate entries, and the model has to learn each one from scratch, even though they obviously belong together.

And remember Rare cases. Most words in a big list are rare. A word that shows up only a handful of times in all the training text gives the model almost nothing to learn its vector from.

Then why not one vector per letter?

That fixes everything at once. There are only around a hundred letters, digits, and symbols in everyday English text, so the list is tiny. No word is ever missing, because every word is made of letters. Typos, names, and new words are all readable.

But it creates a new problem. A single letter carries almost no meaning. The letter "c" tells the model nothing about cats. So the model has to do all the work of assembling meaning from scratch, letter by letter, every time.

It also makes everything much longer. A sentence of ten words becomes fifty or sixty letters. As we'll see in this chapter, the work a model does grows quickly with the length of what it reads, so longer means slower and more expensive.

Whole words are too big, and letters are too small. Is there something in between?

What about pieces of words?

Here's the idea that won. Common words stay whole. Rare words get split into smaller, familiar pieces. A sentence like

The puppy was unbelievably happy.

might be cut into pieces like these:

The · puppy · was · un · believ · ably · happy · .

"Puppy" and "happy" are common enough to keep whole. "Unbelievably" is rarer, so it's split into pieces the model has seen many times in other words: "un" as in "unhappy," and "ably" as in "comfortably." A word the model has never seen can still be read through pieces it knows.

Each of these pieces is called a tokenA piece of text that gets its own vector. Often a word, sometimes only part of one.. Cutting text into tokens is called tokenizationCutting text into tokens., and the tool that does it is the tokenizerThe tool that cuts text into tokens.. The full list of tokens it can use is the vocabularyThe full list of tokens a model can use., which for modern models usually holds somewhere between tens of thousands and a couple hundred thousand tokens.

And because the smallest pieces are single characters, any text at all can be read. In the worst case, a strange word just gets split into very small pieces.

Who decides what the pieces are?

Nobody does, at least not by hand. The pieces are learned from a large amount of text, by counting.

Start with every single character as its own token. Then look through all the text and find the pair of neighboring tokens that appears together most often. Maybe "t" followed by "h." Merge them into a new token, "th." Count again. Maybe "th" followed by "e" is now the most common pair, so it becomes "the." Keep merging the most common pair, over and over, until the vocabulary reaches the size you want.

The result follows naturally from the counting. Frequent words end up as single tokens, because their pieces got merged early. Rare words stay split into pieces, because they never came up often enough to be merged. This method is called bytepairencodingBuilding a vocabulary by repeatedly merging the pair of pieces that shows up together most often., and most modern tokenizers use it or something close to it.

How do tokens become vectors?

Once the tokenizer is done, every token has an ID number in the vocabulary. The model keeps a big table with one vector for every token. To read a sentence, it tokenizes the text, looks up each token's vector, and works with the resulting list of vectors from then on.

Those vectors are the ones we met in Vector representation. They aren't set by hand. They're knobs, learned by gradientdescentThe walk downhill: one small step against the gradient, then another. along with everything else, until tokens that are used in similar ways end up close together.

So by the time the model starts thinking, a sentence is no longer text. It's a sequence of vectors, one per token.

Why does this matter in practice?

Tokens might seem like a technical detail, but they show up everywhere once you use AI in practice.

Tokens are how models are measured and priced. The cost of using a model is usually counted in tokens, both the text you send and the text it writes back. The amount of text a model can work with at once is also measured in tokens. In English, a token averages roughly three quarters of a word, so a thousand tokens is about 750 words.

Not every language is treated equally. A tokenizer learns its pieces from its training text, and that text is often mostly English. Other languages may get fewer whole-word tokens, so the same sentence can be split into more pieces. That means it can cost more, fill up the model's limit faster, and give the model less to work with. It's the problem from Rare cases again, showing up in a new place.

Some odd mistakes come from tokens. Ask a model how many letters are in a word, and it may get it wrong. That sounds strange, until you remember that the model doesn't see letters. It might see "straw" and "berry" as two tokens, with the individual letters hidden inside them. Numbers can be split in uneven chunks too, which is one reason models sometimes stumble on arithmetic.

Do images and sound become tokens too?

Yes, and this is where the idea becomes powerful. A model that reads images cuts each picture into a grid of small square patches, and each patch becomes a token with its own vector. Audio is cut into short slices of sound. Video becomes patches across both space and time.

Once everything is a sequence of vectors, the same kind of model can work with all of it. That's how models that handle text and images together work: the image patches and the word pieces go into one sequence, side by side.

So, how does a model read text?

In pieces. The tokenizer cuts text into tokens, pieces chosen by counting what appears most often in a large amount of text: common words whole, rare words split into familiar parts. Each token is looked up in a table of vectors, and from then on, the model works only with those vectors. The same idea lets models read images and sound, by cutting them into pieces too.

Now every token has a vector. But each token gets the same vector every time it appears, whatever the words around it. What do these vectors actually hold, and how much of a word's meaning can a single list of numbers carry? That's the next waypoint: Embeddings.