In the last lesson we let a slot be a word, to keep things simple. That was a small lie, told on purpose. This lesson gives you the real building block — a token — and it's the missing piece that completes our glossary of how any of this actually works.
We called the bins "words" — mouse, fish, bird — because it was the easiest way to picture the idea. It made the leap from numbers to language feel obvious.
It isn't quite right, though. Real systems don't slice language into whole words. It's worth being precise now, because the real answer is just as simple — and it explains a few things a "words" story can't.
The real building block is called a token. A token is a chunk of text — sometimes a whole word, sometimes only a piece of one, sometimes just a punctuation mark or a space.
If it helps to have a size in mind: a token sits somewhere between a letter and a whole word — closer to a syllable than anything else. Don't lean on that too hard, though. A token isn't cut where you'd break a word out loud; it's cut by how often that chunk of text showed up during training. So it can split a word in an unexpected place, or leave a two-syllable word whole, or turn a single punctuation mark into a token of its own. Syllable is the right size to picture — just not the rule it follows.
A short, common word like "cat" or "the" usually gets a bin all to itself — one token. A longer or rarer word often gets cut into a few familiar pieces instead, each with its own bin.
This isn't an arbitrary quirk. It's the whole reason the board can handle text it has never seen before.
The full set of possible tokens — the board's vocabulary — is a fixed list, typically tens of thousands of pieces. That's a manageable number of bins to have painted floor for. A brand-new name, a typo, a word borrowed from another language, a line of computer code — none of it needs its own dedicated bin. It just gets broken down into smaller, familiar pieces the board already knows, the way "unbelievable" became "un" + "believ" + "able." A vocabulary of whole words only would need millions of entries, and would still stumble on the next new one.
This also explains a well-known, easy-to-reproduce quirk. Ask an AI something like "how many r's are in strawberry," and it can genuinely get it wrong.
It isn't being careless. It's reading "strawberry" as a token or two, the way you might glance at a whole word without inspecting each letter — not as s-t-r-a-w-b-e-r-r-y, one at a time. Asking it to count letters is asking it to see something it was never shown letter by letter in the first place.
Tokens aren't only what flows in and out while you're talking to it. Everything the board was shown during training gets counted the very same way.
The entire mountain of text an AI reads before it ever reaches you — books, articles, code, conversations — is measured in tokens, the same small alphabet, just an enormous pile of it.
Structure, from Lesson 1: the pegs, the shape coins fall through. Weights, from Lesson 2: the floor, painted by every coin that ever passed through training. And now, tokens: the actual alphabet that reading was done in.
Put the three together and there's a name for the whole, assembled thing: a model. "Board" was never wrong — it's still built from pegs and a floor — but from here, this course starts calling it what it actually is.
Every "model" from here on is still that same coin-and-pegs machine — pegs, painted floor, alphabet, nothing added. If you want to see the real device this course borrowed it from, and the history behind it, visit the Galton board page.
One thing was skipped on purpose: a token doesn't touch the floor as a word. It has to become something the floor can work with first — and that step turns out to be one of the most useful ideas in all of this.
Everything from the lessons before this one, plus what this lesson added.
| The board | The word for it |
|---|---|
| The board and its pegs | The structure — the model |
| A single peg | A neuron |
| A single step from one peg to the next | A connection |
| How red or blue a patch of floor is | A weight — a parameter |
| The whole painted floor | The trained model |
| Boards stacked, one on the next | Layers — "deep" |
| Smearing the floor in, coin by coin | Training |
| Dropping one coin through a finished board | Using it — inference |
| Which slot you pour into | Your input — your prompt |
| The bins, relabelled with tokens | The vocabulary — every token it could pick |
| The tallest pile among the bins | The prediction — the likeliest next token |
| Feeding the growing line back in and running again | The loop — how one guess becomes a sentence |
| + New in this lesson | |
| One bin's label, precisely | A token — not always a whole word |
| How many bins there are, in total | Vocabulary size — tens of thousands |
| The size of the mountain of text it was shown | Training data — measured in tokens |
15 rows now. It keeps growing as the course goes on.