A token doesn't touch the floor as a word — it has to become a list of numbers first. This lesson shows what that means, why "cat" and "dog" end up near each other and "cat" and "umbrella" don't, and why France is to Paris as England is to London.
Lesson 4 gave every token a label — a bin, a slot in the vocabulary. But the floor from Lesson 2 doesn't actually compute with labels. It computes with numbers. Something has to turn one into the other, and that something is this lesson.
It's worth slowing down for. This one idea — meaning, turned into a location — is one of the most useful tricks in modern AI outside the model itself. Once you can see it, a lot of things that used to feel like magic stop being mysterious.
An embedding is a list of numbers standing in for a word — a set of coordinates, the same way "3 miles east, 2 miles north" is a set of coordinates for a place. Two coordinates would fit on a napkin; a real embedding usually runs into the hundreds. The idea is exactly as simple as the napkin, though — just with more directions than a page can draw.
The coordinates aren't random, and nobody assigns them by hand. They fall out of training, the same way the floor's colours did in Lesson 2 — the model quietly nudges related words toward each other and unrelated words apart, millions of times, until the map settles into something genuinely useful.
That's the whole picture in miniature. Real embeddings use far more than two numbers, so the actual space can't be drawn on a page — but two is enough to show the idea, the same kind of simplification this course has leaned on since Lesson 1.
Once meaning is a location, comparing two things becomes something you can actually measure: how far apart are their points? Close together means related. Far apart means not. That measurement has a name — similarity — and it's a real number a computer can calculate instantly, for millions of pairs at once.
"Cat" and "dog" sit close. "Cat" and "umbrella" sit far apart. Nobody wrote a rule that says so. It's just where training left them.
This measurement usually goes by a specific name out in the wild: cosine similarity. If you see that phrase in a search tool's settings or an API's documentation, it's this — nothing more exotic than "how close are these two points."
There's a word for "about meaning, rather than exact wording": semantic. A semantic match finds "puppy" when you searched for "young dog," even though not one letter overlaps — because the two sit close together in the space, not because they share spelling.
Zoom out, and whole neighbourhoods appear — clusters of words that all belong to the same topic, sometimes called a domain. Animal words drift into one neighbourhood, weather words into another, entirely on their own. Nobody drew the borders in the figure above. The borders are just where similar things ended up standing next to each other.
Here's what actually convinced people this space captures real meaning, not just rough topic clusters. Relationships turn out to be directions — and the same relationship is the same direction, wherever you stand.
Nobody told the model what a capital city is. That relationship fell out on its own, as a byproduct of reading enough text where those pairs showed up the same way, over and over.
Here's what makes this genuinely useful, not just a neat diagram. If "capital of" is a direction, you can run the whole thing forward: start at a country, take that exact step, and see where you land — even if nobody ever told you the answer.
Take England's coordinates. Add the same offset that carried France to Paris. The result is a brand-new point that nobody plotted in advance — it doesn't have a label yet. It's just a location.
That last move — scanning every word's coordinates and asking which one sits closest to a point — is called a nearest neighbor search. It's the same measurement as similarity, just run in the other direction: instead of comparing two words that already exist, you manufacture a brand-new point through arithmetic, then ask the space what real word is standing nearest to it.
That's the queryable part. This space isn't just a picture to admire — it's something you can actually compute with.
This isn't just a tidy diagram. In 2024, Anthropic's interpretability researchers went looking inside Claude itself and found a specific direction that corresponded to one very particular concept — the Golden Gate Bridge. They turned it up. The result was a version of Claude that steered almost every answer back toward the bridge, no matter what it was actually asked.
It's a strange, fairly funny demonstration of something serious: these directions are real, findable, and closer to a dial than a black box. Optional, but worth a look if you want to see the idea taken further than a textbook example:
This is the real answer to a question that comes up constantly outside this course: how does a search tool find the right result even when you didn't type its exact words? Turn the question into a point, turn everything in the archive into a point, and hand back whatever landed closest.
It's a genuinely useful trick, with a real limit: distance can tell you two things are related, never how. Curious how a platform builds past that limit — using distance to find a starting point, then reasoning further with real structure? See orqo's knowledge graph.
This exact pattern — search, then hand the model what you found — has its own name too, common enough that it gets a lesson of its own later in this course.
It's worth untangling something before moving on. Turning words into embeddings, measuring similarity, finding a nearest neighbor — none of that, by itself, is what people usually mean by "an AI."
Every real language model does have the token-to-embedding step built in, right at the start, exactly as Lesson 4 described it — that much genuinely is the model. But when embeddings get used for search, the way the section above just described, that job is usually handed to a separate, much smaller embedding model — something built to do exactly one thing, turn text into coordinates, and nothing else. It doesn't write sentences. It doesn't hold a conversation.
And the step after it — finding the nearest neighbor — isn't AI in the generating sense at all. It's arithmetic: sort a list of numbers by distance, and see what's closest. A working semantic search tool can run without any large language model anywhere inside it. The two ideas are related — the same core trick, meaning as a location — but they're not the same machine.
Structure, weights, tokens, and now the map they all sit on top of — that's the complete idea. The next lesson puts real numbers on it: how many parameters, how much reading, how big this actually gets.
Everything from the lessons before this one, plus what this lesson added.
| The board | The word for it |
|---|---|
| The board and its pegs | The structure — the model |
| A single peg | A neuron |
| A single step from one peg to the next | A connection |
| How red or blue a patch of floor is | A weight — a parameter |
| The whole painted floor | The trained model |
| Boards stacked, one on the next | Layers — "deep" |
| Smearing the floor in, coin by coin | Training |
| Dropping one coin through a finished board | Using it — inference |
| Which slot you pour into | Your input — your prompt |
| The bins, relabelled with tokens | The vocabulary — every token it could pick |
| The tallest pile among the bins | The prediction — the likeliest next token |
| Feeding the growing line back in and running again | The loop — how one guess becomes a sentence |
| One bin's label, precisely | A token — not always a whole word |
| How many bins there are, in total | Vocabulary size — tens of thousands |
| The size of the mountain of text it was shown | Training data — measured in tokens |
| + New in this lesson | |
| A bin's own address, plotted in space | An embedding — meaning turned into a list of numbers |
| How close two addresses sit | Similarity — how related two things are |
| The measurement's real name, out in the wild | Cosine similarity — same idea as similarity, different name |
| About meaning, not exact spelling | Semantic |
| A neighbourhood of addresses that share a topic | A domain — a cluster of related meaning |
| Scanning every address for whichever one sits closest to a new point | Nearest neighbor — the closest real match to a computed point |
| A small, separate machine that only plots words as points — it never writes a sentence | An embedding model — not the same as a language model |
22 rows now. It keeps growing as the course goes on.