Not part of the fourteen lessons — and genuinely not beginner material anymore. This is the one mechanism that actually ties "context," "the loop," and "layers" together: attention. For anyone who's finished the course and wants to see how the shaping is really done.
Lesson 8 handed the model a growing pile of tokens — background, examples, the whole conversation so far — and Lesson 9's backpack showed that pile getting resent, in full, every single time. But nothing in those lessons ever explained how one token in that pile actually gets to influence how another token is read. This page is that missing piece.
It matters more than it sounds. Take a sentence like "The trophy didn't fit in the suitcase because it was too big." Nothing about the word "it" tells you, by itself, whether it means the trophy or the suitcase. A model gets this right constantly — and the mechanism that lets it is the subject of this whole page.
Before a token is actually read, it briefly compares itself to every other token already in the line, and leans toward whichever ones turn out to matter most for it, right now. That comparing-and-leaning step is called attention.
It's a close cousin of something you already know. Lesson 5's similarity measured how close two things sit in the same space. Attention runs that same comparison internally, between tokens in the very sentence being read — not as an outside search, but as a step baked into reading itself.
The comparison runs through three roles. A query is what a token is asking: broadly, "who's relevant to me?" A key is what a token offers to be found by — its own advertisement for the kinds of questions it can answer. A value is what a token actually hands over, once it's been picked.
Back to the trophy and the suitcase. "It" sends out a query. "Trophy" and "suitcase" both offer keys back. The two barely differ in position or spelling — but training has taught the model that "too big to fit" pairs far more often with the object going in than the container receiving it, so "trophy"'s key matches "it"'s query far more strongly. "It" ends up pulling most of its value from "trophy," almost none from "suitcase."
Here's the real departure from everything earlier in this course. Lesson 3's pile always had one tallest stack — a single winner, taken, the rest discarded. Attention doesn't work that way. Every token gets some weight, in proportion to how relevant it is — a close match pulls hard, a distant one barely pulls at all, but almost nothing gets reduced all the way to zero.
A strictly sequential reader has to carry everything forward, one token at a time, hoping nothing important gets dropped along the way — exactly the pressure behind Lesson 9's backpack getting more expensive as it grows. Attention skips the relay race entirely: any token can reach directly back to any other token in the line, however far away, in a single step.
That reach is also the real reason context gets expensive. Attention compares every token against every other token in the line, so the cost doesn't grow gently as a conversation lengthens — it grows fast. Lesson 9's pricing table wasn't being dramatic.
One round of attention only notices one kind of relevance. Real models run several rounds side by side — called heads — each free to notice something different: one tracking who's doing what to whom, another tracking plain word order, another something no one bothered to name.
Nobody assigns each head its job. Same as everything else in this course: training just leaves each head specializing in whatever turned out to be useful.
Remember Lesson 2's "deep" — boards stacked on boards. A transformer stacks a specific pair, over and over: one round of attention, then an ordinary weighted layer, then attention again. Each round refines what matters based on what the last round already worked out.
By the last layer, a token's representation has been reshaped by everything relevant to it, gathered in from across the whole line, several times over — not because the line got reread, but because attention let each round reach straight to what mattered.
Attention plus an ordinary layer, stacked repeatedly: that whole structure is a transformer. It's the T in GPT — Generative Pre-trained Transformer — and it's the actual mechanism sitting underneath every "context" and "carried history" idea this course already taught the outcome of.
The rest of this course described what pouring more in does. This page is how the pouring actually gets read.
| The board | The word for it |
|---|---|
| Sending a coin through the same fixed floor, alone, no lookaround | An ordinary layer |
| A token's "who's relevant to me?" signal | A query |
| A token's "here's how to find me" advertisement | A key |
| What a token actually hands over, once picked | A value |
| Blending a little of everyone, weighted by relevance | Attention |
| How much weight one token gives another | An attention score |
| Several attention passes side by side, each noticing something different | Multiple heads |
| Attention and a weighted layer, stacked over and over | A transformer |
Every term on this page — layers, context, similarity, the loop — comes from the fourteen lessons. If any of it felt unfamiliar, that's where it's built up from the ground.