Next-token prediction alone doesn't explain why a model answers you helpfully instead of just continuing your sentence, or why it knows when to say no. This lesson names the second training pass behind that — RLHF, and the broader goal it serves.
Go back to Lesson 3's "Cat eats ___." Left purely to next-token prediction, a model just continues whatever pattern is in front of it. So why, when you type "How do I bake bread?", doesn't it just continue with more questions in the same style — "How do I bake a cake? How do I roast a chicken?" — the way a raw pattern-continuer plausibly might?
Real assistants reliably answer you instead. They also, just as reliably, decline to help with clearly harmful requests. Neither behavior is guaranteed by next-token prediction alone — training on the open internet teaches a model to continue any pattern that shows up there, harmful ones included. Something else is happening, on top of everything Lesson 2 already told you.
What Lesson 2 called training is really only the first pass — reading a vast scrape of the internet, painting the floor to continue whatever pattern it's shown. That pass is usually called pretraining, and it's where almost all the reading happens. But nearly every model you'd actually talk to gets a second pass afterward, on a much smaller, carefully chosen set of examples, aimed at one specific goal: behaving like a helpful, truthful assistant instead of a raw pattern-continuer.
That second pass is usually called post-training. Mechanically, nothing about it is new — it's still weights getting nudged, still next-token prediction underneath. It's the same floor, painted a second time, with a far smaller brush and a far more specific target.
RLHF — Reinforcement Learning from Human Feedback — is the best-known way that second pass gets done. For the same prompt, the model produces several different draft answers. A person reads them and ranks them, best to worst. That ranking is used to train a small helper — a reward model — to predict which kind of answer people prefer. The language model then gets nudged, over and over, toward whatever the reward model scores highest.
This doesn't fix Lesson 7. The model isn't reasoning any harder, and it isn't checking its own facts. All that changed is which continuations the floor now favors — toward answering directly, toward admitting uncertainty instead of guessing, toward declining certain requests outright.
That's also the real answer to "why does it refuse to help with that?" It isn't a rule bolted on from outside. Somewhere during this second pass, "decline and explain why" was the continuation that scored highest for prompts that looked like this one — the exact same kind of nudge that once made "mouse" win for "Cat eats ___," just aimed at a different, much narrower target.
RLHF is one specific technique. The broader goal it serves has its own name: alignment — the general project of making a model's behavior actually match what people want, not just what's statistically likely to follow in a wall of internet text. It's an active, fast-moving field; RLHF is the best-known method, not the last word on it.
Every time you've heard a model described as helpful, truthful, and harmless, that's alignment work being described in three words.
Alignment is why a model usually behaves helpfully and knows when to say no. "Usually" isn't "always," and none of it says anything about what happens to what you actually type. The last lesson covers that other half of the question.
Everything from the lessons before this one, plus what this lesson added.
| The board | The word for it |
|---|---|
| The board and its pegs | The structure — the model |
| A single peg | A neuron |
| A single step from one peg to the next | A connection |
| How red or blue a patch of floor is | A weight — a parameter |
| The whole painted floor | The trained model |
| Boards stacked, one on the next | Layers — "deep" |
| Smearing the floor in, coin by coin | Training |
| Dropping one coin through a finished board | Using it — inference |
| Which slot you pour into | Your input — your prompt |
| The bins, relabelled with tokens | The vocabulary — every token it could pick |
| The tallest pile among the bins | The prediction — the likeliest next token |
| Feeding the growing line back in and running again | The loop — how one guess becomes a sentence |
| One bin's label, precisely | A token — not always a whole word |
| How many bins there are, in total | Vocabulary size — tens of thousands |
| The size of the mountain of text it was shown | Training data — measured in tokens |
| A bin's own address, plotted in space | An embedding — meaning turned into a list of numbers |
| How close two addresses sit | Similarity — how related two things are |
| The measurement's real name, out in the wild | Cosine similarity — same idea as similarity, different name |
| About meaning, not exact spelling | Semantic |
| A neighbourhood of addresses that share a topic | A domain — a cluster of related meaning |
| Scanning every address for whichever one sits closest to a new point | Nearest neighbor — the closest real match to a computed point |
| A small, separate machine that only plots words as points — it never writes a sentence | An embedding model — not the same as a language model |
| The total number of coloured patches, across every board | Parameters — the number people quote |
| How many tokens the funnel can hold in one pour | Input tokens — the context window |
| How many tokens come back out | Output tokens — what it writes back |
| Everything you've met so far, built at real-world size | A Large Language Model — an LLM |
| A shorter pile winning, extended forward with full confidence | A hallucination |
| The point where the training reading stopped | The knowledge cutoff |
| How sharp or blended the colour stays before a coin is poured | Temperature — the dial on how often a stray pick wins |
| Pouring the same slot twice, not always the same pile | Non-deterministic — same input, no guaranteed identical output |
| Background, examples, or a role, added right into the pour | Context |
| Asking for many small steps instead of one big leap | Chain-of-thought prompting |
| A friendly chat window built around the board | A product — like ChatGPT or Claude.ai |
| A door that lets software pour in and read out directly | An API |
| What each million tokens costs to send in or get back | API pricing — pricier for bigger models, and for output |
| Recalling the start of the line instead of re-pouring it | Cached tokens — priced far cheaper than fresh input |
| The board asking the app to go do something real before it answers | Tool use — or function calling |
| One shared shape for listing tools and calling them | MCP — Model Context Protocol |
| A program that speaks that shape and hands back its whole toolbox | An MCP server |
| The loop deciding, acting, and watching what happened, on repeat | An agent — a model with tools, looping until the goal is met |
| Deciding its own steps, and knowing when it's done | Autonomy |
| Reading what just happened before deciding what's next | Reactivity |
| Starting on its own, from a schedule or an event | Pro-activeness |
| Talking to other agents, and to people, not just tools | Social ability |
| Search first, hand the model what you found, then let it answer | RAG — Retrieval-Augmented Generation |
| Folding the search step into the loop, so it can search again if needed | Agentic RAG |
| + New in this lesson | |
| The first, giant reading pass, before the floor is fine-tuned at all | Pretraining |
| A second, much smaller painting pass, aimed at a narrower target | Post-training — or fine-tuning |
| A person ranking draft answers, best to worst | RLHF — Reinforcement Learning from Human Feedback |
| A small helper trained to predict which answer people prefer | A reward model |
| Making a model's behavior actually match what people want | Alignment |
51 rows now. It keeps growing as the course goes on.