AI Learner #5: Inference & Decoding — How Models Write Text
The model spits out a probability distribution for every word. But how does it pick the next one? Temperature, top-k, top-p — the knobs that turn raw math into readable prose.
AI Learner #5: Inference & Decoding — How Models Write Text
Your model has done the hard part (Parts 1–4): it read your prompt and produced a score for every possible next token — thousands of them. Now it has to actually pick one. That picking is called decoding, and it’s surprisingly gambly.
From Raw Scores to a Word
Those raw scores are logits. A function called softmax squashes them into clean probabilities that add up to 1. Then the model chooses.
Greedy vs. Sampling
The simplest strategy is greedy: always grab the highest-probability token. Correct, safe, and about as much fun as a tax form.
The alternative is sampling: roll weighted dice, so a likely token usually wins but the occasional surprise sneaks through. That’s where personality comes from.
Temperature: The Sharpness Knob
How wild those dice get is set by temperature. Low temperature sharpens the favorite; high temperature flattens everything and invites chaos. At T=0.5, the top word “mat” grabs 76%. Crank it to 1.5 and that drops to 36%, with the long shots suddenly in the running.
Top-K and Top-P: Trimming the Nonsense
Pure temperature can let in genuinely terrible tokens, so we add filters. Top-k keeps only the k most likely candidates. Top-p (nucleus sampling) is smarter: it keeps just enough tokens to cover, say, 90% of the probability — few options when the model is confident, more when it’s unsure.
The crowd-favorite combo is a little temperature plus top-p: shape the distribution, then trim the garbage. Most chat models run something like this by default.
Repetition Penalties
There’s one more gremlin: repetition. Left alone, a model can get stuck saying “the the the the.” A repetition penalty quietly lowers the odds of words it just used, so it stops looping.
Why This Matters to You
When you drag that “creativity” or “temperature” slider in an AI app, this is what you’re touching — the dials between boring and unhinged. Decoding is how raw probability becomes the sentence you actually read. The model isn’t reciting anything; it’s choosing, one token at a time.
What Comes Next
Coming up: context windows — and why a model’s memory is both bigger and far more fragile than you’d think.
Quick Quiz 🧠
1. What does temperature do?
Answer: It scales the sharpness of the probability distribution before sampling. Low temperature makes the top tokens even more likely (focused); high temperature flattens the distribution (more random).
2. What’s the difference between top-k and top-p?
Answer: Top-k always keeps a fixed number of candidate tokens. Top-p keeps a variable number — just enough to cover a target probability mass — so it adapts to how confident the model is.
3. Why do we need repetition penalties?
Answer: Without them, models can fall into loops, repeating the same high-probability token. The penalty lowers the odds of recently-used tokens to keep output varied.
Source: Decoding Strategies for LLMs, LLM Sampling: Temperature, Top-K, Top-P, Setting Top-K, Top-P and Temperature
Watch the full lesson