AI Learner #6: Context Windows — How Models Remember (and Forget)
LLMs read numbers, not words — but what about memory? Context windows are the model's working memory, and they're far more limited (and fragile) than you'd think.
AI Learner #6: Context Windows — How Models Remember (and Forget)
Every model has a memory limit called its context window. Everything you say, everything it says back, your pasted documents — all of it has to fit in one fixed budget of tokens (and tokens, remember, aren’t words).
Think of it like a cocktail party. You can follow a few conversations at once, but pack the room and the voices blur. The window is exactly that big, and exactly that hard-capped.
What Fits Inside the Window?
So what’s actually in there? The system prompt, the whole conversation so far, your new message, and — crucially — the room left over for the reply. Fill it to the brim and the answer gets squeezed.
Why the Window Exists: The Attention Tax
Why even have a limit? Because attention compares every token to every other token. Double the context and the work roughly quadruples. That quadratic cost is the attention tax, and it’s why a bigger window is so expensive.
“Lost in the Middle”
Bigger windows hide a nasty quirk: lost in the middle. Models reliably recall the start and the end of a long context, and quietly skim everything in between. It’s why burying your key instruction halfway through a giant prompt is a great way to watch the model serenely ignore it.
What Happens When You Overflow
When you do overflow the window, it slides. New tokens push in, the oldest fall off the edge, and the model simply forgets they ever existed. There’s no “out of memory” error — just quiet amnesia.
Why Bigger Isn’t Always Better
A huge window costs more, runs slower, and still gets distracted. Often a tight, well-organized prompt beats a giant sloppy one. Context is the model’s working memory: fixed, finite, and weirdly forgetful in the middle. Manage it well and the model feels sharp; ignore it and it acts like it has amnesia.
What Comes Next
All this attention comparing every token to every other token, every single step, sounds painfully slow. It would be — if not for one clever trick.
Coming up: the KV cache — how models avoid redoing the same work over and over.
Quick Quiz 🧠
1. What lives inside the context window?
Answer: The system prompt, the full conversation history, your current message, and the space reserved for the model’s reply — all sharing one fixed token budget.
2. What is “lost in the middle”?
Answer: A tendency for models to recall information at the beginning and end of a long context well, while paying less attention to material in the middle.
3. Why is a bigger context window so expensive?
Answer: Attention compares every token with every other token, so cost grows roughly with the square of the context length — double the tokens, quadruple the work.
Source: What is a Context Window? (IBM), Lost in the Middle (Liu et al.), Context Windows Explained
Watch the full lesson