AI Learner #7: KV Cache — The Speed Trick Behind Fast Generation

Here’s a dirty secret about how models generate text: by default, they redo a staggering amount of work. Every single new token, they re-read the entire conversation from scratch.

Imagine writing a novel where, to add each new word, you first re-read every word you’ve written so far. By page 300 you’d be exhausted. That’s a model without a cache.

Without a cache, every new token re-reads all the previous ones; with a cache, you reuse stored keys and values

The Solution: Cache the Keys and Values

The fix is the KV cache. For every token, attention already computes a key and a value — so why throw them away? Store them, and you only ever compute the brand-new token. (Better analogy than a novel: a restaurant order pad. The kitchen doesn’t re-ask every table what they ordered — they wrote it down once. The cache is that pad.)

Prefill vs. Decode: Two Phases

This splits inference into two phases. Prefill reads the whole prompt at once and fills the cache — fast and parallel. Decode generates one token at a time, appending each new key and value — the slow part, one token after another, each leaning on the cache below it.

Prefill fills the cache from the prompt; decode appends one token at a time

The Memory Cost: Cache Gets Big, Fast

There’s a catch, naturally. The cache grows with every token, and for long contexts it can balloon to many gigabytes — sometimes bigger than the model’s own weights.

KV cache memory grows with sequence length

Modern Tricks: Shrink the Cache

So modern models shrink it. Instead of giving every attention head its own keys and values, grouped-query attention (GQA) lets groups of heads share — same idea, a fraction of the memory. (Its extreme cousin, multi-query attention, shares one set across all heads.)

Grouped-query attention shares key/value heads to shrink the cache

Why This Matters

The KV cache is why a chatbot replies at a steady clip instead of slowing to a crawl. It trades a chunk of memory for an enormous amount of speed — and it’s why the first token takes a moment (prefill) but the rest stream out fast (decode).

What Comes Next

We’ve made generation fast. But these models are still enormous. How do you fit a 70-billion-parameter brain onto hardware it has no business running on?

Coming up: quantization — making models smaller without making them dumb.

Quick Quiz 🧠

1. What does the KV cache actually store?

Answer: The key and value vectors computed for each token during attention, so they can be reused instead of recomputed for every new token.

2. What’s the difference between prefill and decode?

Answer: Prefill processes the entire prompt at once to populate the cache (fast, parallel). Decode generates one token at a time, appending to the cache (slower, sequential).

3. How does grouped-query attention help?

Answer: It lets multiple query heads share the same key/value heads, drastically shrinking the KV cache memory with little quality loss.

Source: KV Caching Explained (Hugging Face), Coding the KV Cache in LLMs (Sebastian Raschka), Grouped-Query Attention (IBM)