Learn · Speed Hacks · KV Cache
Your Model's Cheat Sheet
In plain English
Instead of re-reading a whole book to answer the second question, the AI writes down sticky notes (KV Cache) so it can answer instantly! 📝
What is actually going on
The KV (Key-Value) Cache stores the output of the attention mechanism for past tokens. During decode, the model only computes attention for the new token against the cached keys and values, turning an O(N²) operation into O(N).
Why it matters
Without KV caching, generating the 1000th token would require re-computing attention for tokens 1 through 999 every single time. Inference would grind to a halt.
Another way to think about it
Leaving your math scratchpad on the desk instead of throwing it away after every problem. 🧮
Check yourself
What does the KV Cache store?
- Web search results
- Keys and Values from the attention layers
- The entire training dataset
- Tokenized text files
Show answer
Keys and Values from the attention layers — It caches the intermediate tensor calculations from the transformer's attention mechanism.
Without a KV Cache, generating long text would become extremely slow.
- True
- False
Show answer
True — The model would have to recalculate the entire prompt and all generated tokens for every new word.
KV Cache trades off memory for ___.
- Speed
- Accuracy
- Privacy
- Model size
Show answer
Speed — By utilizing more RAM to store the cache, it dramatically speeds up the decode phase.