KV Cache
Cached Key-Value tensors that speed up token generation
Performance
In plain English
Your AI's sticky notes. Instead of re-reading a whole book every time you ask a question, it writes down summaries so it can answer the next question faster.
Technically
During autoregressive generation, the Keys (K) and Values (V) from the transformer's attention mechanism are cached for past tokens. This prevents recomputing them for every new token, converting an O(N^2) operation into O(N) per step at the cost of memory: Bytes = 2 × N_layers × N_heads × D_head × precision.