Multi-Head Latent Attention (MLA)
Low-rank latent KV compression pioneered by DeepSeek
Architecture
In plain English
Instead of saving 100 pages of detailed notes per question, the AI compresses them into a 5-page secret code that it unpacks in a microsecond, slashing memory needs by 93%.
Technically
A low-rank attention compression architecture introduced in DeepSeek-V2/V3/R1. Compresses Keys and Values into a single low-rank latent vector c_t^{KV} (e.g. 512 dimensions) plus a decoupled rotary vector (64 dimensions), reducing per-token KV cache memory footprint by >90% compared to Multi-Head Attention.