Memory Bandwidth
How fast your hardware can read model weights — the real speed limit on local inference
Performance
In plain English
Imagine reading a whole book out loud every time someone asks you a question. It doesn't matter how clever you are; what matters is how fast you can flip the pages. Memory bandwidth is your page-flipping speed.
Technically
The rate at which data moves between memory and the compute units, measured in GB/s. Decoding one token requires reading every active weight from memory, so single-stream generation is bandwidth-bound rather than compute-bound: tokens/sec is approximately bandwidth divided by the bytes of active weights. An RTX 5090 at 1792 GB/s reading a 19 GB Q4 model has a hard ceiling near 94 tok/s before any software overhead. This is why a Mac Studio with 512 GB of slow unified memory can hold a model an RTX 5090 cannot, yet generate fewer tokens per second on a model that fits both.