Learn · Foundations · The Three Axes
Bandwidth — How Fast
In plain English
Imagine drinking a milkshake through a tiny straw. 🥤 It doesn't matter how fast you can swallow; the straw slows you down. That's memory bandwidth!
What is actually going on
Memory bandwidth (measured in GB/s) is the speed at which data can be transferred from RAM to the processor cores. During generation (decoding), the entire model weight matrix must be loaded for every single token.
Why it matters
For LLMs, memory bandwidth is almost always the primary bottleneck for generation speed, not compute power.
Another way to think about it
A Ferrari (CPU/GPU) stuck on a dirt road (low bandwidth) goes no faster than a bicycle. 🏎️
Check yourself
What does memory bandwidth measure?
- Size of the RAM
- Speed of data transfer from RAM to processor
- Number of CPU cores
- Internet download speed
Show answer
Speed of data transfer from RAM to processor — Bandwidth is the 'width of the pipe' connecting memory to the compute cores.
During generation, LLMs are usually bottlenecked by:
- Compute power (TFLOPs)
- Storage space
- Memory Bandwidth
- Network latency
Show answer
Memory Bandwidth — Because generation is auto-regressive, it requires constant memory reads per token.
Increasing CPU cores always increases generation speed linearly.
- True
- False
Show answer
False — More cores don't help if they are starved for data due to limited memory bandwidth.