Learn · The Inference Pipeline · Measuring Performance
Tokens Per Second — Speed Matters
In plain English
Tokens per second is how fast the AI can type. If it's slower than you can read, it gets really annoying! 🏃💨
What is actually going on
Throughput during the decode phase is measured in tokens/s. It is calculated by dividing the total generated tokens by the total decode time.
Why it matters
Humans read at about 5-10 tokens/s. A local model should generate at least 20-30 tokens/s for a comfortable, snappy user experience.
Another way to think about it
It's the speedometer on the AI's dashboard. 🏎️
Check yourself
What is a good target tokens/s rate for a comfortable reading experience?
- 1-2 tok/s
- 5-10 tok/s
- > 30 tok/s
- 1000 tok/s
Show answer
> 30 tok/s — While humans read at ~10 tok/s, 30+ tok/s ensures the text appears instantly and doesn't hold up the user.
Tokens per second measures the speed of the prefill phase.
- True
- False
Show answer
False — Tokens per second is the metric used for the decode phase (generation).
If an AI generates 150 tokens in 5 seconds, what is its speed?
- 5 tok/s
- 30 tok/s
- 150 tok/s
- 750 tok/s
Show answer
30 tok/s — 150 / 5 = 30 tokens per second.