Learn · The Inference Pipeline · The Two Phases
Prefill — Reading Your Input
In plain English
The Prefill phase is the AI speed-reading your prompt all at once! 📚 It builds a mental map of what you asked.
What is actually going on
During prefill, the model processes all input tokens in parallel. This is highly compute-bound (matrix-matrix multiplications) and generates the initial KV cache state.
Why it matters
A slow prefill results in high latency before the user sees any response (bad TTFT).
Another way to think about it
Reading the instructions before assembling furniture. 🪑
Check yourself
What happens during the Prefill phase?
- The model generates the final token
- The model processes the entire input prompt
- The model downloads from the cloud
- The model compresses weights
Show answer
The model processes the entire input prompt — Prefill involves reading and processing the user's prompt to establish context.
The Prefill phase processes tokens one-by-one.
- True
- False
Show answer
False — Unlike decode, prefill processes input tokens in parallel.
The prefill phase is typically bottlenecked by:
- Compute (FLOPS)
- Memory Bandwidth
- Storage speed
- Network ping
Show answer
Compute (FLOPS) — Because it processes a large matrix of input tokens at once, it utilizes the GPU's compute cores heavily.