Learn · The Inference Pipeline · How Generation Works
From Prompt to Response
In plain English
When you talk to the AI, it reads your whole message, thinks for a second, and then spits out the answer one word at a time. ⏱️
What is actually going on
The lifecycle of an inference request involves tokenizing the prompt, passing it through the transformer in a 'prefill' phase to generate the first token, and then auto-regressively 'decoding' subsequent tokens.
Why it matters
Understanding this lifecycle allows developers to optimize the right part of the stack—whether it's latency (Time To First Token) or throughput (Tokens/s).
Another way to think about it
It's like reading the whole test question (prefill), then writing the essay word-by-word (decode). 📝
Check yourself
What is the very first step when you send text to an LLM?
- Decoding
- Tokenization
- KV Caching
- Printing
Show answer
Tokenization — Raw text must first be converted into integer token IDs before the model can process it.
The model generates the entire response instantly in one step.
- True
- False
Show answer
False — LLMs are autoregressive, meaning they generate responses one token at a time.
What are the two main phases of inference?
- Training and Testing
- Prefill and Decode
- Loading and Unloading
- Prompting and Fine-tuning
Show answer
Prefill and Decode — Prefill processes the prompt; Decode generates the new tokens.