Speculative Decoding
Using a small model to draft answers and a big model to check them
Performance
In plain English
Like a boss (big model) letting an intern (small model) write a rough draft. If the draft is good, the boss approves it instantly. If not, the boss fixes it.
Technically
An optimization technique where a smaller, faster draft model predicts γ future tokens. The larger target model verifies them in parallel in a single forward pass via rejection sampling (α = min(1, p(x)/q(x))), achieving 2x-3.2x latency improvements without altering output probability distribution.