Learn · NVIDIA Hardware & DGX · NVFP4: Blackwell's 4-Bit Language
NVFP4 Explained
In plain English
Quantization is squishing AI memory into smaller boxes. NVFP4 is NVIDIA's special tiny box — only 4 bits per number! — with a clever trick: every little group of numbers gets its own mini ruler (scale), so the squish barely loses any smarts. Blackwell chips have hardware that reads these tiny boxes for free, at full speed. 📦
What is actually going on
NVFP4 is NVIDIA's 4-bit floating-point format (E2M1 elements) aligned with the OCP MX microscaling spec. Weights are stored in 4 bits with shared E4M3 scale factors per small block (16-element microblocks). Blackwell 5th-gen Tensor Cores dequantize and apply these scales natively during dot products — no software penalty. Result: roughly half the memory of INT8/FP8, double the tensor-core throughput, and far less accuracy loss than naive INT4 because per-block floating-point scales preserve dynamic range.
Why it matters
NVFP4 is what makes frontier-scale models realistic on deskside hardware. It's how the DGX Spark claims ~1 PetaFLOP and runs 200B models in 128GB, and how DGX Station serves models that would otherwise need a datacenter. NVFP4 checkpoints (e.g. Nemotron, Llama variants) are becoming a standard distribution format.
Another way to think about it
Photocopying a book in tiny 4-bit print — but each page gets its own magnifying-glass ruler so you can still read everything. The book is 4x smaller; comprehension barely drops. 📚
Check yourself
What is the key trick that makes NVFP4 more accurate than plain INT4?
- Bigger weights
- Per-block floating-point scale factors
- It stores 8 bits secretly
- It skips hard numbers
Show answer
Per-block floating-point scale factors — Each microblock of 4-bit values gets its own E4M3 scale, preserving dynamic range where fixed-scale INT4 loses precision.
Where does NVFP4 decode happen with zero software overhead?
- In system RAM
- On the CPU
- Inside Blackwell 5th-gen Tensor Cores
- On the network card
Show answer
Inside Blackwell 5th-gen Tensor Cores — Blackwell Tensor Cores fetch packed FP4 plus scales and apply scaling during accumulation, natively in hardware.
NVFP4 roughly halves the memory footprint of FP8/INT8 weights.
- True
- False
Show answer
True — 4 bits vs 8 bits: model weights shrink ~2x versus FP8/INT8, with small accuracy loss thanks to microscaling.