Learn · Popular Open Source Models · Meta Llama & Mistral Foundations
Mistral NeMo & Mixtral MoE
In plain English
Mistral from France built the pioneer Mixture-of-Experts models. Their 12B NeMo and Mixtral 8x7B models deliver extreme efficiency with their high-speed Tekken tokenizer! 🥐⚡
What is actually going on
Mistral NeMo (12B) and Mixtral 8x7B/8x22B utilize the Tekken tokenizer (131k vocab size for high compression) and sparse top-2 expert routing. Mixtral 8x7B carries 46.7B total parameters but activates only 12.8B per token, running comfortably in ~28GB RAM (Q4_K_M).
Why it matters
Mistral pioneered practical sparse MoE for consumer local inference, setting the architectural foundation for subsequent models like DeepSeek-V3.
Another way to think about it
A fast French bullet train that switches tracks dynamically to pick up speed. 🚄
Check yourself
How many active parameters are computed per token in Mixtral 8x7B (46.7B total)?
- 7B
- 12.8B
- 46.7B
- 80B
Show answer
12.8B — Mixtral 8x7B activates 2 of its 8 experts per token, evaluating ~12.8B active parameters.