Learn · Foundations · Your First Mental Model
Tokens — The AI's Alphabet
In plain English
AI doesn't read words like we do. It chops words into Lego blocks called 'tokens'. 🧱 'Apple' might be one token, but 'Supercalifragilistic' might be five!
What is actually going on
A token is the fundamental unit of data processed by an LLM, mapped to an integer ID. Using Byte-Pair Encoding (BPE), models compress text into dense integer arrays before processing.
Why it matters
Token limits dictate how much text a model can process at once (context window) and affect API pricing and generation speed.
Another way to think about it
Tokens are the syllables of the AI language. 🗣️
Check yourself
What is a token?
- A whole sentence
- A chunk of text or part of a word mapped to an ID
- An image pixel
- A type of memory
Show answer
A chunk of text or part of a word mapped to an ID — Tokens are sub-word units that the AI actually processes.
One word always equals exactly one token.
- True
- False
Show answer
False — Long or uncommon words are often split into multiple sub-word tokens.
What is the common algorithm used to create these tokens?
- Byte-Pair Encoding (BPE)
- Bubble Sort
- Gradient Descent
- AES-256
Show answer
Byte-Pair Encoding (BPE) — BPE iteratively merges common pairs of characters into single tokens.