What Is a Tokenizer? Why the Same Sentence Costs More in Russian Than in English
In the last article I said a model can't read words: "Hola mundo" becomes token ids, then vectors of numbers. I skipped the first step. Who decides what a token is? That's the tokenizer, and it turns out the answer depends a lot on which language you write in.
What a tokenizer does
A tokenizer has a fixed vocabulary: a list of text pieces, each with a number. To read your text, it splits the text into pieces from that list and outputs their numbers:
"The children went to school" → The | ␣children | ␣went | ␣to | ␣school → [976, 3953, 5981, 316, 3474]
(Those are GPT-4o's ids; every tokenizer numbers its pieces differently.) The model only ever sees the numbers. Each number picks one row of the model's embedding table, which is a tensor of shape (vocabulary size, hidden size). For Qwen3.5-2B that's 248,320 × 2,048: about 509 million numbers, roughly a quarter of the whole model, just to give every token its vector.
Where the pieces come from: byte-pair encoding
Most modern tokenizers learn their vocabulary with byte-pair encoding (BPE). The idea fits in three lines:
- Start with every word split into single characters (really, single bytes).
- Count which pair of neighbouring pieces appears most often in the training text, and merge it into one new piece.
- Repeat until the vocabulary is as big as you want: 50,000 pieces, 200,000, 250,000.
Try it in part 1 of the explainer below, on a tiny Spanish corpus: lluvia, llueve, llover, lloviendo. The first merge is l + l → ll, because "ll" appears 14 times. Then ll + u → llu. After a few merges, frequent chunks of Spanish have become single pieces.
That's the whole trick, and it's also the whole problem: the tokenizer learns the pieces that were frequent in its training text. If that text was mostly English, English words become single tokens and everything else gets chopped into small, expensive pieces.
The experiment: one meaning, four languages, four tokenizers
I wrote three ordinary sentences and translated them into Spanish, Russian and Chinese (my own translations):
| Sentence 1 | |
|---|---|
| English | The children went to school early because it was raining. |
| Spanish | Los niños fueron temprano a la escuela porque estaba lloviendo. |
| Russian | Дети рано пошли в школу, потому что шёл дождь. |
| Chinese | 因为下雨，孩子们很早就去上学了。 |
Then I ran all three sentences, in all four languages, through four real tokenizers:
| Tokenizer | Vocabulary | Used by |
|---|---|---|
| GPT-2 (2019) | 50,257 | the classic English-first BPE |
| o200k | 200,019 | GPT-4o |
| Qwen3.5 | 248,077 | Qwen3.5 models |
| Gemma 4 | 262,144 | Gemma 4 models |
Total tokens for the three sentences (↓ fewer is better), and the ratio to English:
| Tokenizer | English | Spanish | Russian | Chinese |
|---|---|---|---|---|
| GPT-2 | 34 | 63 (1.85×) | 185 (5.44×) | 110 (3.24×) |
| GPT-4o (o200k) | 34 | 38 (1.12×) | 47 (1.38×) | 42 (1.24×) |
| Qwen3.5 | 34 | 38 (1.12×) | 44 (1.29×) | 28 (0.82×) |
| Gemma 4 | 34 | 38 (1.12×) | 43 (1.26×) | 33 (0.97×) |
All four tokenizers agree on English: 34 tokens. They disagree wildly on everything else. Part 2 of the explainer shows every split, and part 3 draws this table as bars.
English: the home language
Every tokenizer splits the English sentence the same way, one token per word plus the full stop:
The | ␣children | ␣went | ␣to | ␣school | ␣early | ␣because | ␣it | ␣was | ␣raining | .
English is the language these vocabularies were built around, so common English words are single pieces. Even GPT-2, from 2019, handles it perfectly.
Spanish: close, but accents cost
Spanish shares the Latin alphabet with English, so modern tokenizers do well: about 12% more tokens than English with GPT-4o, Qwen3.5 and Gemma 4. Some of that is just that Spanish sentences are longer (197 characters vs 170 for the same three sentences).
GPT-2 is another story: 1.85×. It never learned ñ as part of a word:
GPT-2: Los | ␣ni | ñ | os | ␣fu | er | on | ␣tem | pr | ano | ...
GPT-4o: Los | ␣niños | ␣fueron | ␣temprano | ...
And even the modern ones stumble on less common words. "lloviendo" is ␣l | lo | viendo in Qwen3.5 and ␣ll | ov | iendo in GPT-4o. The accented í in "increíble" becomes its own token in three of the four tokenizers: incre | í | ble.
Russian: the alphabet tax
Russian is where the old tokenizer falls apart. Cyrillic letters take 2 bytes each in UTF-8, and GPT-2's vocabulary barely contains Cyrillic, so it falls back to bytes. The first Russian sentence alone costs 52 tokens in GPT-2. Many letters are split into two byte-tokens, half a letter each, shown in dashed red in the explainer. The model has to reassemble "Д" from two meaningless fragments.
The same sentence in the modern tokenizers:
Qwen3.5: Д | ети | ␣рано | ␣пошли | ␣в | ␣школу | , | ␣потому | ␣что | ␣ш | ёл | ␣до | жд | ь | .
That's 15 tokens, close to English's 11. But Russian is still the most expensive of the four languages in every modern tokenizer: 26–38% more tokens than English. Russian words change their endings a lot (школа, школу, школе, школы), and each form competes for a place in the vocabulary.
Chinese: no spaces, and the biggest surprise
Chinese has no spaces between words and each character takes 3 bytes in UTF-8. For GPT-2 that's a disaster: 110 tokens for three short sentences, 3.24× English, with most characters cut into 2–3 byte fragments. The word 难以置信 ("unbelievable") is 9 tokens.
But look at Qwen3.5, a model trained with a lot of Chinese text:
Qwen3.5: 因为 | 下雨 | ， | 孩子们 | 很早就 | 去 | 上学 | 了 | 。
Nine tokens, and many of them are whole words or even phrases: 孩子们 ("the children"), 很早就 ("very early"). 难以置信 is one token. Over the three sentences, Qwen3.5 needs 28 tokens for Chinese and 34 for English: Chinese is cheaper than English. GPT-4o needs 42 for the same Chinese.
That's the clearest proof of the rule: a tokenizer is cheap in the languages it was trained on. Qwen's vocabulary was built from a lot of Chinese, so Chinese gets big, efficient pieces.
Why this matters
More tokens for the same meaning isn't a cosmetic detail:
- Cost. APIs charge per token. A Russian user of a GPT-2-style tokenizer would pay 5.4× more than an English user for the same request; with GPT-4o, about 38% more.
- Speed. Models generate a fixed number of tokens per second, so more tokens means a slower answer.
- Context window. A 128k-token window holds less Russian than English.
- Quality. A model that sees words as byte fragments has to work harder to understand them.
Researchers have measured this across many languages: Petrov et al. found that the same text translated into different languages can differ in tokenized length by up to 15 times (Language Model Tokenizers Introduce Unfairness Between Languages, 2023).
The good news, from my small test: the gap is shrinking. Vocabularies went from 50k pieces to 200k–262k, and Spanish, Russian and Chinese all got dramatically cheaper.
Measure your own text
Don't trust my three sentences for your workload; count your own. This is the exact code I used (Python, with tiktoken, tokenizers and huggingface_hub installed):
import tiktoken
from huggingface_hub import hf_hub_download
from tokenizers import Tokenizer
gpt4o = tiktoken.get_encoding("o200k_base")
qwen = Tokenizer.from_file(hf_hub_download("mlx-community/Qwen3.5-2B-4bit", "tokenizer.json"))
for text in ["The children went to school early because it was raining.",
"Дети рано пошли в школу, потому что шёл дождь.",
"因为下雨，孩子们很早就去上学了。"]:
print(len(gpt4o.encode(text)), len(qwen.encode(text).ids), text)
It prints 11 11, 16 15 and 14 9: GPT-4o and Qwen3.5 token counts for each sentence.
How much to trust these numbers
- Three sentences per language, translated by me. Real text will give different ratios; the direction (old tokenizers punish non-English, modern ones much less, Qwen favours Chinese) is the robust part.
- Token counts, not quality. This measures how text is chopped, not how well each model understands it.
The one-line mental model
A tokenizer is a dictionary of text pieces learned from its training data. Languages it saw a lot get big, cheap pieces; languages it rarely saw get chopped into small, expensive ones, sometimes down to half a letter.