Why My Chinese Reports Skipped Records: Alphabets, Case and Tokenization
In the first article I said a model only sees numbers. In the second I showed how a tokenizer chops text into pieces, and why Russian and Chinese can cost more than English. This one goes one level lower: before the tokenizer, every character is already a number. Which number depends on the alphabet, and even on upper or lower case.
I learned this the hard way, long before anyone talked about tokens.
The reports that skipped records
About 20 years ago I worked on a Sage Accpac implementation in Shenzhen, China. My part was printing reports. One report had to list all the items, and the way we asked for "all items" was a range filter:
item number >= "" AND item number <= "ZZZZZZ"
From the empty string up to "ZZZZZZ": that looks like everything. It wasn't. The reports skipped every record whose item had Chinese characters, and I had to figure out why.
The answer is that the computer doesn't compare letters. It compares numbers.
Every character is a number
Each character has a number called its code point. Here are a few, with their UTF-8 bytes (Python 3.14):
for ch in ["A", "a", "Ñ", "ñ", "А", "а", "电"]:
print(ch, ord(ch), hex(ord(ch)), ch.encode("utf-8").hex(" "))
A 65 0x41 41
a 97 0x61 61
Ñ 209 0xd1 c3 91
ñ 241 0xf1 c3 b1
А 1040 0x410 d0 90
а 1072 0x430 d0 b0
电 30005 0x7535 e7 94 b5
(The fifth and sixth lines are Cyrillic А and а, not Latin. They look the same and they are different numbers.)
Upper and lower case are different numbers
| Upper | Code | Lower | Code | Difference |
|---|---|---|---|---|
| A | 65 (0x41) | a | 97 (0x61) | +32 |
| Z | 90 (0x5A) | z | 122 (0x7A) | +32 |
| Ñ | 209 (U+00D1) | ñ | 241 (U+00F1) | +32 |
| А (Cyrillic) | 1040 (U+0410) | а | 1072 (U+0430) | +32 |
| 电 | 30005 (U+7535) | no case |
In ASCII, upper and lower case differ by exactly 32, which is one bit (0x20). That's why old code could change case by flipping a single bit. Unicode kept the same +32 for Ñ/ñ and for most Russian letters, although it's not always one bit there (Я/я are 0x42F and 0x44F), and Ё/ё break the pattern entirely: they're 80 apart.
Chinese has no upper and lower case at all. 电 is 电.
Alphabets are different sizes
- English: 26 letters.
- Spanish: 27. The extra one is ñ. The Real Academia Española's 2010 Ortografía removed ch and ll from the alphabet: they are digraphs, two letters for one sound, not letters of their own (RAE). Accented vowels (á, é, í, ó, ú) and ü are not separate letters either, but each one is a separate code point.
- Russian: 33 letters.
- Chinese: no alphabet. Every word is written with characters, and there are a lot of them. China's 2013 Table of General Standard Chinese Characters lists 8,105, of which 3,500 are the common level-one set taught in basic education. Unicode goes much further: Python 3.14 (Unicode 16.0) knows 97,668 CJK unified ideographs.
import unicodedata
print(unicodedata.unidata_version, sum(1 for c in range(0x110000)
if unicodedata.name(chr(c), "").startswith("CJK UNIFIED IDEOGRAPH")))
16.0.0 97668
More characters means bigger numbers, and bigger numbers need more bytes. In UTF-8, English letters take 1 byte, ñ, á and Cyrillic take 2, and most Chinese characters take 3.
Why "ZZZZZZ" isn't the highest value
Back to Shenzhen. When a computer compares two strings in a plain binary comparison, it compares them number by number, from the first character. "ZZZZZZ" starts with Z, which is 0x5A. Anything whose first character is a bigger number than 0x5A sorts after "ZZZZZZ", so <= "ZZZZZZ" throws it out.
Every Chinese character is far above 0x5A. In UTF-8 the first byte of 电 is 0xE7. And in the Chinese encodings systems used back then, GBK and GB2312, every Chinese character starts with a lead byte between 0x81 and 0xFE, also above 0x5A (电 is B5 E7 in GBK). Whichever encoding you use, the Chinese items fall outside the range.
Here's the same filter in Python, which compares strings by code point:
items = ["ABC-100", "abc-200", "NIÑO-1", "ZZZ-1", "电脑-01", "Z-999", "ÁRBOL-7"]
print("printed:", [i for i in items if "" <= i <= "ZZZZZZ"])
print("skipped:", [i for i in items if not ("" <= i <= "ZZZZZZ")])
printed: ['ABC-100', 'NIÑO-1', 'ZZZ-1', 'Z-999']
skipped: ['abc-200', '电脑-01', 'ÁRBOL-7']
The Chinese item is gone, as in my reports. Two more surprises:
- Lowercase codes are skipped too. 'a' is 97 and 'Z' is 90, so "abc-200" sorts after "ZZZZZZ" in a binary comparison. Many databases use case-insensitive collations by default, which hides this, until one day a column or a server uses a binary one.
- Spanish can be skipped as well. "ÁRBOL-7" starts with Á (0xC1), above Z. "NIÑO-1" survives only because it starts with N; the Ñ comes later and is never compared.
The fix is not a bigger fake value. "ZZZZZZ" was a guess at "the highest possible string", and there isn't one you can safely type. When you mean "all items", leave the upper bound out (or drop the filter). When you really need a range, compare with a collation that matches the language of the data, and test it with data that isn't English. Part 2 of the explainer lets you toggle the upper bound and add your own item codes.
What this has to do with tokenization
A tokenizer reads the same numbers. Most modern tokenizers work on UTF-8 bytes and merge frequent byte sequences into tokens; the others fall back to raw bytes for any character they don't know. So the alphabet and the case of your text change what the model receives.
I ran a few words through the same four tokenizers as last time (token count; GPT-2 / GPT-4o / Qwen3.5 / Gemma 4):
| Text | GPT-2 | GPT-4o | Qwen3.5 | Gemma 4 |
|---|---|---|---|---|
| hello | 1 | 1 | 1 | 1 |
| HELLO | 3 | 2 | 2 | 1 |
| niño | 2 | 2 | 2 | 2 |
| NIÑO | 4 | 2 | 2 | 2 |
| año | 2 | 2 | 2 | 1 |
| ano | 1 | 1 | 1 | 1 |
| школа ("school") | 6 | 3 | 2 | 2 |
| ШКОЛА | 10 | 4 | 3 | 3 |
| 电脑 ("computer") | 5 | 1 | 1 | 1 |
And the splits behind some of those numbers (angle brackets are raw bytes, pieces of one character):
GPT-2 HELLO HE | LL | O
GPT-2 NIÑO NI | <c3> | <91> | O
GPT-2 ШКОЛА <d0> | <a8> | <d0> | <9a> | <d0> | <9e> | <d0> | <9b> | <d0> | <90>
GPT-4o ШКОЛА Ш | К | ОЛ | А
Qwen3.5 школа ш | кола
Gemma 4 HELLO HELLO
What I take from it:
- "hello", "Hello" and "HELLO" are unrelated numbers to the model. In GPT-4o they are token 24912, token 13225, and the pair 111642 + 2699. The tidy +32 between 'a' and 'A' doesn't survive tokenization; the model has to learn from data that they mean the same thing.
- Capitals cost more, because ALL-CAPS text is rarer in training data, so fewer upper-case pieces made it into the vocabulary. One sentence per language, as written and then in ALL CAPS:
| Tokenizer | English | Spanish | Russian |
|---|---|---|---|
| GPT-2 | 11 → 20 | 23 → 33 | 52 → 74 |
| GPT-4o | 11 → 17 | 13 → 24 | 16 → 30 |
| Qwen3.5 | 11 → 18 | 13 → 24 | 15 → 24 |
| Gemma 4 | 11 → 13 | 13 → 23 | 15 → 28 |
(The sentences are the first ones from the tokenizer article.) Shouting nearly doubles the cost of Spanish and Russian in GPT-4o.
- "año" and "ano" are different words (one means "year"; the other is a word you don't want on a birthday card), and to the tokenizer the ñ is a different 2-byte character. GPT-2 splits ñ off into its own token; in NIÑO it can't even keep Ñ whole and falls back to its two raw bytes.
- The old tokenizer reads Russian capitals byte by byte. ШКОЛА is 5 letters and 10 bytes, and GPT-2 needs all 10 tokens. Modern vocabularies fixed most of this.
- Chinese has no case to worry about, and the modern tokenizers learned 电脑 as one token. GPT-2 needs 5 byte fragments for those 2 characters.
One surprise: Gemma 4 keeps "HELLO" as one token, but " HELLO" with a leading space, as it would appear in a sentence, is two (␣HE | LLO). Tokenizers are full of these small inconsistencies.
How much to trust this
- The code points, byte counts and the filter demo are exact. They come from Python's own Unicode data; every code block above is code I ran, with its real output. The token numbers come from the same four tokenizers and loading code as the previous article.
- The token counts are a handful of words and three sentences. They show the direction (capitals and non-Latin alphabets cost more, especially in old tokenizers), not a rate you can budget with.
- Real databases compare strings with collations, not always raw bytes. Whether lowercase, accented or Chinese codes fall outside a range depends on the collation; my demo uses plain code-point order.
- The Shenzhen story is from memory, about 20 years ago. I remember the symptom and the cause; I'm not going to pretend I remember the exact code.
The one-line mental model
Letters are numbers, and the numbers don't care what you meant. 'a' is not 'A', "ZZZZZZ" is not the end of the alphabet, and a tokenizer sees "HELLO" as a different, more expensive thing than "hello".