1 · How a tokenizer learns its pieces (BPE on a tiny Spanish corpus)
Next merge
Reset
2 · Tokenizer
Language
tokens
Dashed red = a token that is only part of a character (raw bytes).
3 · Tokens for the same 3 sentences, per language