Tokenisation: why Chinese uses more slots than English

A Primary 4 Chinese teacher demoed in class: "How many characters are in this classical line?" The AI said nine. She counted on the board: eight without punctuation, ten with. The pupils laughed, but she asked the sharper follow-up: "If it cannot count characters, why trust it to mark essays?" That question is more useful than the laughter.

Lesson two described token-by-token generation. This is module one, lesson three: what a token actually is, and why the Chinese version of the same material usually occupies more of your allowance. Without this lesson, talk of context windows, RAG and cost has no shared unit.

---

Standard courier boxes

Before a model reads or writes text, it packs the text into fixed-size boxes. Those boxes are tokens, numbered and listed in a vocabulary of roughly a hundred thousand slots. Common English fragments often fill one box whole; rare strings may need two or three boxes end to end.

How are the sizes chosen? Builders take an enormous text corpus, repeatedly merge the character pairs that most often appear together, using an algorithm called byte pair encoding. The crucial detail: the sizes are tuned mainly on English-dominant text — like a courier whose standard cartons were designed around A4 letters and English word lengths. Shipping the same information in Chinese often takes more boxes.

A token is therefore neither a character nor a word, but a vendor-defined packaging unit. The model handles nothing else from first step to last.

Why Chinese fills more slots

OpenAI's English rule of thumb is about four letters per token, so a hundred tokens is roughly seventy-five English words.

Chinese has no spaces between words and a vastly larger character set. In the same vocabulary, a common character is often one box, while names, place names, rare characters and some punctuation need two or three. In practice:

• A 300-word English passage runs to roughly 380 to 420 tokens.
• A Chinese passage of the same meaning runs to roughly 350 to 600 tokens, depending on vocabulary.
• A thirty-page school-based Chinese curriculum document easily reaches thirty to fifty thousand tokens.

One thing people forget: output counts too, usually at several times the input rate. Asking for a two-thousand-character lesson plan in Chinese consumes more allowance than expected. The quickest check is to paste your material into OpenAI's tokenizer tool and watch where the cuts fall.

Billing counts boxes used, not characters written — the same meaning in Chinese often needs more boxes.

Context windows and school documents

The context window is how many tokens the model can hold at once. Current flagship models sit around a million tokens, which sounds unlimited, but the unit is tokens, not characters.

For a school, a million tokens could hold a year's Chinese teaching material in theory, but that is not how you should work. Two reasons: some providers price very long requests at a premium, and burying the relevant passage in irrelevant text makes the model more likely to miss it — covered in chunking and reranking. The right approach is RAG with only relevant passages: fewer boxes per request, better accuracy.

A Secondary 3 panel head's trial and error

A Secondary 3 Liberal Studies panel head wanted every school-based note, three years of papers and all rubrics in the AI so colleagues could ask marking expectations for a unit and get an immediate answer.

His first approach pasted everything at the start of each chat. Uploads were slow, citations sometimes pointed at the wrong unit, and the same document in English used about thirty per cent fewer tokens — briefly tempting him to translate all notes into English, which was obviously unworkable since students read them in Chinese.

He switched to indexed passages and sent only the three to five most relevant sections, requiring cited sources in the answer. Usage fell from around fifty thousand tokens per question to around three thousand, and accuracy rose. That was a retrieval problem, not a language problem.

Why counting and acrostics fail

Asking the model to "list every character and count again" is a character-level task. Internally it only has boxes, not a character lane. A cut may fall mid-character, or two punctuation marks may share a box — so eight characters on the first count and nine on the second does not mean the model is broken. The task does not match the internal representation.

Acrostics fail for the same reason: generation is still token-by-token, and the first token of a line may not align with the first character you had in mind. Send exact counting to Word or a spreadsheet; use the model for comments and analysis.

Where people go wrong

• The drift: assuming one Chinese character equals one token and using character count directly in pricing. Self-check: Paste a 200-character school paragraph into the tokenizer. Is the token count closer to 1.2–1.8 times the character count? If not, fix the formula before quoting.

• The drift: rewriting Hong Kong curriculum context in English to save tokens while demanding Traditional Chinese output. Self-check: Did output tokens shrink? If not, you saved a few instruction lines at the cost of year-level and subject wording drift.

• The drift: pasting a whole PDF every chat and complaining the AI "has bad memory". Self-check: Does a single request exceed ten thousand tokens? If yes, ask whether retrieval of three relevant sections is possible first — that is what RAG is for, not a bigger model.

Try it once

Time: about fifteen minutes. You need: any AI tool; optionally OpenAI's tokenizer page.

Do this: paste the following:

How many Chinese characters are in the following, excluding punctuation? List them one by one, then count again to confirm.
「香港特別行政區政府教育局課程發展處」
Also estimate roughly how many tokens this uses, and explain why character counting is error-prone for you.

If you have the tokenizer tool, paste the same line there and record the actual token count.

Deliverable: three lines — the AI's character count, your hand count, and the tokenizer token count (or the AI's estimate).

Done when: you can tell a colleague the AI miscounts because it operates on boxes, not characters, and you can quote "character count times about 1.5" as a good enough token estimate for planning context fit.

Next lesson: what parameters actually store. Previous lesson: the Transformer in plain language.