Tokenisation, and why Chinese costs more than English

Short answer: the model never sees characters. It sees pre-cut fragments, and a Chinese sentence produces more fragments than an English sentence of the same meaning — so the Chinese version of the same material uses more of your allowance.

It starts with a question that should have been easy

A Primary 4 Chinese teacher preparing a stroke-order exercise asked the AI how many Chinese characters were in a short classical quotation. The AI said nine. Counting by hand, it was ten with punctuation and eight without. Nine was not a defensible answer under any counting rule.

She tried again, asking for a four-line acrostic poem whose first characters spelled out a four-character idiom. Three of the four first characters were right. The fourth was wrong.

The same AI can write a structurally sound lesson plan and analyse rhetoric in a classical text, yet it cannot count to nine. This is not a shortfall in intelligence. It simply cannot see the things you are calling characters.

The Lego box: the pieces the model has to work with

Imagine building a sentence out of Lego. Your box does not contain one brick per Chinese character, nor one brick per English word. It contains a set of fragments decided in advance by the manufacturer — on the order of a hundred thousand pieces. Some are whole common words, some are half a word, some are a letter cluster, some are just punctuation.

When a model reads a sentence, the first thing it does is break the text into these ready-made fragments. When it writes, it assembles fragments one at a time. Those fragments are tokens.

How is the box decided? In short, the builders take an enormous pile of text, find the letter pairs that most often occur together, merge them into a single brick, and repeat that tens of thousands of times. The result is a box where common things are one brick and rare things take several. The usual algorithm is called byte pair encoding, or BPE. This is also why everyday words tend to land on exactly one token while unusual proper nouns get chopped into pieces.

The point that matters: the box is built mostly from English-dominant text.

Why Chinese costs more

OpenAI's own rule of thumb for English is that one token is about four characters of text, so a hundred tokens is roughly seventy-five English words.

Chinese behaves differently. There are no spaces marking word boundaries and the character set is vastly larger, so in the same box a common Chinese character is roughly one token, while rarer characters, names, place names and some punctuation need two or three.

In practical terms:

• A 300-word English passage runs to roughly 380 to 420 tokens.
• A Chinese passage of the same meaning runs to roughly 350 to 600 tokens, depending on vocabulary.
• A thirty-page school-based Chinese curriculum document easily reaches thirty to fifty thousand tokens.

And one thing that is regularly forgotten: output counts too, usually at several times the input rate. Asking for a two-thousand-character lesson plan in Chinese consumes considerably more than people expect.

The quickest way to see all this is to paste your own teaching material into OpenAI's tokenizer tool and watch where it cuts.

How this affects the context window

The context window is the model's working memory — how much it can hold in its hands at one time. Current flagship models sit around a million tokens, which sounds effectively unlimited. But the unit is tokens, not characters.

For a school the real constraint works out like this. A million tokens could hold a year's worth of Chinese teaching material, but that does not mean you should load it. Two reasons: cost, since some providers price very long requests at a premium, and accuracy, because burying the relevant passage in a mass of irrelevant text makes the model more likely to miss it. That phenomenon gets a full treatment in the lesson on chunking and reranking, and the right approach is retrieving only the relevant passages with RAG.

An example from the classroom

A Secondary 3 panel head wanted to load the panel's school-based notes, three years of assessment papers and all the marking rubrics into an AI so colleagues could ask what the marking expectations were for a given unit and get an immediate answer.

His first approach was to paste everything in at the start of each conversation. Three problems followed. Uploads were slow. The model sometimes cited a completely unrelated unit. And he noticed that the same document translated into English used about thirty per cent fewer tokens, which briefly tempted him to convert all the school-based notes to English — obviously unworkable, since the students read them in Chinese.

The real fix was not to compress the language but to change how the material is reached: cut the documents into passages, index them, and send only the three to five most relevant passages along with the question, requiring the answer to cite its source. That is what a school knowledge base does. Token usage fell from around fifty thousand per question to around three thousand, and accuracy went up rather than down.

Try it yourself

Paste the following into any AI tool to watch token boundaries cause trouble in real time:

How many Chinese characters are in the following, excluding punctuation? List them one by one, then count again to confirm.
「香港特別行政區政府教育局課程發展處」
Also tell me roughly how many tokens this uses, and why counting characters is error-prone for you.

Most models get the count wrong, or produce two different counts in the same reply. Nothing is broken. You have simply asked for character-level counting, which sits exactly on the seam of the Lego box.

What this means for your classroom

1. Send exact counting to a word processor. Whether an essay reaches 500 words is a job for the word count in Word; a class average is a job for a spreadsheet. Use AI for the comments and the analysis, not the arithmetic.
2. Multiply Chinese character counts by about one and a half. To judge whether a document will fit, "characters times 1.5" is a good enough token estimate for planning purposes.
3. Do not trade prompt clarity for tokens. Two more lines of constraints cost far less than regenerating output that came back unusable.
4. In pricing conversations, ask how you are actually charged. Ten dollars per million tokens means nothing to a school that does not know its annual token volume. A per-school annual fee with AI usage included is far easier to budget, because it moves the risk of token volatility off the school.

Next

You now know the model reads and writes in fragments. But where is the knowledge of which fragments go together actually stored? In the parameters — the number every press release boasts about. The next lesson looks at what that number really means.

Continue with what parameters actually store. Previous lesson: the Transformer architecture in plain language.