Pre-training, self-supervised learning and data cleaning

Short answer: pre-training means covering up part of an enormous pile of text, asking the model to guess what was hidden, nudging the parameters when it guesses wrong, and repeating that trillions of times. Nobody teaches it item by item. It teaches itself with cloze exercises.

A teacher asked whether someone feeds it, lesson by lesson

A primary General Studies teacher voiced something many people quietly wonder: "It knows so much. Is there a team feeding it every day, the way we teach students one lesson at a time?"

It is a natural picture and the opposite of the truth. Teaching by hand at the scale of human knowledge would take millions of people centuries. The actual method is far cleverer, and every language teacher has done a version of it — setting a cloze exercise.

Cloze: a worksheet that sets its own questions and marks itself

Imagine a student who never tires, and an extremely large library. Your teaching method is this.

Open a page at random, cover one word with your finger, and ask what word belongs there. The student guesses. You lift your finger, and the answer is right there on the page — so you never need to prepare an answer key. A wrong guess makes the student adjust slightly. A right guess reinforces. Then turn the page and cover another word.

Repeat a few trillion times.

That is self-supervised learning. The "self" is the crucial part: the answer is already inside the data, so no human labelling is required. That is the only reason the whole enterprise can run at this scale. If every item needed a human-written answer, the cost would be impossible.

Each adjustment nudges the parameters described in lesson 4. After trillions of guesses, those parameters hold a compressed record of the regularities of language, fact and reasoning.

There is a corollary teachers should notice: the model's underlying task, throughout, is guessing the next word. It is not trained to be truthful, and it is not trained to be helpful. Those two properties are added in the next two stages, covered in lesson 7 and lesson 8.

Library acquisitions: rubbish in, rubbish out

If the student is self-teaching by reading, then what sits on the shelves decides what is learned.

That is what data cleaning is for. Raw text scraped from the web contains encyclopaedia entries, academic papers and news reports, and also enormous volumes of duplicated advertising, machine-translated sludge, hate speech, corrupted characters and personal data. The training team's judgements look a lot like a librarian's acquisitions decisions:

• Deduplication. An article appearing ten thousand times online will be over-memorised if all ten thousand copies go in. It is the library buying ten thousand copies of one book.
• Quality filtering. Rules and classifiers remove obviously low-quality and machine-generated text.
• Harmful content removal. Reducing hate speech, adult content and violent detail at source is far more effective than patching afterwards.
• Personal data handling. Stripping phone numbers, addresses and identity numbers as far as possible.
• Language and domain mix. Good Chinese requires enough good Chinese. Good mathematical reasoning requires textbooks and code.

This stage is enormous engineering work and very hard for outsiders to verify, which is why the model card lesson matters: a model card is one of the few places where the data description is documented at all.

This stage also plants the seeds of two later problems. Bias in the data gets learned, and anything absent from the data gets invented. Both belong to lesson 9.

Scale: why no school does this

A few orders of magnitude, useful in a meeting:

• Pre-training a frontier model takes thousands of high-end GPUs running continuously for weeks to months.
• Electricity and hardware run into the tens or hundreds of millions of dollars.
• Training data is measured in trillions of tokens which, using the conversion from lesson 3, is far more than a person could read in a lifetime.

So the conclusion is clear: no school should consider pre-training a model. What schools need, from cheapest to dearest:

1. Better prompts, which cost nothing. See prompt engineering.
2. A school knowledge base, at moderate cost. See RAG.
3. Light fine-tuning on an existing model, at higher cost. See LoRA and QLoRA.

More than nine in ten school requirements are met by the first two.

An example from the classroom

A Chinese panel head noticed that AI-written model compositions read fluently but occasionally used vocabulary from Taiwan or the mainland rather than Hong Kong conventions, and year-group names came out in the wrong register. She assumed at first that her prompt was at fault, rewrote it several times, and the problem persisted.

The real cause is in the pre-training data. The total volume of Traditional Chinese text online is far smaller than that of Simplified Chinese or English, and Hong Kong usage is only a slice of that. What the model learned as "what Chinese looks like" leans towards whichever variety supplied the most text.

Her eventual approach had three layers. First, a fixed usage-convention line in the panel's shared prompt template, with ten common word pairs listed. Second, five years of the panel's best model compositions loaded into the school knowledge base so the AI had house style to work from. Third, the word-pair list filed in the co-planning folder so new colleagues had it from day one.

Note what she did. She did not try to change the model. She changed what she sends the model. For a school that is almost always the right direction.

Try it yourself

To see how the training data mix shows up in the output, paste this into any AI tool:

Write three sentences in Traditional Chinese about how a school uses a projector to show teaching videos, and in brackets note the alternative wordings you considered but did not use.
Then tell me: when writing Traditional Chinese, whose regional vocabulary are you most likely to slip into, and why?

Most models will concede that Traditional Chinese training data is comparatively scarce and list a few words they commonly mix up. That answer is itself evidence of the data mix.

What this means for your classroom

1. Write the usage conventions into the prompt instead of assuming the default is right. A line specifying Hong Kong Traditional Chinese conventions belongs at the top of every panel template.
2. Close school-specific gaps with a knowledge base, not with training. The model never saw your school during pre-training. That is structural, and no amount of better questioning fixes it.
3. Evaluate models with your own materials. An overall benchmark says nothing about the quality of its Hong Kong Chinese. Three of your own worksheets will answer the question in fifteen minutes.
4. Start here when explaining to colleagues why AI gets things wrong. It was not trained to be truthful; it was trained to continue smoothly. That single sentence dissolves a lot of over-trust.

Next

After pre-training the model holds vast linguistic regularities, but it still does not know how to answer a question. Type a question and it may simply write more questions. Turning it into something that responds is the next stage.

Continue with fine-tuning, SFT and where instruction-following comes from. Previous lesson: embeddings, positional encoding and self-attention.