Pre-training, self-supervised learning and data cleaning

During break, a primary General Studies teacher asks the IT support colleague in the pantry: "It knows so much — is someone teaching it lesson by lesson?" The reply: "If we taught it that way, millions of people would need centuries. It taught itself with a karaoke game." That points to this lesson — the first stop in module 2 (the three training stages), right after seeing how the components inside a model read a sentence.

Pre-training means covering part of an enormous pile of text, asking the model to guess what was hidden, nudging the parameters from lesson 4 when it guesses wrong, and repeating that trillions of times. Nobody teaches it item by item; it learns by checking answers already inside the data. The underlying task throughout is guessing the next word — not being trained to be truthful, and not being trained to be helpful. Those two properties arrive in lesson 7 and lesson 8.

---

Karaoke fill-in

Picture lyrics on a karaoke screen with every few characters blanked out. You fill them in. A wrong fill nudges your "judgement habit" slightly; a right fill reinforces it. Then the next song, another blank.

Repeat a few trillion times.

That is self-supervised learning. The crucial part: the answer is already inside the data, so no human labelling is required. If every item needed a hand-written key, the cost would be impossible — the only reason the whole enterprise can run at this scale.

Each nudge adjusts parameters. After trillions of guesses, those parameters hold a compressed record of regularities in language, common facts and reasoning. So when a student asks whether someone taught the model everything, the accurate reply is not "a team feeds it daily" but "it read more text than any person could in a lifetime, and every page was a fill-in game".

The magazine editorial desk

If the model teaches itself by reading, then what gets submitted to training decides what it learns — much like a school magazine editor deciding which student pieces make the issue.

What editors do

Raw text scraped from the web contains encyclopaedia entries, papers and news, and also enormous volumes of duplicated advertising, machine-translated sludge, hate speech, corrupted characters and personal data. The training team's judgements look a lot like an editor's:

• Deduplication. An article appearing ten thousand times online will be over-memorised if all copies go in — like printing the same submission ten thousand times in one issue.
• Quality filtering. Rules and classifiers remove obviously low-quality and machine-generated text.
• Harmful content removal. Reducing hate speech, adult content and violent detail at source beats patching afterwards.
• Personal data handling. Stripping phone numbers, addresses and identity numbers as far as possible.
• Language and domain mix. Good Chinese requires enough good Chinese. Good mathematical reasoning requires textbooks and code.

This stage is enormous engineering work and very hard for outsiders to verify — which is why the model card lesson matters: a model card is one of the few places the data description is documented at all.

Two timed fuses

Bias in the data gets learned. Anything absent from the data gets invented. Both belong to lesson 9.

The model never saw your school during pre-training — that is structural, and no amount of better questioning fixes it.

Scale and what schools should do instead

A few orders of magnitude, useful in a meeting:

• Pre-training a frontier model takes thousands of high-end GPUs running continuously for weeks to months.
• Electricity and hardware run into the tens or hundreds of millions of dollars.
• Training data is measured in trillions of tokens which, using the conversion from lesson 3, is far more than a person could read in a lifetime.

No school should consider pre-training a model. What schools need, from cheapest to dearest:

1. Better prompts, which cost nothing. See prompt engineering.
2. A school knowledge base, at moderate cost. See RAG.
3. Light fine-tuning on an existing model, at higher cost. See LoRA and QLoRA.

More than nine in ten school requirements are met by the first two.

Three layers from a Chinese panel head

A Chinese panel head noticed AI-written model compositions read fluently but occasionally used vocabulary from Taiwan or the mainland — "video" in the wrong register, "hard disk" spelt the mainland way, year groups named in the wrong form. She rewrote her prompt several times; the problem persisted.

The cause is in the pre-training data. The total volume of Traditional Chinese text online is far smaller than Simplified Chinese or English, and Hong Kong usage is only a slice of that. What the model learned as "what Chinese looks like" leans towards whichever variety supplied the most text.

Her fix had three layers, all changing what she sends the model, never the model itself:

1. A fixed usage-convention line in the panel's shared prompt template, with ten common word pairs listed.
2. Five years of the panel's best model compositions loaded into the school knowledge base for house style to work from.
3. The word-pair list filed in the co-planning folder so new colleagues had it from day one.

Where people go wrong

• The mistake: assuming uploaded teaching materials "train" the model. Self-check: knowledge base documents are retrieved and cited; they do not change parameters. What you need to confirm is whether conversations are retained for provider training under your contract and deployment.
• The mistake: choosing a model from an overall benchmark without testing your own worksheets. Self-check: three of your own materials in fifteen minutes will show whether Hong Kong usage is usable.
• The mistake: explaining AI errors to colleagues as "its data is out of date". Self-check: many school-specific failures happen because your school was never in the data at all, not because dates are old — start from "it was only trained to continue smoothly".

Try it once

Time: about 12 minutes. You need: any conversational AI tool.

1. Paste the prompt below and ask for Traditional Chinese:

   Write three sentences about how a school uses a projector to show teaching videos. After each sentence, in brackets, list alternative wordings you considered but did not use.
   Then in two sentences say: when writing Traditional Chinese, whose regional vocabulary are you most likely to slip into, and why?
   

2. Compare the model's "slip" words against your panel's usage list (or draft five pairs on the spot: Hong Kong vs mainland/Taiwan forms).
3. Hand in: the three sentences, the model's admitted slip tendency, and one sentence from you: "Which property of pre-training data does this reflect?"

You got it right if: the model concedes that Traditional Chinese training data is comparatively scarce, or that Hong Kong usage is a small share — direct evidence of the data mix. Next: turning a model that continues text into one that answers questions — fine-tuning and SFT. Previous lesson: embeddings and self-attention.