Where hallucination and bias come from, and how to manage them

Short answer: hallucination is not a fault. It is what a machine that always fills the blank does when it meets something it has never seen. Bias is that same machine faithfully reflecting the world in its training data.

It starts with a school history

A teacher preparing for an anniversary asked an AI to draft a short school history. She supplied the school name and some known details, and the draft that came back read beautifully: founding year, the first principal's name, a fire in the 1950s, the relocation in the 1970s, even the street name of the old premises.

Three of those items were invented. The first principal's name was wrong, the fire never happened, and the street name had been borrowed from a different road in the district. The most unsettling detail was that the invented passages read more fluently than the true ones.

Her question was the right one: if it did not know, why did it not say so?

Why it does not say "I don't know"

Answering that only requires stringing the previous lessons together.

Lesson 2 established that each step does one thing: score the possible next words and draw one. Lesson 4 established that parameters store patterns, not entries. Lesson 6 established that the sole pre-training objective is continuing smoothly. Lesson 8 established that alignment rewards being helpful.

Put those four together and the conclusion is clear:

When the model meets a fact it has never seen, it does not encounter a blank and stop. It encounters the question of what the next word should be, and that question always has an answer. So it assembles one from the closest patterns available. What does the history of a 1950s Hong Kong school usually contain? Hence the fire.

It is like a student doing a cloze exercise who would rather write something plausible than leave a gap, because a gap scores zero for certain. The difference is that the student knows they are guessing and the model does not.

Four causes of hallucination

Separating the causes is useful, because each has a different remedy.

One: absent from the data. Your school's history, last year's board minutes, an individual student's results — none of it was ever in the training data. The remedy is to supply the data, which is a school knowledge base with RAG.

Two: too thin in the data. Obscure Education Bureau circular numbers, specialist terminology in minority subjects, local Hong Kong institution names. The model has seen a little, so it produces a mixture of true and false — which is more dangerous than knowing nothing, because the falsehoods are embedded in accurate material.

Three: out of date in the data. Curriculum reform, examination arrangements, anything after the model was built. That has its own specific cause, the knowledge cutoff, which is the next lesson.

Four: led by the question. The most overlooked cause. Ask for the contents of chapter three of a circular that does not exist and most models will write one for you, because your question presupposed its existence. Ask instead whether such a circular exists, what its number is, and to say so plainly if it does not or if the model is unsure, and the result is completely different.

Three sources of bias

Society as reflected in the data. If "nurse" mostly co-occurs with women and "engineer" mostly with men in the training text, the role assignments in generated teaching materials will carry that ratio forward. The model has no position; it is faithfully reflecting its data.

The backgrounds of the annotators. The demonstration data of lesson 7 and the ranking data of lesson 8 were written and scored by a specific group of people. Their cultural background, linguistic habits and value judgements enter the model. For a Hong Kong school, the practical upshot is that the model's sense of what counts as appropriate is not necessarily a Hong Kong classroom's.

Language and regional imbalance. English data vastly exceeds Chinese, Simplified Chinese vastly exceeds Traditional, and Hong Kong usage is only a slice of Traditional. The result is weaker handling of Hong Kong curriculum context, as discussed in lesson 6.

In teaching materials, bias shows up most often in three ways: stereotyped pairings of gender with occupation in example characters, examples assuming a particular family structure or level of affluence, and over-marking of ornate but empty writing.

An example from the classroom

A Primary 6 General Studies teacher preparing material on occupations in Hong Kong asked for twelve character profiles of one hundred words each, with illustration prompts.

The first version was instructive. Of the twelve, the doctor, engineer, pilot and lawyer were all men; the nurse, kindergarten teacher, social worker and clerk were all women. Four of the profiles also mentioned living in a private housing estate with professional parents, while more than half the pupils in her school live in public housing.

Her handling of it is worth copying. Rather than editing twelve profiles, she rewrote the prompt: balanced gender distribution, diverse family backgrounds, at least three occupations that do not require a degree, and for each person a sentence on why they chose that line of work. The second version was far more usable.

Then she did something more important. She filed that prompt in the panel's shared template folder and spent ten minutes of co-planning time comparing version one with version two. Colleagues subsequently spotted the same pattern in other subjects.

That is the practical shape of bias management: not expecting the model to be fair, but making "what to check" a panel-wide habit. A fuller discussion is in the lesson on privacy, bias and copyright ethics.

Five measures you can use today

1. Give it a way out. End every prompt with a standing instruction: if you are unsure of any item, write "teacher to verify" rather than filling it in. This single line does more than any amount of warning.
2. Require sources. In an environment with a knowledge base, ask for citations and actually open them. Treat any factual claim without a citation as unverified.
3. Separate facts from prose. Let the AI handle structure and sentences while you supply the facts. School history, figures, dates, names and policy reference numbers always come from a person.
4. Verify by asking backwards. After receiving an answer, start a fresh turn: "Which of the statements above are you actually unsure about? List them." Models are noticeably more candid under that framing than while generating.
5. Double-check high-stakes output. External documents, student comments and anything touching results get a second colleague's read. That is not distrust of AI; those things warranted a second pair of eyes anyway.

Try it yourself

Paste these two prompts into any AI tool one after the other and compare:

First: Explain in detail the main content of chapter three of the Education Bureau's 2019 "Guidelines on Artificial Intelligence Teaching in Primary and Secondary Schools".

Second: Did the Hong Kong Education Bureau issue a document in 2019 titled "Guidelines on Artificial Intelligence Teaching in Primary and Secondary Schools"? If not, say so plainly. If you are unsure, say you are unsure. Do not describe its contents while unsure.

The first prompt frequently produces a very presentable chapter summary. The second is usually far more honest. The only difference is how the question was asked — which happens to be the part teachers control most directly.

What this means for your classroom

1. Treat AI as a drafting tool, not a verification tool. It is excellent at organisation and expression and poor at guaranteeing accuracy. Once the division of labour is clear, both sides do better work.
2. Route every school-specific fact through the knowledge base. This is not only about accuracy but about auditability: a citation is what makes checking possible.
3. Build a checklist at panel level. Are genders and family backgrounds varied? Do figures have a source? Has the policy name been verified? Individual habits slip; a panel checklist does not.
4. Demonstrate verification to students rather than banning use. Letting a class watch an AI invent something once is far more effective than ten warnings not to trust it. It also makes an excellent information literacy lesson in its own right.

Next

This lesson handled the case where the model does not know. One particular kind of not-knowing deserves its own lesson: the model did know, but its knowledge stops on a particular date. Next we look at knowledge cutoffs and how a school should manage model versions.

Continue with knowledge cutoffs and model versioning. Previous lesson: alignment with RLHF and DPO. Related reading: LLM safety, hallucination and prompt injection risk.