The Transformer in plain language: how models append one token at a time

During a Secondary 2 demo, a student asked: "Sir, does it think first and then answer?" The teacher paused and said, "It's more like… it spits out one word at a time." The class laughed, but the joke was accurate: the model does not compose a full script internally. It repeatedly asks what the next token should be.

Lesson one made the case for understanding LLMs. This is module one, lesson two: what happens after a sentence goes in. No formulas — just a scene you may already live every week, changing a prompt and watching the output shift.

---

The live caption desk

Picture the school radio running a live segment. The host must read listener messages aloud without seeing the full reply first — only the text so far, then the next word; after each word the whole line goes back through the system and the next token is chosen.

That is how the output side of a Transformer works. Given "During recess on the playground", the model might score "it" at 0.28, "the" at 0.19, "I" at 0.11, draw one token, append it, and loop until stopping.

Three daily staffroom facts follow from that single mechanism. Mark the same essay twice and the marks may differ, because each step is a draw, not a fixed top pick. Long outputs drift, because each step only cares whether the next token follows smoothly. Wrong facts arrive with confidence, because "I am not sure" is rare in training text and therefore low in probability — lesson 9 takes that apart.

Three rows in the control room

The input side is richer. A sentence passes through dozens of layers — imagine three rows of colleagues in the control room, each receiving the annotated text from the row above and adding another layer of notes.

The first row handles surface pairing: which words tend to sit together. The second handles syntax and reference: whether "he" means Ming or Wah. The third handles intent and tone: complaint, explanation or irony. No row alone "understands" the sentence. It is the stack of notes that finally supports the next-token guess.

Before writing anything, each row does the same job: decide which positions in the sentence matter most right now. When processing "he", attention lands on clues like "borrowed" and "grateful". That is self-attention. Lesson 5 uses the analogy of underlining key words in an exam question. For now, hold on to one idea: every layer, for every position, reallocates where to look.

Why that mattered historically

Before Transformers in 2017, the usual approach read strictly left to right, like passing a note along a row, so information from the start had to survive the whole journey to reach the end. Transformers lay the whole sentence out at once — any position can look directly at any other — which is why long-distance reference works far better. It also means concrete conditions in a long prompt can still be seen, but the earlier and sharper they are, the more they narrow the draw at each step.

Two prompts for a Primary 5 reading sheet

A Primary 5 Chinese teacher needed a reading sheet on public transport in Hong Kong. Her first prompt was one line: "Write me a Primary 5 Chinese reading comprehension worksheet."

The passage was usable, but two questions asked for two hundred characters — plainly wrong for Primary 5. Her second attempt:

"Write a 350-character expository passage for Primary 5 students on public transport in Hong Kong. Then set five comprehension questions: two retrieval, two inferential, one open-ended. Use Hong Kong Traditional Chinese conventions. Do not include answers."

The second version was print-ready. Through the mechanism above, the first prompt only gave "Primary 5, Chinese, comprehension, worksheet" at each guessing step. The second added length, genre, question-type mix and language rules. Every concrete number in a prompt is a rail that narrows the draw in every row of the control room.

A prompt is not a polite request. It is a rail that applies at every generation step — the more specific it is, the smaller the draw.

That is why prompt engineering works, and it has little to do with whether the model is "obedient".

Three implications for your classroom

Put conditions in the prompt; do not expect inference. Year level, length, genre, things to avoid — the model has no commonsense knowledge that a Primary 5 pupil cannot write a two-hundred-character analysis. Drop one condition and the output may drift toward secondary-level habits mid-stream.

Stage long outputs. Asking for a whole lesson plan in one go invites internal inconsistency. "Write the learning objectives first; I will confirm before you write the activity sequence" usually improves quality because confirmed text re-enters the context at each step.

The architecture travels. Whether the school runs OpenAI, Claude or Ollama on campus, the underlying design is a Transformer. Token-by-token assembly and whole-sentence visibility still apply after a provider change.

Where people go wrong

• The drift: assuming the model composes a full answer internally, so the same question should always produce the same text. Self-check: Run the same prompt three times. If wording clearly differs, you are dealing with sampling — marking and answer checking still need teacher review.

• The drift: writing a long, polite prompt with no hard constraints on length, year level or question type. Politeness does not automatically become a rail. Self-check: Remove every "please" and "kindly". Do measurable constraints remain? If all that is left is "write a good worksheet", drifting output is expected.

• The drift: asking for "full paper + answers + rubric + parent letter" in one go, then blaming the model when the second half weakens. Self-check: Split into at least two chats and make the second explicitly reference confirmed text from the first. If you have not tried that, complain after you have.

Try it once

Time: about twelve minutes. You need: any AI tool.

Do this: paste the following:

Continue this in one sentence: "During recess this morning, out on the playground"
Then, on a new line, list four other tokens you considered for the first word of your continuation, and say briefly why you did not choose them.

Deliverable: a screenshot or copied reply with the continuation sentence and the four alternatives highlighted.

Done when: you can tell a colleague in one sentence that the model appends token by token rather than composing the whole paragraph first, and you can add that the model's self-reported candidate list is not a precise internal log but is enough to show the unit of work is "the next token".

Next lesson: tokenisation, and why Chinese costs more — the model never sees "words", only cut fragments. Previous lesson: why teachers need to understand LLMs.