Embeddings, positional encoding and self-attention

Short answer: the model places every fragment on an enormous seating plan, gives each position a class number, and then at every layer re-decides which positions it should be looking at right now.

A student asked how it knows "principal" relates to "vice-principal"

A Secondary 4 computer studies teacher was introducing databases when a student asked something unexpected: "Sir, I type 'principal' and it understands, I type 'VP' and it understands. Does it have a dictionary?"

Not a dictionary. A dictionary would give the definition of "vice-principal" but would not know that a vice-principal is closer to a curriculum coordinator than to a school bus driver. Models manage the second thing because they do something quite different: they turn meaning into position.

Component one: embeddings, an enormous seating plan

Imagine arranging seating for a whole-school event, and deciding to seat by interest: basketball in one area, drawing in another, students who like both somewhere in between. Once you have done that, something useful happens — knowing where a student sits tells you roughly what they are like, even if you have never met them.

An embedding is that seating plan, except it is not a two-dimensional hall layout but a plan with one or two thousand directions. That sounds alarming and means something simple: each direction represents one facet of similarity, perhaps positive against negative tone, perhaps formal against colloquial, perhaps person against object. More directions means several kinds of similarity can be respected at once, instead of the drawing groups falling apart the moment you sort by basketball.

The first thing a model does with a sentence is swap each token fragment for its coordinates on this plan. Training is the process of adjusting those coordinates until things that occur together sit together.

This has a very practical consequence for schools: semantic search. Ask a school knowledge base what to do when a student falls ill at school, while the staff handbook heading reads "procedures for handling student indisposition", and the two share almost no words — yet they sit close together on the plan, so the system finds it. This is the foundation on which a RAG knowledge base works, and the reason building one requires an embedding model.

Component two: positional encoding, the class number

The seating plan has one serious gap: it does not remember order.

"Dog bites man" and "man bites dog" use the same three words, so they land on the same three sets of coordinates. Looking only at coordinates, the two sentences are identical. For a General Studies teacher the difference is not small.

As lesson 2 explained, the Transformer breakthrough was letting every position see every other position at once, and the price of that design is that it has no inherent notion of before and after. So order is added separately: each fragment receives a class number marking where in the sequence it appeared. That is positional encoding.

In practice the position information is blended into the coordinates themselves, so each fragment's coordinates carry both what it means and where it sits.

There is a useful corollary for teachers: order inside your prompt matters. Putting the most important instruction at the start or the end usually works better than burying it in the middle, and in a long reference document the middle is the part most easily overlooked. That phenomenon is discussed properly in the lesson on chunking and reranking.

Component three: self-attention, underlining the question

Now the central component.

Picture a student tackling a comprehension question: "According to paragraph two, what is the writer's attitude towards land reclamation?" A student who knows what she is doing does not start reading from the top. She underlines the question first — "paragraph two", "writer", "attitude", "reclamation" — and goes back with those four words in mind.

Self-attention does exactly that, and does it separately for every position in the sentence. While processing the word "attitude", the model scores every other position for relevance, looks harder at the high scorers and skims the rest. Those scores are not written by hand; they are learned.

Multi-head attention is the same operation done many times over. Instead of one student underlining, picture six subject teachers reading the same sentence: the Chinese teacher notices the rhetoric, the History teacher the dates, the Liberal Studies teacher the stance, the English teacher the sentence pattern. Each underlines differently, and combining six sets of underlining gives a richer reading than any one of them. A model layer typically has dozens of such heads.

This also answers the question left open in lesson 2. In "Ming borrowed Wah's notes, and he was very grateful", certain heads put their attention on "borrowed" and "grateful" while processing "he", and the reference resolves.

An example from the classroom

A secondary school was building a knowledge base holding the student handbook, parent circular templates, crisis handling procedures and every panel's marking criteria. The vice-principal in charge worried about one thing: teachers ask in colloquial wording, while documents use formal wording. How would the system connect a casual question about students fighting with a heading about handling physical conflict between students?

The first test surprised him. It found it, ranked first. Embeddings are the reason: the casual and formal phrasings sit close together on the plan.

The second test exposed the real weakness. He asked about the weighting of composition marks in Secondary 3 Chinese, and the system returned the Secondary 3 English marking criteria. The reason is equally clear: the two documents sit very close together, and closeness alone cannot separate subjects. There are two fixes — keep subject and year-level labels attached when documents are cut into passages, and add a reranking step that discards obviously wrong passages. Both belong to lesson 16.

The lesson here is worth remembering: embeddings find similar meaning, not matching conditions. Any retrieval involving a subject, a year group or a year needs those conditions handled explicitly.

Try it yourself

Paste this into any AI tool to see what embedding closeness feels like:

Sort these six terms into two groups by closeness of meaning, and explain your grouping:
principal, form teacher, school bus, parents' day, school bag, curriculum coordinator
Then tell me: if you had to arrange all six along a single line running from "people" to "objects", what order would you use?

The groupings are usually sensible, and the line it produces is, roughly speaking, one direction on the seating plan.

What this means for your classroom

1. Choose the embedding model carefully and check provider limits. Not every provider offers one, so ask during procurement; otherwise the knowledge base and speech features may need a separate arrangement.
2. Colloquial questions are fine. Teachers do not need to memorise official document wording to find things, which matters a great deal for adoption.
3. State the year group, subject and year explicitly. Similar meaning is not the same as correct conditions, and naming the conditions sharply reduces wrong retrievals.
4. Keep critical instructions out of the middle of a long prompt. Put them at the start or the end, or restate them at the end. The difference in practice is noticeable.

Next

That completes module 1. You now know what components sit inside a model. Module 2 asks a different question: where did the numbers on those components come from in the first place? That is the story of the three training stages, starting with pre-training.

Continue with pre-training, self-supervised learning and data cleaning. Previous lesson: what parameters actually store.