Embeddings, positional encoding and self-attention

After school on Wednesday, a Secondary 5 Liberal Studies teacher tests the school knowledge base in the staff room. She types "what if a student faints on campus" and the system returns paragraph three of the Indisposition Procedures guide, source cited. She turns to a colleague: "It found it without any of my words." That find depends on the three mechanisms in this lesson.

Reading a sentence is not looking words up one by one. The model places each token on a map of meaning, marks where in the sequence it appeared, then lets every position decide which other positions to listen to right now. This closes module 1 (foundations): the previous lesson covered what parameters store; this one covers how they behave when a sentence arrives.

---

The floor plan

An embedding is like a museum floor plan — exhibits are grouped by theme, not alphabet. Environmental displays cluster together, history sits in another wing, technology in a third. Items on a similar theme end up near each other whether or not they share a label.

From catalogue number to position

A token is only a catalogue number — "T-48291" tells you nothing by itself. An embedding converts that number into coordinates on the plan: "environmental wing, near the entrance, beside the recycling demo." Training repeatedly adjusts those coordinates until words that tend to appear together drift closer.

The map is not two-dimensional. It has hundreds or thousands of directions at once. One direction might separate people from objects, another formal from colloquial register, another on-campus from off-campus. More directions mean several kinds of similarity can coexist without one sort of grouping wrecking another.

What it means for a school knowledge base

That map is what makes semantic search work. A teacher asks what to do when a student faints; the handbook heading reads "procedures for handling student indisposition." Almost no shared wording, yet the two land on nearly the same spot, so a RAG knowledge base finds it. Building a knowledge base requires an embedding model for exactly this reason.

Shot numbers

The floor plan has a blind spot: it does not remember order.

"Dog bites man" and "man bites dog" share the same three token coordinates. For a General Studies teacher the difference is not small. The Transformer design lets every position see every other at once (lesson 2 covered that), and the price is no built-in notion of before and after.

Positional encoding fills the gap. Think of a film storyboard: besides its place on the thematic map, each token gets a shot number — frame 1, frame 2, frame 3. That number is blended into the coordinates, so each fragment carries both what it means and where it sits in the sequence.

A practical corollary for teachers: order inside your prompt matters. The most important instruction at the start or end usually beats burying it in the middle; the middle of a long reference is the part most easily overlooked — discussed properly in the lesson on chunking and reranking.

Counterpoint in the score

Self-attention is the core reading action. Picture a composer opening a full score: while handling the note "he", she is not reading left to right only — she is listening across the staves for which other notes form counterpoint with it. "Borrowed" and "grateful" weigh heavily; "notes" less; "very" less still.

Every position does this

Self-attention runs separately for every position in the sentence. While processing "attitude", the model scores every other position for relevance, looks harder at high scorers and skims the rest. Those scores are learned, not hand-written.

Multi-head attention runs the same operation in parallel lanes. Four subject teachers read the same passage at once: Chinese notices rhetoric, History notices dates, Liberal Studies notices stance, English notices sentence pattern. Each lane "hears" different counterpoint; merging the lanes yields a richer reading than any one alone. A model layer typically has dozens of such heads.

This also resolves the pronoun puzzle left open in lesson 2. In "Ming borrowed Wah's notes, and he was very grateful", certain heads focus on "borrowed" and "grateful" while processing "he", and the reference resolves.

Embeddings find similar meaning, not matching conditions — that single line decides whether you name the year group and subject when you query a knowledge base.

A knowledge-base dry run

A secondary school loads its student handbook, crisis procedures and panel marking criteria into a knowledge base. The vice-principal runs two tests.

Round one: a colloquial question about students fighting against a formal heading on handling physical conflict between students. Different wording, similar map position — ranked first, pass.

Round two: "weighting of composition marks in Secondary 3 Chinese" — the system returns Secondary 3 English marking criteria. The two documents sit too close on the map; "similar meaning" cannot separate "different subject". Fix: keep subject and year labels when cutting documents, and add a reranking step to discard obviously wrong passages (lesson 16).

The lesson: any retrieval involving subject, year group or calendar year needs those conditions stated explicitly — in the question or in metadata — not left to embedding alone.

Where people go wrong

• The mistake: treating embeddings as keyword search, only "smarter". Self-check: test with two phrases that share no words (e.g. "fainted" vs "indisposition"); if it still retrieves, you have semantic search, not keywords.
• The mistake: procuring chat models without asking whether the provider offers an embedding API. Self-check: Anthropic and Poe provide no embedding model; a school that disables OpenAI entirely must confirm Azure OpenAI or a local alternative is ready.
• The mistake: burying critical instructions in the middle of a long prompt because the model "read everything". Self-check: run the same instruction at the start, middle and end and compare outputs; start or end is usually noticeably more reliable.

Try it once

Time: about 12 minutes. You need: any conversational AI tool.

1. Paste the six terms below and ask the model to "sort them into two groups by closeness of meaning and give one sentence on what each group shares": principal, form teacher, school bus, parents' day, school bag, curriculum coordinator
2. Then ask it to "place all six on a single line from 'people' on the left to 'objects' on the right".
3. Hand in: both groupings, the line order, and one sentence from you: "Which two terms did you expect in the same group but the AI split? Why?"

You got it right if: groupings are sensible (principal / form teacher / curriculum coordinator lean "people"; school bus / school bag lean "objects"; parents' day leans "event"); the line runs people → event → objects — roughly one direction on the embedding map. That completes module 1. Next, module 2 asks where the numbers on these components came from: pre-training. Previous lesson: what parameters actually store.