What parameters actually store, from 10B to a trillion

Short answer: a parameter count is roughly a count of brain cells. It sets how many patterns a model can hold, but what it holds are patterns and habits, not entries you can look up.

The principal asked a numbers question

An IT committee was reviewing AI procurement when the principal looked at "a trillion parameters" on the vendor slide and asked a very direct question: "The other company says seven hundred billion. Is that thirty per cent worse? Should we take the bigger one?"

A perfectly reasonable question, and the answer is that the comparison is close to meaningless — rather like ranking two teachers by how many brain cells each has. Cell count matters in the sense that too few would obviously be a problem, but it tells you nothing about whether the teacher is trained, knows the curriculum, or can talk to a class.

Brain cells: a useful analogy and where it stops

Think of parameters as the strengths of the connections between brain cells. An adult brain has an enormous number of connections, and the strength of each one contributes to your habits, intuitions and skills. There is no single cell holding the Secondary 3 curriculum outline; your knowledge is spread across the strengths of countless connections.

Model parameters work like that: a vast collection of numbers, each tiny and meaningless alone, that together determine what the next token is most likely to be. Training is the process of nudging those numbers repeatedly until the prediction gets good.

So "ten billion parameters" means the model has ten billion such dials. More dials means room for finer patterns, but whether the dial settings are any good depends on the data and the training method, which is lesson 6.

What parameters store: experience, not a bookshelf

This is the most important point in the lesson.

A teacher with twenty years in Chinese language teaching does not carry copies of textbooks in her head. She carries experience: which essay openings students handle well, which characters they most often miswrite, which question types need groundwork first. Ask her to quote the second sentence of the third paragraph of a specific classical text and she may not manage it, yet she knows the structure and emotional arc of the whole piece without opening a book.

Models are the same. They saw a great deal of text during training, but the text is not stored. The patterns in the text are compressed into the parameters. So:

• It knows how an argumentative essay is usually structured. The pattern was learned well.
• It knows Hong Kong secondary schools generally run from Secondary 1 to Secondary 6. Common enough to have become a pattern.
• It does not reliably know the third recommendation in your school's 2018 external review report. That appeared zero or one times in training and could not be compressed in.

And when you ask the third kind of question, it will not say it has not seen this. It will use the patterns from the first two kinds to assemble something plausible. That is the mechanism behind hallucination, covered fully in lesson 9.

Put differently: parameters are not a database, so do not use them as one. Your school's own information belongs in a retrievable, citable knowledge base, not in a hope that it made it into the weights.

An example from the classroom

A primary school IT coordinator wanted to run a model on a school server so teachers could draft circulars without going to the internet. He had two options: a seven-billion-parameter model that would run on a machine the school already had with a dedicated graphics card, or a seventy-billion model that would need about a hundred thousand dollars of new hardware.

He did something sensible first. He took twenty circulars actually written over the past year, generated versions with both models, and had three panel heads score them blind.

The seven-billion version came close on format and register. The differences showed up in two places: the logic in long paragraphs occasionally drifted, and it made more errors on less common Education Bureau policy names. Once both models were given retrieval over the school's own documents, the second gap almost disappeared.

His conclusion was to use the small model with a knowledge base and put the saved budget into teacher training. This is a typical and correct school decision: for most school tasks, putting the right documents in front of the model beats tripling its size. The lesson on small language models and the one on local deployment go into the practical detail.

Reading the labels

A few labels appear in model names. Recognising them is enough:

• 7B, 13B, 70B. B is billion. A 7B model is in the "runs on one graphics card" class; a 70B needs a workstation or several cards.
• MoE, or mixture of experts. Total parameters may run into the hundreds of billions, but only a small fraction activates for any one answer — rather like a school with sixty teachers where each question is routed to the two relevant subject specialists. Total parameters and parameters used per answer are two different numbers, and marketing quotes the first.
• Undisclosed. Most closed flagship models, including the current top tiers from the major providers, do not publish a parameter count at all. If someone is comparing the parameter count of a closed model, that figure is almost certainly an outside estimate.

Parameter count also drives hardware requirements, and the quantisation lesson explains how to squeeze the same model into less memory.

Try it yourself

To feel the difference between patterns and entries, paste this into any AI tool:

From memory, write out in full the complete marking criteria for the Chinese Language paper of the Hong Kong Diploma of Secondary Education. Do not summarise or paraphrase.
If you cannot, explain instead which kinds of thing you do remember and which kinds you do not, with one example of each.

Models usually admit that what they hold is the general structure and common criteria rather than the wording. That admission is a fair description of what parameters are.

What this means for your classroom

1. Do not select on parameter count. Ask instead whether it handles Traditional Chinese and Hong Kong usage, whether data can stay on campus, and whether there is teacher monitoring and an audit trail. Those three questions matter more than any number.
2. Supply documents for anything school-specific. Teaching schedules, school rules, rubrics, circular templates — all of it belongs in a knowledge base the AI can cite, not in something it is expected to remember.
3. A small model with good documents usually beats a large model with a bare prompt. In an on-campus deployment this trade-off can save a six-figure hardware budget.
4. Use the brain-cell analogy when explaining to parents and colleagues. It is accurate enough, and it leads naturally to the next sentence: "which is why it knows the patterns but not your school's details." That one sentence prevents a lot of misunderstanding.

Next

Parameters are a pile of numbers, but what shape do they take and what job do they do? The next lesson opens up three key components: embeddings that turn words into coordinates, positional encoding that keeps track of order, and self-attention that decides where to look.

Continue with embeddings, positional encoding and self-attention. Previous lesson: tokenisation and why Chinese costs more.