Quantisation, from FP16 down to INT8 and INT4

Short answer: quantisation saves the model as a smaller file. The size drops sharply, ordinary users notice nothing, and yet detail really has been lost. A school trades some accuracy for cheap hardware and offline capability — Edor.ai's on-premises deployment supports INT8 and INT4, and both sides of that ledger deserve to be stated.

It starts with a quotation

An IT coordinator at a primary school received two quotations for running an open-weight model on campus. The first specified a professional accelerator card and came to six figures. The second used a gaming graphics card at a tenth of the price. He asked the vendor: "Same model — why the enormous gap?"

The answer is one word: quantisation.

Saving a photo as a smaller JPEG

Take a photo on your phone and the raw file might be over 30 MB. Send it to a colleague on a messaging app and what arrives may be 2 MB. Same photo, less detail, and on screen most of the time you genuinely cannot tell — until you zoom right in to read the street sign in the background.

Quantisation works the same way. A model holds billions of numbers (lesson 4 explains what those numbers are). Each one is normally stored in 16 bits, precise to many decimal places. Quantisation stores them in 8 bits, or even 4, which amounts to rounding every number onto a coarser scale.

• FP16 (16-bit floating point): the raw file. Most complete, largest.
• INT8 (8-bit integer): about half the size. Like a high-quality JPEG — the eye barely registers it.
• INT4 (4-bit): about a quarter of the size. Like a medium-quality JPEG — fine day to day, visible when you zoom in.

A model with seven billion parameters needs somewhere in the mid-teens of gigabytes of graphics memory at FP16. Squeezed to 4-bit it fits in a few gigabytes. That is the tenfold gap between the two quotations.

The ruler analogy

If JPEGs are still too abstract, try another. Suppose you are recording the height of all 800 pupils in the school:

• In millimetres: 172.4 cm. Most precise, most digits to write.
• In centimetres: 172 cm. Saves space, and is perfectly adequate for lining pupils up by height.
• In five-centimetre bands: the 170 cm band. Halves the space again, still good enough for grouping, but two pupils three centimetres apart now fall in the same band.

Quantisation is adjusting the marks on that ruler. With coarser marks, the overwhelming majority of judgements are unaffected, because the model was never deciding on the fifth decimal place. But when two options were already extremely close, a coarse scale means the model cannot tell them apart, and it picks the wrong one.

This is why the loss is not spread evenly. What breaks first is always what was marginal to begin with: long multi-step reasoning, where small errors accumulate at every step; precise arithmetic; rare vocabulary and minority languages. Everyday polishing, summarising and rewriting are almost untouched.

An example from the classroom

That same primary school later ran a very simple test that every school should copy. They prepared 30 items, all from situations teachers actually face:

• 10 General Studies concept explanations, Primary 4 to Primary 6 level
• 10 Chinese composition feedback tasks with the student text attached
• 5 step-by-step explanations of mathematics word problems
• 5 draft school circulars in English and Chinese

The same items went to FP16, INT8 and INT4 versions, and two teachers judged blind, answering only "usable as is / needs small edits / needs heavy edits".

Roughly what happened: concept explanations and circular drafts sat between "usable as is" and "needs small edits" on all three versions, with little to choose between them. Composition feedback started repeating phrasing at INT4. Step-by-step mathematics degraded clearly at INT4, with two items getting an intermediate step wrong while sounding entirely confident.

The school's decision then became obvious: the on-campus quantised model handles text work, and mathematics or anything needing precise reasoning goes to a cloud model. That is not a compromise; it is matching the tool to the task.

What the hands-on part looks like

In Ollama, the environment schools most often use, the quantisation level is written into the model tag. These commands show how to pull different quantisations of the same model:

# list the models already downloaded and their sizes
ollama list

# pull a 4-bit build (the q4 in the tag is the quantisation level)
ollama pull qwen2.5:7b-instruct-q4_K_M

# pull the 8-bit build of the same model for comparison
ollama pull qwen2.5:7b-instruct-q8_0

# run each in turn and compare output on the same prompt
ollama run qwen2.5:7b-instruct-q4_K_M

How to read the tag: q4 and q8 are the bit widths, and suffixes like K_M denote different compression strategies, with K_M generally being a common balance of size and quality. These files are usually in the GGUF format introduced in lesson 12.

Hardware expectations again belong in ranges rather than single figures, because they depend on model size, bit width and context length (how much text is handled at once):

• Three billion parameters at 4-bit: in the region of 4 GB to 6 GB of VRAM.
• Seven to eight billion at 4-bit: roughly 6 GB to 10 GB; the same model at 8-bit needs about twice that.
• Long contexts, such as reading a whole document at once, push memory up further and faster than most people expect.

A useful rough estimate: model size in gigabytes is about the parameter count in billions multiplied by the bytes per parameter, plus thirty to fifty per cent headroom for context and computation. At 4-bit each parameter is about half a byte, so seven billion parameters is roughly 3.5 GB, or about 5 GB to 6 GB once headroom is included.

What this means for your classroom

1. Hardware budgets can differ tenfold while students notice nothing. For explaining concepts, rewriting sentences or generating practice items, a quantised model is usually good enough, and the saving can go into teacher training instead.
2. Do not use a quantised model where marks are at stake. Anything needing precise figures, multi-step reasoning or affecting grades belongs on an uncompressed or cloud model, with teacher review retained.
3. Offline capability is a genuine advantage. When the external link drops, when the network is congested on an inspection day, or on a school trip with no connectivity, the on-campus quantised model keeps running.
4. Retest on every version change. Change the quantisation level and behaviour changes. Rerunning your own 30 items is worth more than any published figure.

How this works inside Edor.ai

Edor.ai's on-premises deployment supports running local models at INT8 or INT4. In practical terms, one server with a single consumer graphics card, sitting in the school's own machine room, can serve local AI inference to the whole school without a professional accelerator.

It is only fair to state both sides of the trade.

• What the school gains: sharply lower hardware cost; faster inference on the same card; continued operation when the external connection fails; and data that never leaves the campus network, which satisfies even the strictest school data policy.
• What the school gives up: some accuracy, concentrated in multi-step reasoning and precise calculation; the need for IT colleagues to own updates and backups; and capability that trails the newest cloud flagships.

Because providers are switchable per module (OpenAI, Azure OpenAI, Anthropic, Poe, local Ollama), a school does not have to choose one or the other: routine low-risk work goes to the on-campus quantised model, while marking and long reasoning go to a cloud model. For the full comparison of deployment models, see on-premises versus cloud deployment and the open versus closed cost and privacy trade-off.

In summary

Quantisation is not a free lunch. It is a priced trade: some accuracy in exchange for a tenfold difference in hardware cost and the ability to work offline. Schools that understand the ledger put quantised models on suitable tasks rather than applying one setting to everything.

The next lesson actually gets a model running: running models in school with Ollama and LM Studio. The previous lesson is parameter-efficient fine-tuning.