From FP16 to INT4: how quantisation shrinks models for school hardware

On a wet Tuesday in October 2026, a Form 2 class captain reads aloud from the sports-day height sheet in the PE storeroom. Column A lists every student rounded to the nearest centimetre; column B keeps one decimal place. A boy at the back calls out, "I'm one-sixty-two here and one-sixty-one-point-five there — which height goes on my report?" The PE teacher taps the page and says, "Both are you. One column just stores fewer digits."

Quantisation does the same thing to a language model's billions of stored weights: each number is rounded to a coarser scale so the file shrinks and ordinary hardware can load it. This is module 5, immediately after LoRA and QLoRA showed how to fine-tune on a compressed base, and immediately before Ollama and LM Studio walks you through pulling a tagged build onto a school machine.

---

Height charts and stored precision

Every year the school nurse records heights. You could write 162.347 cm for each student, or round to 162.3 cm, or round again to 162 cm. Nobody grows a different body — you just keep fewer digits on the chart. Rank order stays roughly the same; only edge cases change when two students sit half a millimetre apart.

A model holds billions of parameters (lesson 4). Quantisation is the same rounding decision applied to those stored weights:

• FP16 — full half-precision storage, the most digits kept, the largest footprint and the highest VRAM demand;
• INT8 — about half the file size, like dropping from tenths to whole centimetres; everyday language tasks often look unchanged;
• INT4 — about a quarter of the original size, one more step of rounding; faster inference and a smaller disk image, but fine judgements fail sooner.

A seven-billion-parameter model at FP16 commonly needs mid-teens gigabytes of VRAM just to load weights and run a modest context window. The same architecture at 4-bit often fits in a few gigabytes — the gap that turns a six-figure accelerator quote into a consumer graphics card in the IT cupboard.

What degrades first

Rounding error is not evenly distributed across school work. What slips first is usually:

• long chains of reasoning where a small mistake in step three poisons step seven;
• precise arithmetic and working that must be checked line by line;
• rare vocabulary, dialect words and minority languages that the model saw less often during training.

Rewriting a parent notice, smoothing a paragraph and drafting five concept-check questions often survive INT4 without a teacher noticing. That is why blind testing matters more than reading a vendor brochure.

Quantisation is not a free lunch — it is a priced trade: you buy hardware headroom and offline resilience by accepting coarser stored weights, and the invoice arrives in tasks that need exact intermediate steps.

Splitting work by subject

One primary school later blind-tested 36 real items — twelve General Studies explanations, twelve Chinese comment drafts, six mathematics word problems and six circular rewrites. Two teachers scored outputs blind as usable as is, small edits or heavy edits:

• General Studies explanations and circular drafts: little visible difference across FP16, INT8 and INT4;
• Chinese comment drafts: INT4 started repeating the same praise phrases by item eighteen;
• step-by-step mathematics: INT4 produced confident but wrong intermediate steps on two of six word problems while INT8 missed none.

The routing rule they wrote down: on-campus quantised models for text polish and drafts; mathematics, mark-bearing feedback and long reasoning on cloud or uncompressed models. Match precision to task, not one global setting for the whole school.

Rough VRAM arithmetic

A back-of-envelope estimate: size in GB ≈ parameter count in billions × bytes per parameter + 30–50% headroom for context and runtime. At 4-bit each parameter costs roughly half a byte, so seven billion parameters land near 3.5 GB of weights, or about 5–6 GB once headroom is included. Doubling context length often pushes VRAM up faster than teams expect, which is why the smallest quantised build plus a long chat history can still overflow a card that looked comfortable yesterday.

Ollama tags and GGUF files

School pilots often install Ollama, where the quantisation level is baked into the model tag:

ollama list
ollama pull qwen2.5:7b-instruct-q4_K_M
ollama pull qwen2.5:7b-instruct-q8_0
ollama run qwen2.5:7b-instruct-q4_K_M

q4 and q8 name the bit width; suffixes such as K_M identify the compression recipe inside the file. K_M is a common school starting point — smaller than the highest-quality export, gentler on VRAM than the most aggressive 4-bit recipe. The on-disk container is usually GGUF; see lesson 12 for how that format differs from the original training checkpoint.

When IT asks "are these the same model?", the honest answer is: same architecture and training lineage, different rounding on the stored weights — like two height columns on one sheet.

On-campus quantisation inside Edor.ai

On-premises deployment in Edor.ai supports local inference at INT8 or INT4: one server with a consumer graphics card can serve staff across the campus network, keep running when the external link fails, and keep prompts on the LAN.

Providers switch per module among OpenAI, Azure OpenAI, Anthropic, Poe and local Ollama — routine low-risk drafting on the quantised on-campus build, marking rubrics and multi-step reasoning on cloud. Broader deployment context lives in on-premises versus cloud and open versus closed trade-offs.

After every quantisation change — new Ollama tag, driver update, card swap — rerun a fixed school question set and log results in teaching evaluation notes. Useful for grant reporting, and it stops staff meetings from debating memory. Keep the items in a shared folder with the quantisation tag and date in the filename, for example gs-q4-blindtest-2026-09.txt. When a vendor ships an updated base model, you rerun the same file instead of arguing from recall.

For structured comparison methodology, see lesson 27.

Places that go wrong

• Wrong turn: one INT4 setting for every module to maximise card savings. Self-check: blind-test at least thirty mixed items spanning mathematics, Chinese comments and circulars; if mathematics or reasoning scores land on heavy edits, split workloads instead of squeezing harder.
• Wrong turn: trusting published benchmark scores instead of school items. Self-check: prepare at least twenty real classroom prompts — the exact ones teachers already use — and retest after every tag change; MMLU does not know your rubric.
• Wrong turn: sending mark-bearing feedback through a quantised build with no teacher review. Self-check: anything that affects grades keeps human review, or routes to an uncompressed or cloud model regardless of how confident the output sounds.

Try it once

Time: about twelve to fourteen minutes. Materials: a machine with Ollama or any runner that can load two different quantisation tags; if local setup is unavailable, compare two cloud endpoints with different context limits as a rough stand-in.

1. Write one mathematics word problem that requires step-by-step working and name the year group it targets.
2. Run it once at 8-bit (qwen2.5:7b-instruct-q8_0 or equivalent) and once at 4-bit (qwen2.5:7b-instruct-q4_K_M or equivalent).
3. Hand in: both full outputs side by side, a highlighted line where trust breaks down (if any), and one routing sentence — for example, "4-bit for circular drafts, cloud for Form 3 algebra working".

You got it right if you can point to a specific faulty intermediate step rather than saying "it got dumber". If both outputs match, note which task type that was and keep the tag for that workload. Next up: getting a build running locally in Ollama and LM Studio. Previous lesson: LoRA and QLoRA.