Reading a model card, and what GGUF and Safetensors mean

An IT coordinator did go and browse a model page after the previous lesson, then sent a screenshot: "There are fourteen files here, named Q4_K_M, Q5_K_S, Q8_0, and one that says safetensors. I have no idea what any of it means. Can you just tell me which one to download?"

This lesson answers that. The good news is that you only need three ideas.

A model card is a nutrition label

Buy a box of biscuits and the back of the packet carries a label: ingredients, energy per serving, allergen warnings, who it is suitable for. You do not need to understand food chemistry to work out from that label whether you can offer the box to a pupil with a nut allergy.

Every public model should carry that kind of label, and it is called a model card. It is a documentation file inside the model repository, written by the developers, and it usually covers:

• What the model is: how many parameters, which architecture, how long a context it supports.
• Training data: what material it learned from, which languages it covers, when the data stops.
• Intended and out-of-scope uses: what the developers designed it for, and what they say it is not for.
• Limits and biases: situations where it is known to fail, and groups it may treat unevenly.
• Evaluation results: scores on standard benchmarks.
• Licence: whether commercial use is permitted, and whether there is a cap on organisation size.

Four things to check, in order

As a teacher or panel head you do not need to read every word. Check four places in this order and it takes five minutes:

1. Language first. Many strong models state honestly on their own card that they target English and are not recommended for multilingual use. For a Hong Kong school, that single sentence can rule an option out.
2. The data cut-off date. This decides whether it knows about last year's curriculum changes and the current public examination arrangements. See knowledge cut-offs and model versioning.
3. The limits section. The weaknesses a developer admits to are usually exactly the ones a school will step on, such as "may produce content that sounds reasonable but is inaccurate".
4. The licence. Is institutional use permitted? Is there a cap on users or pupils?

If a model card holds one paragraph of marketing, no training data description and no statement of limits, that absence is the most important information available: this model does not belong in a school's formal workflow.

GGUF is like a compressed video format

Now for those fourteen files.

Imagine you filmed a two-hour lesson and the original file is 80GB. To put it on the school website for parents you compress it: a 1080p version at 4GB, 720p at 1.5GB, 480p at 600MB. Same content, some detail lost, but an ordinary phone plays it smoothly.

GGUF does this for models. It is a file format that compresses and packages model weights so they can run on an ordinary computer or a single GPU. One model appears in several GGUF versions, and the code in the filename tells you the compression level:

• Q8_0: barely compressed, closest to the original, uses the most memory.
• Q5_K_M: a middle option with a good balance of quality and size.
• Q4_K_M: the most common practical choice, roughly a quarter of the original size, with quality loss that is usually imperceptible.
• Q2_K: heavily compressed. It runs, but output degrades noticeably and it is generally not advisable.

A useful rule of thumb: ask how much GPU memory your machine has, then pick the version whose file size sits comfortably below it. Why compression works at all, and where it starts making a model noticeably worse, is covered in quantisation from FP16 to INT4. How to run it in school is in local deployment with Ollama and LM Studio.

One more thing worth knowing: GGUF packages the model's configuration alongside the weights in a single file, so it behaves like a ready-to-open parcel rather than a set of parts that must be assembled correctly.

Safetensors is like an envelope that cannot hide anything

The other common format is Safetensors, and it solves a safety problem rather than a size problem.

The older format for storing weights could technically carry executable instructions inside the file. In other words, while you thought you were loading a pile of numbers, you might also be running somebody else's code. For a school server that is a real risk.

The design principle of Safetensors is simple: the file may contain numbers only, never instructions. It is a transparent envelope, where a glance confirms there is nothing inside but paper. It also loads faster. It is now the recommended format, so a school downloading weights manually can relax on seeing Safetensors, and should ask about provenance on seeing an older .bin or pickle file.

An example from the classroom

A primary school received funding and bought a server with a single consumer GPU carrying 16GB of memory, intending to run Chinese writing feedback in-house. The IT coordinator worked through it like this.

He read the model cards first, confirming that the shortlisted model's card stated support for Traditional Chinese, a recent data cut-off, and a licence permitting educational institutions. That step eliminated two candidates that had looked strong, because their own cards conceded that Chinese performance was mediocre.

Then he looked at the file list. The original Safetensors files came to 16GB, which does not fit into 16GB of GPU memory once you leave room for the conversation itself, so he took the Q4_K_M GGUF version at about 4.9GB. The remaining memory was enough to serve several teachers at once.

Finally he tested it, running twenty de-identified old compositions through and asking three Chinese teachers to blind-rate the AI feedback. The conclusion was that it was usable for a first pass but still needed teacher review, which is exactly how we configure AI marking in the platform.

Not one line of code was written. The whole decision rested on reading a label and a set of filenames.

What this means for your classroom

1. You can ask the right question before you buy. "Please send us the model card for the model you use" is a completely reasonable and highly effective request. An inability to answer is itself the answer.
2. On-premises stops being a vague promise. When a vendor says a model can run in school, you can ask which quantised version and how much GPU memory. Those two questions separate practice from marketing.
3. A model's limits belong in the teacher guidance. The line on the model card about content that sounds reasonable but is inaccurate should appear verbatim in your staff guidelines, because it is the justification for the review requirement. See where hallucination and bias come from.
4. Do not over-compress to save memory. A model squeezed to Q2 may hold up in a demonstration but will make visible errors when giving pupils feedback. Prefer a smaller model at a higher quality setting.

A small experiment to run now

If your school already has a local model, feed the same content to a lightly and a heavily compressed version using this prompt:

Sort the three pupil answers below into correct, partly correct and incorrect, and give one sentence of reasoning for each.
Judge only from the sentences I have supplied. Do not add information beyond them.
1. Most of Hong Kong's water comes from the Dongjiang.
2. All of Hong Kong's water is rainwater it collects itself.
3. Some of Hong Kong's water comes from local reservoirs.

The lightly compressed version sorts all three reliably. An over-compressed version often misjudges the third or writes vague reasoning. It is the clearest demonstration of what compression costs.

What comes next

Once you can read a model, the next question is whether a small model is good enough, which is the subject of the rise of small language models. To revisit how to find model pages, return to Hugging Face, the GitHub of AI. To understand what a parameter count represents, see what parameters actually store.