Model cards and file formats explained

After the previous lesson the IT coordinator actually opened a model page and held up his tablet in the staff room: "Fourteen files — Q4_K_M, Q5_K_S, Q8_0, and something called safetensors. Which one do I drag into Ollama?"

This lesson follows the Hugging Face "specimen cabinet". Once you have read the labels, the next step is the manual and the codes on the boxes.

---

Read the manual before you plug it in

Buy a lab projector and you flip through the warranty card and usage limits before switching it on. Public models deserve the same habit. The model card inside the repository is the manual the developers wrote for users, and it usually covers:

• What the model is: how many parameters, which architecture, how long a context it supports.
• Training data: what material it learned from, which languages it covers, when the data stops.
• Intended and out-of-scope uses: what the developers designed it for, and what they say it is not for.
• Limits and biases: situations where it is known to fail, and groups it may treat unevenly.
• Evaluation results: scores on standard benchmarks.
• Licence: whether commercial use is permitted, and whether there is a cap on organisation size.

A four-step read in five minutes

As a teacher or panel head you do not need every word. Check four places in this order:

1. Language first. Many strong models state honestly on their own card that they target English and are not recommended for multilingual use. For a Hong Kong school, that single sentence can rule an option out.
2. The data cut-off date. This decides whether it knows about last year's curriculum changes and the current public examination arrangements. See knowledge cut-offs and model versioning.
3. The limits section. The weaknesses a developer admits to are usually exactly the ones a school will step on, such as "may produce content that sounds reasonable but is inaccurate".
4. The licence. Is institutional use permitted? Is there a cap on users or pupils?

If a model card holds one paragraph of marketing, no training data description and no statement of limits, that absence is the most important information available: this model does not belong in a school's formal workflow.

Baggage tiers and GGUF codes

Now for those fourteen files.

Before a flight you check baggage allowances: 20kg full fare, 15kg economy, 7kg carry-on. The same clothes packed differently fit different limits. GGUF is that kind of packing for model weights: compress and bundle them so they can run on an ordinary computer or a single GPU. One model appears in several GGUF versions, and the code in the filename tells you the compression level:

• Q8_0: barely compressed, closest to the original, uses the most memory.
• Q5_K_M: a middle option with a good balance of quality and size.
• Q4_K_M: the most common practical choice, roughly a quarter of the original size, with quality loss that is usually imperceptible.
• Q2_K: heavily compressed. It runs, but output degrades noticeably and it is generally not advisable.

Matching GPU memory

A useful rule of thumb: ask how much GPU memory your machine has, then pick the version whose file size sits comfortably below it. Original Safetensors often will not fit into 16GB once the conversation itself needs room. Why compression works at all, and where it starts making a model noticeably worse, is covered in quantisation from FP16 to INT4. How to run it in school is in local deployment with Ollama and LM Studio.

GGUF also packages the model's configuration alongside the weights in a single file, so it behaves like a ready-to-open parcel rather than a set of parts that must be assembled correctly.

Safetensors versus older formats

The other common format is Safetensors, and it solves a safety problem rather than a size problem.

The older format for storing weights could technically carry executable instructions inside the file. In other words, while you thought you were loading a pile of numbers, you might also be running somebody else's code. For a school server that is a real risk.

The design principle of Safetensors is simple: the file may contain numbers only, never instructions. Like a lab that accepts only clearly labelled reagent bottles rather than mystery mixtures. It also loads faster. It is now the recommended format, so a school downloading weights manually can relax on seeing Safetensors, and should ask about provenance on seeing an older .bin or pickle file.

During procurement, "please send the model card" often beats ten pages of brochure. An inability to answer is itself the answer.

Choosing files for a primary server

A primary school received funding and bought a server with a single consumer GPU carrying 16GB of memory, intending to run Chinese writing feedback in-house. The IT coordinator worked through it like this.

He read the model cards first, confirming that the shortlisted model's card stated support for Traditional Chinese, a recent data cut-off, and a licence permitting educational institutions. That step eliminated two candidates that had looked strong, because their own cards conceded that Chinese performance was mediocre.

Then he looked at the file list. The original Safetensors files came to 16GB, which does not fit into 16GB of GPU memory once you leave room for the conversation itself, so he took the Q4_K_M GGUF version at about 4.9GB. The remaining memory was enough to serve several teachers at once.

Finally he tested it, running twenty de-identified old compositions through and asking three Chinese teachers to blind-rate the AI feedback. The conclusion was that it was usable for a first pass but still needed teacher review, which is exactly how we configure AI marking in the platform.

Not one line of code was written. The whole decision rested on reading a manual and a set of filenames.

Common misreadings

• The claim: The smallest Q2 file is the smartest choice because it saves memory. What actually happens: Over-compression produces visible errors in classroom feedback. Self-check: have you compared Q4 and Q2 on the same prompt?
• The claim: "On-premises deployment" needs no documentation if the vendor says so. What actually happens: Practice and marketing separate at the model card. Self-check: can they link to the official page?
• The claim: Limits on the card are polite disclaimers you can ignore. What actually happens: A line about content that sounds reasonable but is inaccurate belongs verbatim in staff guidance, because it justifies the review requirement. See where hallucination and bias come from.

Try it once

Time: about twelve minutes. Materials: if your school already has a local model, you need lightly and heavily compressed versions; if not, read any 8B model card on Hugging Face for the first half.

Task: if you can run a model, feed the same content to lightly and heavily compressed versions using:

Sort the three pupil answers below into correct, partly correct and incorrect, and give one sentence of reasoning for each.
Judge only from the sentences I have supplied. Do not add information beyond them.
1. Most of Hong Kong's water comes from the Dongjiang.
2. All of Hong Kong's water is rainwater it collects itself.
3. Some of Hong Kong's water comes from local reservoirs.

If you can only read a model card, copy one sentence each from the language, limits and licence sections that best represent the model.

Hand in: three lines of notes: (1) was the lightly compressed version more stable than the heavy one (or does the card state Chinese support clearly); (2) is there one limits sentence you would paste into staff guidance; (3) if deploying, would you pick Q4 or Q8 and why in one line.

Done right: you can explain either the cost of compression or why missing sections on a card are warnings, and you would ask "please send the model card" in a procurement meeting.

Continue with the rise of small language models. Previous lesson: Hugging Face, the GitHub of AI. To understand what a parameter count represents, see what parameters actually store.