Parameter-efficient fine-tuning with LoRA and QLoRA on one GPU

Short answer: fine-tuning a large model does not require retraining the whole thing. LoRA lays a sheet of tracing paper over the model and trains only that sheet; QLoRA shrinks the original first, so one consumer graphics card is enough. This is exactly the route Edor.ai's on-premises deployment takes, letting a school fine-tune on its own server.

It starts with a meeting in August

Mid-August, an IT coordinator at a secondary school raised a question in a planning meeting: "Our Chinese panel has used the same composition feedback format for over a decade. Every time we ask the AI to write it, the register comes out wrong and teachers rewrite it anyway. Is there a way to teach it once and for all?"

The panel head added: "And we would rather not send years of student compositions to a server overseas."

Put together, those two sentences describe the problem this lesson solves: how to turn a general model into one that follows your school's conventions, with the whole process staying on campus.

The tracing-paper analogy

Picture the model as a printed textbook, a thousand pages thick. You want to change how it phrases things. There are two ways:

• Reprint the whole book. Reset and reprint all thousand pages. That is full fine-tuning. It is the most thorough option and the machinery, electricity and time it demands are entirely out of reach for an ordinary school.
• Lay a sheet of tracing paper on top. The book stays as it is. A transparent sheet sits over it and you write your corrections only on the sheet. When reading, the eye sees the book and the sheet together and gets the corrected version. Not one word of the original changes.

The second approach is LoRA, low-rank adaptation. The whole family of parameter-efficient fine-tuning (PEFT) methods shares one core idea: freeze the original model and train only a small new piece bolted on top.

As for the word "low-rank", think of it this way. When a teacher marks a set of scripts, it looks like a thousand separate decisions, but the teacher is really applying a handful of principles — "replace colloquial wording", "every claim needs evidence" — and those few principles explain all thousand edits. LoRA assumes the changes needed for fine-tuning behave the same way: the edits are spread across the whole book, but only a very small number of underlying principles drive them. That is why the tracing paper can be so thin.

How thin? For a model with seven billion parameters, a LoRA sheet is typically under one per cent of the original. It is also why, after training, the only file the school has to store and back up is that small sheet rather than a whole model.

QLoRA: photocopy the book smaller first

LoRA settles the question of how much you train, but not where the original book sits. During training, all thousand pages still have to be laid open in the graphics card's memory (VRAM).

QLoRA adds one more step. It photocopies the original in a coarser but far more compact form — this is quantisation, squeezing each parameter from 16 bits down to 4 — and then lays the tracing paper over the shrunken copy. The original takes about three-quarters less space, and a consumer card can hold it.

Quantisation is the subject of the next lesson. For now, one sentence is enough: QLoRA is a quantised original plus a LoRA sheet. For the trade-offs, see quantisation from FP16 down to INT8 and INT4.

An example from the classroom

Back to that secondary school. What they eventually did came in four steps.

1. Collect examples. Three Chinese teachers picked 400 compositions and the feedback they had originally written from two years of marking records. The panel head reviewed each one and removed student names, class numbers and anything else identifying.
2. Shape the format. Each record became a pair: input is the student composition plus the task requirements, output is the teacher's feedback. This step is the slowest and takes roughly seventy per cent of the total effort.
3. Train. QLoRA ran overnight on one machine in the IT room with a single consumer graphics card.
4. Compare before use. Before term started, a fresh batch of compositions went to both the original and the fine-tuned model, and three teachers judged blind which feedback read more like "something our school would write".

The result is worth noting. The fine-tuned model was clearly closer to school conventions in tone and format, but no better at judging the depth of an argument. That is exactly what you should expect: fine-tuning teaches a way of speaking, not professional judgement.

What the hands-on part looks like

The skeleton of a QLoRA training configuration looks roughly like this. It is not here for you to copy and run, but so that you can follow what IT colleagues or a vendor are describing in a meeting:

base model    : an open-weight model in the 7B to 8B parameter class
loading       : loaded in 4-bit (this is the Q in QLoRA)
LoRA rank     : 8 to 32 (the thickness of the tracing paper; thicker holds
                more but overfits more easily)
LoRA alpha    : commonly twice the rank
applied to    : the projection matrices in the attention layers
learning rate : in the region of 1e-4 to 2e-4
epochs        : 2 to 3 (more when data is scarce, but watch for overfitting)
output        : an adapter file of tens to hundreds of MB, not a full model

Hardware expectations deserve an honest answer. VRAM requirements are a range, not a number, and they depend at once on model size, how many bits the quantisation uses, sequence length (how much text goes in at a time) and batch size:

• Small models under three billion parameters with QLoRA: in the region of 8 GB to 12 GB of VRAM. A mid-range gaming card usually copes.
• Seven to eight billion parameters: roughly 12 GB to 24 GB depending on sequence length. This is where most schools land.
• Thirteen billion and above: generally 24 GB as a starting point, and you trade sequence length for space.

In other words, one card in the RTX 4090 class (24 GB) already covers almost everything a school genuinely needs. The real bottleneck is never the graphics card; it is whether anyone is willing to spend the time curating those 400 examples.

When not to fine-tune

This is the most important section of the lesson and the one least often said out loud. In these situations, fine-tuning is the wrong tool:

• You want the model to know new facts — this year's school calendar, a revised discipline policy, the latest curriculum guide. Use retrieval instead; see knowledge bases and RAG.
• The output format is merely inconsistent. Try enforced structured output and a tighter prompt first; see prompt engineering in one lesson.
• The requirement changes every month. Fine-tuning freezes something in place, and things that move quickly should not be frozen.
• You have no reviewed examples. Rubbish in, rubbish out. Fine-tuning without clean data only trains bad habits in more firmly.

What this means for your classroom

1. Fine-tuning is a style calibrator, not a master key. When you find the AI's content broadly correct but "not how our school writes it", that is where fine-tuning earns its place.
2. Your marking records are an asset. The feedback and edits teachers write in the platform day to day accumulate into the material closest to your school's standard — provided it was de-identified on collection and checked by a panel head.
3. This can be done on campus. No cloud supercomputer is needed; one machine in the server room is enough. For schools whose data policy says nothing may leave the campus network, that is decisive.
4. Without a comparison you cannot claim an improvement. Test before and after blind on the same set of items, or you are only reporting a feeling. See automated evaluation and leaderboards.

How this works inside Edor.ai

Edor.ai offers cloud and on-premises deployment with identical features. Schools that choose on-premises can run LoRA and QLoRA fine-tuning on their own server. This is not a theoretical possibility but a capability the deployment supports: a single consumer card in the RTX 4090 class is enough to fine-tune a small or mid-sized open-weight model, with no supercomputer and no rack of GPUs to buy. Training data, the adapter file and inference all stay inside the campus network.

Three practical points go with it. First, teacher tools and marking records export in one click as JSON or CSV, which is the least painful starting point for building a training set — and the data belongs to the school in the first place. Second, the fine-tuned model is served back into the platform through the local Ollama provider, so the teacher interface does not change at all. Third, our default recommendation is still to start with the school knowledge base: for eight schools in ten, RAG already solves the problem, and fine-tuning is reserved for panels with unusually strict requirements.

For the wider comparison of deployment models, see on-premises versus cloud deployment for schools.

In summary

LoRA turns "reprint the book" into "add a sheet of tracing paper", and QLoRA shrinks the book first. Together they moved fine-tuning out of the data centre and into the school server room. Once the technical barrier drops, though, the real barrier becomes visible: whether you have a clean, reviewed set of examples that genuinely represents your school's standard.

The next lesson unpacks the Q in QLoRA: quantisation from FP16 down to INT8 and INT4. The previous lesson on development frameworks is LangChain and LlamaIndex.