Fine-tuning school tone on one GPU: how LoRA, QLoRA and PEFT work

At a July panel meeting the Chinese head said: "We have used the same comment patterns for over a decade—'I encourage you to…', 'You should strengthen…'. Every AI draft sounds like another region and teachers rewrite each one." IT added: "And we do not want years of compositions leaving the campus network."

This opens module 5. Module 4 covered retrieval and agents; this lesson asks: when the problem is "not how our school writes" rather than "not knowing school facts", can tone be taught on a single graphics card? LoRA and QLoRA are what recently made that plausible on campus.

---

An overlay on the original score

When an orchestra rehearses, the full score runs to hundreds of pages while each player needs only one part. To adapt a piece for an inter-school competition you can:

• Reprint the entire score with every page changed—that is full fine-tuning. Thorough, and far beyond what an ordinary school can afford in compute and time.
• Clip one page of arrangement notes onto the original—the score stays untouched; the addendum says "slow the second section half a beat" and "strings down an octave". Eyes read both pages; ears hear the adaptation. Base weights frozen, only a thin adapter trained—that is LoRA, low-rank adaptation.

The whole parameter-efficient fine-tuning (PEFT) family shares one sentence: freeze the base, update a removable thin layer. For a seven-billion-parameter model, LoRA is often under one per cent of the size; what you back up and move is usually a tens-to-hundreds-of-megabytes adapter file, not the full model.

"Low-rank" means what looks like a thousand edits may be driven by a handful of principles—avoid colloquial wording, every comment needs a concrete example—so the overlay can stay extremely thin.

QLoRA: shrink the score before clipping the overlay

LoRA settles how much you train, not how much desk space the full score occupies during rehearsal—training still opens the whole model in VRAM.

QLoRA adds one step: load the base in 4-bit quantisation (see the next lesson on quantisation), cutting memory to roughly a quarter of FP16, then train LoRA on top. QLoRA = quantised base + LoRA adapter.

Rough VRAM bands

Treat these as ranges, not single figures—they depend on model size, sequence length and batch:

• Under three billion parameters with QLoRA: about 8–12 GB; a mid-range gaming card often suffices;
• Seven to eight billion: about 12–24 GB, where most schools land;
• Thirteen billion and above: usually from 24 GB upward, with shorter input sequences to trade for space.

One RTX 4090 (24 GB) already covers almost every genuine school scenario. The real bottleneck is whether anyone will curate reviewed examples.

Fine-tuning teaches how to write, not what to write—worth pinning beside the panel whiteboard.

Four steps at one secondary school

Back to that Chinese panel. They proceeded as follows:

1. Collect: three teachers picked 260 compositions and original comments from two years of records; the panel head reviewed each and removed names, classes and identifying detail.
2. Format: each row became input = composition plus task requirements, output = teacher comment. This step consumed about 65% of the effort.
3. Train: QLoRA overnight on one machine in the IT room, about nine hours (seven-billion-parameter model, rank 16).
4. Blind compare: before term, the same fresh compositions went to both the base and fine-tuned models; three teachers picked which read more like "something our school would write". The fine-tuned model won clearly on tone and format, not on judging depth of argument.

That is exactly what you should expect: fine-tuning calibrates style, not professional judgement.

When not to fine-tune

• New facts (calendar, discipline policy) → use RAG;
• Occasional format drift → tighten prompt engineering first;
• Requirements change every month → fine-tuning freezes behaviour;
• No human-reviewed examples → you only train bad habits in more firmly.

How Edor.ai on-premises connects

Schools choosing on-premises deployment can run LoRA / QLoRA on their own server—a single RTX 4090-class consumer card is enough for small and mid-sized open-weight models, with training data, adapters and inference staying on the campus network. Teacher tools and marking records export in one click as JSON for dataset building; the fine-tuned model returns through local Ollama with an unchanged teacher UI.

We still recommend starting with the school knowledge base: RAG covers roughly eight needs in ten; fine-tuning is for panels with unusually strict tone. Deployment comparison: on-premises versus cloud.

Where people go wrong

• Wrong turn: fine-tuning school policy text. Self-check: ask the tuned model to quote the third paragraph—if it cannot cite reliably, switch to the knowledge base.
• Wrong turn: fine-tuning before shared prompt templates exist. Self-check: standardise templates for two weeks first; templates become future training examples.
• Wrong turn: judging only the target task, no regression test. Self-check: rerun five tasks the base already handled well and watch for catastrophic forgetting.

Try it once

Time: about 12 minutes. Materials: any AI tool.

1. Paste:

Below is an example of ideal Chinese-panel comment tone (~80 characters):
"I encourage you to add more sensory detail in the narrative; strengthen links between paragraphs; next time try echoing the question keyword in the closing."
Using that tone, write one comment on this opening sentence (max 60 characters, no explanation):
"Today I was very happy going back to school because I saw my classmates."

2. Run the same prompt again with this line added at the top: "Match the example tone, but the content must respond to the pupil text."
3. Hand in: both outputs plus one sentence from you: "If all 120 teachers need this tone, would you change prompts, build retrieval, or collect examples for LoRA—and why?"

You got it right if the second run feels closer to the example and your sentence reserves LoRA for stable tone with many reviewed examples. Next lesson unpacks the Q in QLoRA: quantisation.