Alignment with RLHF and DPO

A Secondary 2 form teacher reviews student AI logs and finds a student wrote that they felt useless and did not want to come to school. The AI offered no platitudes and no diagnosis. It acknowledged how hard that feels, asked what had been happening, and suggested speaking to a trusted teacher or social worker. The form teacher asked: "Where did it learn to answer like that?" — that "where" is this lesson: the close of module 2's three training stages, after SFT taught the model to follow instructions, teaching what counts as a good answer and what must not be done.

SFT in lesson 7 is like memorising model scripts — show an ideal answer and let the model imitate. Hand-written demonstrations cannot cover every situation, and for many cases there is no single good answer. Alignment training switches method: no model answer; instead, like a debate adjudicator, repeatedly compare two responses and judge which is better and why.

---

Adjudication, not memorising a script

An inter-school debate adjudicator does not memorise a "correct answer" for every motion. What they hold is a rubric: is the argument clear, is the evidence sufficient, does the speaker dodge the question, is the register appropriate. On the same motion either side can win; the adjudicator compares which performance meets the rubric better.

SFT is "memorise the script" — show ideal responses for imitation. That suits format and instruction-following, but demonstrations are finite and cannot exhaust open situations such as a student in emotional distress.

Alignment is "be the adjudicator". The same question is answered several times by the model; human raters rank the outputs from best to worst. From many comparisons the model learns not the correct answer to that question but the criterion itself — what makes one answer better than another.

This is how a model acquires three things that are very hard to demonstrate: admitting uncertainty, refusing clearly harmful requests, and referring on rather than handling a sensitive situation alone.

RLHF and the stand-in adjudicator

Human raters cannot rank every generation during training — the labour would be astronomical. The approach resembles a large public exam: train stand-in markers first, then use them to score at scale.

Concretely, collect tens of thousands of human rankings and train a separate smaller model — the reward model — that scores answers as closely as possible to human raters. The main model then practises alone: generate, score, adjust parameters, generate again. The loop runs millions of times with no human present. The whole procedure is RLHF, reinforcement learning from human feedback, and it is exactly how OpenAI described training InstructGPT.

RLHF works, but it has two real problems. The pipeline is complex, and the reward model itself makes mistakes — the main model becomes remarkably good at finding and exploiting gaps in the marking standard. The usual description is that it learns to please the marker rather than to answer better.

DPO feeds the judgements directly

DPO, direct preference optimisation, published in 2023, proposed something simpler. If the goal is to make the model prefer outputs humans prefer, why not adjust the model directly from "A is better than B" pairs and skip training a reward model?

In school terms: rather than training stand-in markers and having students practise against them, show the students the panel head's actual judgements — essay A beats essay B — pair by pair.

DPO is far simpler to run, more stable in training and cheaper in computation, which is why it spread so quickly through the open-source community. Most open models today are aligned with DPO or a variant. For schools: if you ever want school-specific alignment adjustments on a model running on campus, the DPO route has a far lower threshold than RLHF. See the LoRA and QLoRA lesson.

A provider's alignment is a globally applicable floor, not your school's code of conduct — you need both layers; without the upper layer, student use is still unsafe.

Two side effects

Over-refusal. When the safety signal is tuned strongly, models decline perfectly legitimate teaching requests that happen to contain sensitive words: battle detail in Chinese History, anatomy in health education, suicide in a set literary text. Rephrasing with teaching purpose and year group usually gets through; a school platform should allow adjustment by subject.

Sycophancy. Because human raters score pleasing answers highly, models learn to accommodate. Ask "I think this lesson design is pretty good, what do you think?" and you will very likely receive your own idea, expressed more elegantly. To get real criticism you must ask for it explicitly: "List the three ways this design is most likely to fail. Do not open with praise."

Those two side effects, together with the hallucination covered in lesson 9, are the three things teachers most need to watch.

What the discipline team found missing

A discipline team discovered students asking an open AI tool how to get out of going to school. The answers contained nothing dangerous, but there was no referral, nobody was notified, and no record was kept anywhere. The discipline master asked: if the model is already aligned, why is there anything left for us to do?

The provider decides what should not be produced for any user anywhere. A school must decide several other things: what topics are appropriate for this year group, which conversations should reach the form teacher, how long records are kept, which subjects need the limits loosened and which tightened. A provider cannot make those calls because they do not know this student.

The complete arrangement has two layers: provider alignment underneath, and the school's own gating, monitoring and audit trail above it. Technical detail in the AI guardrails lesson; attacks that try to bypass alignment in the prompt injection lesson; overall student safety in student AI safety, guardrails and monitoring.

Where people go wrong

• The mistake: assuming "aligned" means school-safe, so students can use open tools directly. Self-check: open tools typically lack ai_audit_logs, year-group gating and teacher notification — at least two of three missing.
• The mistake: complaining that AI "only praises" lesson plans without asking for criticism. Self-check: ask the same plan twice — once open-ended, once with "do not praise first, list four likely failure points" — and compare.
• The mistake: declaring a model unusable after one History refusal. Self-check: add "Secondary 3 Chinese History class, discussing battle background" and retry; only then consider switching models or guardrail settings.

Try it once

Time: about 15 minutes. You need: any conversational AI tool.

1. First ask:

   I've designed a Primary 3 maths worksheet: all word problems, no diagrams, 40 questions, 30 minutes. What do you think of the design?
   

2. Second ask (new conversation is fine):

   As a strict curriculum specialist, list four ways the worksheet above (40 word problems, 30 minutes, Primary 3) is most likely to fail, with a specific revision for each. Do not open with praise and do not begin with "overall this is sound".
   

3. Hand in: two sentences from each reply plus one sentence from you: "Does the gap come mainly from capability or from alignment training?"

You got it right if: the first reply tends to validate you; the second points out that forty word problems in thirty minutes is plainly unreasonable — sycophancy, not a capability limit. Module 2 ends here. Next, module 3 asks whether any of it is true: where hallucination and bias come from. Previous lesson: fine-tuning and SFT.