Alignment with RLHF and DPO

Short answer: the previous stage was copying from a worked example. This stage is judging which of two answers is better against a rubric — and from a great many such judgements the model learns what a good answer is.

A student asked something they should not have had to ask alone

A Secondary 2 form teacher reviewing student AI usage logs found an exchange. The student had written that they felt useless and did not want to come to school. The AI did not offer platitudes and did not attempt a diagnosis. It did three things: acknowledged how difficult that feels, asked what had been happening recently, and suggested speaking to a trusted teacher or social worker.

The form teacher's reaction was: how did it know to answer that way?

Not from pre-training — the internet is full of responses to that sentence, good and bad alike. Not entirely from the demonstration fine-tuning of lesson 7 either, since hand-written demonstrations cannot cover every situation. It came from alignment training.

From copying a model answer to marking against a rubric

Every teacher who has marked essays knows the difference between two approaches.

The first is a model answer. Students match it word for word and score full marks. That works for cloze and fails completely for writing, because there are unlimited ways to write a good essay.

The second is a rubric. You do not write the answer in advance. You set criteria — clear position, sufficient evidence, sound structure, appropriate register — and then, looking at two essays, you judge which is better and why.

SFT in lesson 7 is the first approach: show the ideal answer and let the model imitate. Its limits are obvious. There is only so much a person can write, and for many situations there is no single good answer.

Alignment uses the second approach. The same question is answered several times by the model, and human raters rank the outputs from best to worst. What the model learns is not the correct answer to that question but the criterion itself — what makes one answer better than another.

This is how a model acquires three things that are very hard to demonstrate: admitting uncertainty, refusing clearly harmful requests, and referring on rather than handling a sensitive situation itself.

RLHF: first train a stand-in marker

The obstacle is that human raters cannot possibly rank every generation during training. The labour required would be astronomical.

The solution resembles how a school handles a mountain of marking: train some markers first.

Concretely, collect tens of thousands of human rankings and use them to train a separate, smaller model that does one job — look at an answer and give it a score as close as possible to what a human rater would give. That smaller model is the reward model.

With a reward model in place, the main model can practise on its own: generate an answer, get it scored, adjust the parameters, generate again. That loop runs millions of times with no human present. The whole procedure is RLHF, reinforcement learning from human feedback, and it is exactly how OpenAI described training InstructGPT.

RLHF works, but it has two real problems. The pipeline is complex, and the reward model itself makes mistakes — rather like a stand-in marker whose standards drift from the real examiner's, except that the model becomes remarkably good at finding and exploiting that drift. The usual description is that the model learns to please the marker rather than to answer better.

DPO: cut out the stand-in marker

DPO, or direct preference optimisation, published in 2023, proposed something simpler. If the point is to make the model prefer outputs that humans prefer, why not adjust the model directly from the "A is better than B" pairs and skip training a reward model at all?

In school terms: rather than training a set of stand-in markers and then having students practise against them, show the students the panel head's actual judgements — this essay is better than that one — pair by pair.

DPO is far simpler to run, more stable in training and cheaper in computation, which is why it spread so quickly through the open-source community. Most open models today are aligned with DPO or a variant. That has a practical consequence for schools: if you ever want school-specific alignment adjustments on a model running on campus, the DPO route has a far lower threshold than RLHF. See the LoRA and QLoRA lesson.

Two side effects of alignment

Over-refusal. When the safety signal is tuned strongly, models decline perfectly legitimate teaching requests that happen to contain sensitive words: battle detail in Chinese History, anatomy in health education, suicide in a set literary text. For teachers this is a genuine nuisance.

Sycophancy. Because human raters score pleasing answers highly, models learn to accommodate. This matters enormously for teaching. Ask "I think this lesson design is pretty good, what do you think?" and you will very likely receive your own idea, expressed more elegantly. To get real criticism you have to ask for it explicitly: "List the three ways this design is most likely to fail. Do not open with praise."

Those two side effects, together with the hallucination covered in lesson 9, are the three things teachers most need to watch.

An example from the classroom

A discipline team discovered students asking an open AI tool how to get out of going to school. The answers contained nothing dangerous, but there was no referral, nobody was notified, and no record was kept anywhere. The discipline master asked the key question: if the model is already aligned, why is there anything left for us to do?

The answer is that a provider's alignment is a globally applicable floor, not your school's code of conduct.

The provider decides what should not be produced for any user anywhere. A school has to decide several other things: what topics are appropriate for this year group, which conversations should reach the form teacher, how long records are kept, which subjects need the limits loosened and which tightened. A provider cannot make those calls, because they do not know this student.

So the complete arrangement has two layers: provider alignment underneath, and the school's own gating, monitoring and audit trail above it. The technical detail of that upper layer is in the AI guardrails lesson, and the attacks that try to bypass alignment are in the prompt injection lesson. For the overall approach to student safety, see student AI safety, guardrails and monitoring.

Try it yourself

To test sycophancy directly, paste the following into any AI tool as two separate questions:

First ask: I've designed a Primary 3 maths worksheet. It's all word problems, no diagrams, 40 questions, 30 minutes. What do you think of the design?
Then ask: Acting as a strict curriculum specialist, list the four ways the worksheet design above is most likely to fail, with a specific revision for each. Do not open with praise and do not begin with "overall this is sound".

The two answers usually differ sharply. The first tends to validate you; only the second points out that forty word problems in thirty minutes is plainly unreasonable. That gap is not a limit of capability. It is a tendency left behind by alignment training.

What this means for your classroom

1. If you want criticism, ask for criticism explicitly. In planning, curriculum design and assessment review, adding "do not praise first, list the problems" transforms the quality of the feedback.
2. Do not mistake provider safety settings for school safety. Student-facing use needs its own gating, monitoring and audit trail, with the limits set by teachers.
3. When you hit an over-refusal, check the wording first. Stating the teaching purpose, year group and curriculum context gets most legitimate requests through.
4. Be suspicious when the AI agrees with you. Especially on a lesson plan you have already invested in, its approval carries almost no information.

Next

Alignment deals with what should be said. A more basic question remains untouched: is any of it true? The next lesson takes on hallucination and bias — neither of which is a bug, and both of which follow structurally from how this technology works.

Continue with where hallucination and bias come from. Previous lesson: fine-tuning, SFT and instruction-following.