Automated evaluation and leaderboards, from MMLU to Chatbot Arena

Short answer: a leaderboard is like a public examination ranking. It tells you something real, and it will not tell you whether that school suits your child. Read the rankings, but let your own thirty questions make the decision.

It starts with a procurement meeting

A vice principal put a printed leaderboard on the table: "This model is first. Shall we use this one?"

The IT coordinator asked one question: "Is there anything on that chart that measures how natural its Traditional Chinese writing sounds?"

Silence. The chart had mathematics, science, code and a multilingual average, but nothing that answered "does the feedback it writes for our Chinese teachers read like something a person would write".

That does not make leaderboards useless. It means the question they answer is not the question a school is asking.

The public examination ranking analogy

Picture a school ranking table. It is real data, computed from real public examination results, and the school at the top genuinely is strong. But if you chose where to send your child on that table alone, what would you miss?

• The table does not measure school ethos, teacher-student relationships, extracurricular life or home-school communication.
• The table does not measure whether it suits your child. Some children flourish in an intensely competitive environment and some wilt.
• Some schools optimise for the ranking through selective admission and concentrating resources on the subjects that count. The scores are real; what they represent has shifted.
• The table describes the past, and the school has a new principal this year.

The four limitations of an AI leaderboard map onto those four points one for one.

Three common leaderboards and what each measures

MMLU is a multiple-choice paper spanning dozens of subjects, from secondary to professional level. Its strength is breadth and easy comparison; its weakness is the distance between multiple choice and real use — picking the right option is not the same as explaining it to a Primary 5 class. It arrived early, and most current models now cluster near the top, so its power to discriminate has faded.

GPQA is graduate-level science question answering, deliberately designed so that a search engine does not help: even a well-educated non-specialist with internet access struggles. It measures deep reasoning. Its direct relevance to schools is limited — few teachers need graduate chemistry — but it is a useful indirect signal, because models that do well here tend to be steadier at multi-step reasoning.

Chatbot Arena works quite differently. It sets no questions. Real users see answers from two anonymous models side by side, vote for the better one, and a chess-style rating system produces the ranking. Its strength is that it measures actual human preference rather than exam performance; its weakness is that the voters are not your teaching staff, and human preference has its own tilt — longer, tidily formatted, confidently phrased answers tend to win even when they are no more accurate.

LLM-as-a-Judge: using AI to grade AI

When there are too many items for humans to grade, the industry practice is to appoint a strong model as the judge and have it score each answer. This is LLM-as-a-Judge, and it is cheap, fast and repeatable.

It also has several known preferences that matter when a school runs its own evaluation:

• A preference for length. Between two equally correct answers, the longer one tends to score higher.
• A preference for tidy formatting. Headings and bullet points look more professional to a judge.
• Position effects. Swapping the order of two answers can change the verdict.
• Family favouritism. Judge models sometimes rate models with a similar style more generously.

The practical remedies are simple: run every pair twice with the order swapped, write out explicit criteria instead of only asking "which is better", and calibrate with a small amount of human review.

Leaderboards are gameable

This has to be said plainly, because it changes how a school should read the numbers.

Data contamination is the main mechanism. Public test questions circulate online, models read them during training, and a correct answer at test time may only be recall. This is not necessarily deliberate cheating by a vendor, but the result is the same: the score overstates real capability. It is also why newly released benchmarks, private question sets, and items created after a model's release date are especially valuable.

There is a softer form of gaming too: optimising for the shape of the questions on the board. When the whole industry watches a handful of benchmarks, models keep improving on those question types while nothing pressures them to improve where nobody is measuring — such as whether Traditional Chinese phrasing sounds natural.

The conclusion is not that leaderboards are fake. It is that a leaderboard tells you which models deserve a place on your shortlist, and cannot make the final decision for you.

An example from the classroom

One secondary school did something very simple and very effective: it built a school-based evaluation set.

Two afternoons of work produced 40 items, five to ten from each of five panels, all drawn from things teachers genuinely do:

• Chinese: three junior secondary compositions, asking for three specific improvement suggestions and no score
• English: five vocabulary exercise generations and two pieces of composition feedback
• Mathematics: five step-by-step explanations of Secondary 3 word problems
• General Studies and Liberal Studies: five concept explanations that require Hong Kong local references
• Administration: three draft circulars in English and Chinese, and two meeting-note summaries

The scoring was deliberately crude: two teachers read blind and answered only "usable as is / needs small edits / needs heavy edits". No five-point scale, no weighted averages.

The result contradicted the chart on the meeting table. A model placed well down the ranking was clearly better at Traditional Chinese feedback and circular drafts, while the top-ranked model did lead on step-by-step mathematics. So the school's decision was not "which one" but "which model for which module".

That set of 40 items became a standing tool, rerun whenever a vendor updates a model, whenever the quantisation level changes (lesson 21), and whenever fine-tuning is considered (lesson 20).

What this means for your classroom

1. Use leaderboards to filter, not to decide. Let them narrow twenty candidates to three, then let your own questions settle it.
2. Build your own thirty questions this term. They are the highest-return two afternoons in the whole AI procurement process.
3. Record a baseline or you will not notice regressions. When a vendor quietly changes a model version or the school adjusts a setting, without a baseline you are left with impressions.
4. Do not ask which model is best; ask which model for which module. Different tasks have different winners, and this is how you cut cost and raise quality at the same time.

How this works inside Edor.ai

The platform is built so that the approach above can actually be carried out rather than merely recommended.

Administrators can switch provider per module (OpenAI, Azure OpenAI, Anthropic, Poe, local Ollama), so a school can run one set of school-based questions across several models and then point composition feedback, mathematics explanations and circular drafting at whichever performed best. The ai_audit_logs audit trail records which model and how much usage each interaction consumed, giving a comparable baseline before and after a change.

There is one commonly overlooked resource too: instant marking feedback always requires teacher review, and every edit a teacher makes is itself an act of human evaluation. Accumulated, those edits are material both for the school's evaluation set and for the fine-tuning discussed in lesson 20.

For specifications and positioning across models, see the AI model database and the 2026 all-model comparison.

In summary

A public leaderboard is a useful map, but not a survey of your own terrain. It is built from other people's questions, other people's preferences and other people's use cases, and it can be gamed. What genuinely answers "which model should we use" is a list of forty items from your own panels, rerun every time something changes.

The last lesson of the course looks further ahead: what comes next, multimodality, reasoning models and embodied AI. The previous lesson is privacy, bias and copyright.