Reading AI leaderboards — MMLU, GPQA and Chatbot Arena

At the IT team weekly meeting, the colleague handling procurement spread an A3 leaderboard on the table. "This one is first — shall we sign today?" The Chinese panel representative turned a page and asked, "Is there a column that tests whether its Secondary-level composition comments read like something our panel would actually write?" Five seconds of silence. Nobody could point to that row.

A public leaderboard is like a territory-wide public examination ranking: the numbers are real, but the question they answer may not be yours. This lesson follows privacy, bias and copyright and explains what MMLU, GPQA, Chatbot Arena and LLM-as-a-Judge each measure — and why the final call still belongs to your own thirty questions from the panels.

---

What a leaderboard actually measures

A leaderboard answers: under fixed question types and fixed scoring, which model performs best on average. It does not answer whether the model's Traditional Chinese circulars read naturally, whether Cantonese oral feedback is accurate, or whether step-by-step mathematics explanations suit Secondary 3 level.

Picture choosing a school for your child from a public examination ranking table alone. The school at the top genuinely is strong, but the table does not measure ethos, teacher-student relationships or extracurricular life, and it does not measure whether the environment suits this child. Some schools optimise for the ranking through selective admission and concentrating resources on counted subjects — the scores are real; what they represent has shifted. AI leaderboards carry the same four limits: someone else's scenarios, someone else's preferences, past performance, and question types that can be gamed.

A leaderboard narrows twenty candidates to three; your school-based items cast the final vote.

Three public benchmarks

MMLU

MMLU is a multiple-choice paper spanning dozens of subjects, from secondary to professional level. Its strength is breadth and easy comparison; its weakness is the distance between multiple choice and real classrooms — picking the right option is not the same as explaining it clearly to a Primary 5 class. It arrived early, and most current models now cluster near the top, so its power to discriminate has faded. Use it as a rough filter for "does this model have broad baseline knowledge", not as proof it suits your school.

GPQA

GPQA is graduate-level science question answering, deliberately designed so a search engine does not help — even a well-educated non-specialist with internet access struggles. It measures deep multi-step reasoning. Direct relevance to schools is limited — few teachers need graduate chemistry answered — but steady GPQA performance is an indirect signal that a model is less likely to skip steps in mathematics word problems and logical chains. That is a hint, not a procurement decision.

Chatbot Arena

Chatbot Arena sets no fixed paper. Real users see answers from two anonymous models side by side, vote for the better one, and a chess-style rating produces the ranking. Its strength is actual human preference; its weakness is that voters are not your teaching staff, and human preference has its own tilt — longer, tidily formatted, confidently phrased answers tend to win even when they are no more accurate. For schools, Arena rankings help you understand what general users find pleasant; they do not replace school-based evaluation.

Using AI to judge AI

When there are too many items for humans to grade, the industry appoints a strong model as judge — LLM-as-a-Judge. It is cheap, fast and repeatable, which suits batch comparison inside a school.

Known judge preferences

Between two equally correct answers, the longer one tends to score higher; headings and bullet points look more "professional"; swapping the order of two answers can change the verdict (position effect); judge models sometimes rate models with a similar style more generously. These are not bugs — they are biases you must assume when using LLM-as-a-Judge.

How to run a school evaluation

Practical remedies are simple: run every pair twice with order swapped; write explicit criteria (for example, "could this comment go to parents unchanged" and "does it cite local facts correctly") instead of only asking "which is better"; calibrate with a small human sample — ten teachers blind-read ten items, and if agreement with the AI judge falls below seventy per cent, do not let the AI judge decide alone.

Why scores get inflated

Data contamination is the main mechanism. Public test questions circulate online, models read them during training, and a correct answer at test time may only be recall. That is not necessarily deliberate cheating by a vendor, but the score still overstates real capability. Items created after a model's release date, newly published benchmarks and a school's own private question set therefore carry more weight than old chart positions.

There is softer gaming too: when the whole industry watches a handful of benchmarks, models keep improving on those question types while nothing pressures improvement where nobody measures — such as whether Traditional Chinese phrasing sounds natural. The conclusion is not that leaderboards are fake. It is that a high score only means "worth shortlisting" — it cannot make the final decision for you.

Thirty to fifty school-based items

One secondary school's approach is worth borrowing. Two afternoons produced forty items — five to ten from each of five panels, all drawn from work teachers genuinely do: three junior secondary compositions asking for three improvement suggestions and no score; English vocabulary generation and composition feedback; five Secondary 3 word-problem step-by-step explanations; five General Studies concept explanations requiring Hong Kong local references; three draft circulars and two meeting-note summaries.

Scoring was deliberately crude: two teachers read blind and answered only "usable as is / needs small edits / needs heavy edits". The result contradicted the chart on the meeting table — a model placed well down the ranking was clearly better at Traditional Chinese feedback and circular drafts, while the top-ranked model did lead on step-by-step mathematics. The school's decision was not "which one" but which model for which module.

That set of forty items became a standing tool, rerun whenever a vendor updates a model, quantisation level changes (lesson 21), or fine-tuning is considered (lesson 20). Edor.ai lets administrators switch provider per module; ai_audit_logs records model and usage per interaction for a comparable baseline; every edit a teacher makes during marking review adds human samples to the evaluation set. For model specifications see the AI model database and the 2026 all-model comparison.

Where people go wrong

Treating first place on a chart as the procurement decision. Self-check: does your evaluation set include at least one item that tests "this scenario"? If forty items include zero on Cantonese or Traditional Chinese feedback, the chart has almost no reference value for you.

Items drawn from imagined tasks rather than real planning. Self-check: for each item, can you name which teacher, which week and which class would use it? If not, rerunning the evaluation will not change the procurement conclusion.

No baseline rerun after a model change. Vendors quietly update versions; the school adjusts quantisation — without rerunning the same items, you are left with impressions. Self-check: in ai_audit_logs or your own records, is there a snapshot of "date, model, forty-item results"?

Try it once

What to do: Spend fifteen minutes drafting the first five items of your school evaluation set and running two models on each.

Step 1 (5 minutes): From work you will genuinely do this week, write five paste-ready prompts — for example, "give three improvement suggestions for this Secondary 2 composition, no score" and "summarise this meeting note into a five-sentence circular". Tag each with subject and difficulty.

Step 2 (7 minutes): Pick two models that differ clearly on a leaderboard (they need not be the top two) and run all five prompts on each.

Step 3 (3 minutes): For each result, choose one of three: usable as is / needs small edits / needs heavy edits. Count which model has more "usable as is" results.

What to hand in: A sheet listing the five items, the three-way verdict for both models, and one sentence on "which model is worth testing further in which subject".

How you know it worked: All five items come from real teaching scenarios; each three-way verdict has a concrete reason, not "felt better"; the conclusion points to "which model for which module" rather than "which is best". If both models score "needs heavy edits" on all five, the prompts are too hard or too vague — fix the items before comparing models.

The final lesson of the course looks further ahead: multimodality, reasoning models and embodied AI.