Question setting and rubric prompts
A good paper is derived backwards from the learning focus, not written from whatever comes to mind. Build the table of specifications first, then the items, then the rubric — and finally have AI review the paper you wrote.
以繁體中文閱讀Short answer: state which learning focuses the paper assesses, how many marks each carries and at which cognitive level — then ask for items. Reverse the order and you will always be rearranging the paper afterwards.
A paper that looked complete
A Secondary 3 Science teacher generated a unit test:
Write a Secondary 3 Science unit test on force and motion, 20 questions, multiple choice and short answer.
Twenty items appeared quickly, neatly laid out and professional-looking. Then she checked them against the learning focuses: fourteen tested definitions and formula recall, two involved analysing experimental data, and "interpreting a speed-time graph" — the most important focus in the unit — was not tested at all.
The paper was not bad. The distribution was wrong. And it was wrong because the prompt never mentioned distribution.
Why this works
Setting a paper involves two professional judgements: what is assessed (content) and at what depth (cognitive level). Draw those as a grid, learning focuses down the side and cognitive levels across the top, and write marks in each cell. That grid is the table of specifications.
An analogy: the specification table is to a paper what the quantities in a recipe are to a dish. Without it you may salt the same dish three times and forget the sugar entirely. AI will not work out the balance for you, because it does not know where the emphasis of your unit lay. Give it the table and it fills the cells faithfully.
So question setting is two steps: produce the table, confirm it, then produce the items. This is the task decomposition principle from lesson 4.
Template 1: the table of specifications first
You are an assessment design assistant for a Hong Kong secondary Science teacher. We will work in two steps. Do step one only.
Task (step one): design a table of specifications for a unit test.
Context: Secondary 3, unit "force and motion". Learning focus: (1) describe types of force and their effects; (2) use Newton's first law to explain everyday phenomena; (3) read acceleration and displacement from a speed-time graph; (4) analyse simple experimental data and draw a conclusion. Eight lessons of teaching time, with the emphasis on (3) and (4). The test is 40 marks in 45 minutes.
Output format: a table with the four learning focuses down the side and three cognitive levels across the top (recall and understanding / application / analysis and evaluation), each cell giving suggested marks and item count. Below the table, three bullet points justifying the allocation, plus a total-marks check.
Constraints: analysis and evaluation must carry at least 30% of the marks; do not write any items yet, that is step two; every learning focus needs at least one item; if you think a learning focus is unsuitable for written assessment, say so and explain rather than forcing marks into the row.
When the table arrives, spend a minute checking the proportions and adjust before moving to step two.
Template 2: items from the table
You are a question-setting assistant for a Hong Kong secondary Science teacher.
Task: write items according to the confirmed table of specifications below.
Context: [paste the revised table]. Students have covered: [list]. Not yet covered: [list, for example calculations using Newton's second law]. Our school's house format: four-option multiple choice, short-answer items with marks shown, and data items where the teacher adds the figure later.
Output format: work cell by cell through the table. Each item gives the number, the question, the learning focus number, the cognitive level, the marks and the answer. Multiple-choice items add a "distractor rationale" column stating which student misconception each wrong option targets. End with a total-marks check.
Constraints: use only methods and concepts students have already been taught; distractors must reflect errors students genuinely make, never filler options; for items needing a figure, describe the figure in words and mark it "figure to be supplied by teacher" rather than inventing data; where a scientific constant, unit or experimental value is uncertain, mark it "teacher to verify".
The distractor rationale column is the most valuable part of this prompt. Forcing the model to give each wrong option a pedagogical reason makes the filler options obvious at a glance — and those are the items to rewrite.
Template 3: writing the rubric
For an extended response, the whole job is turning levels into observable behaviour.
You are an assessment design assistant for a Hong Kong secondary Citizenship and Social Development teacher.
Task: design a marking rubric for one extended-response question.
Context: Secondary 5. The question asks students to take and justify a position on whether Hong Kong should further restrict private cars in urban areas, in about 350 words, worth 12 marks. Students have studied argument structure and data interpretation, but commonly list points without explaining cause and effect.
Output format: a table with three dimensions (position and arguments, evidence and use of data, counter-argument and conclusion) and four levels each. Each cell holds one observable performance descriptor, with the maximum marks for that dimension noted beneath it. After the table, three additions — (a) the common borderline judgements and how to resolve them; (b) two sample answer extracts, one at the top level and one mid-range, each with the marking rationale; (c) a one-paragraph student-facing version of the rubric.
Constraints: descriptors must be observable and must not use unjudgeable wording such as "outstanding" or "deep understanding"; the three dimensions must total exactly 12 marks; sample extracts must be synthetic text you write yourself and must never quote real student work; where a policy, regulation or statistic is involved, mark it "teacher to verify" rather than stating an unverified figure.
Template 4: reviewing the paper you just wrote
Switching roles for a review pass is the best return on two minutes in this lesson.
You are an experienced panel head for Hong Kong secondary [subject], vetting a colleague's draft paper.
Task: review the paper below item by item and identify problems.
Context: [year level], unit "[name]", [total] marks in [minutes] minutes. Students have covered [list] and have not covered [list]. The table of specifications is attached: [paste].
Output format: a four-column table — item number, problem type (ambiguous / out of syllabus / wrong answer / marks disproportionate / wording above year level / no problem), the specific issue, and a suggested fix. After the table, three sentences on whether the paper matches the specification table and whether the time limit is realistic.
Constraints: review item by item, not as a general impression; do not rewrite the paper, identify problems and suggest fixes only; if you cannot determine whether an answer is correct, mark it "teacher to verify" rather than guessing; take the not-yet-covered list seriously and flag every item that relies on an untaught method.
Paper:
[paste]
Common mistakes
- Writing items first and the specification table afterwards. The table then just records what happened and loses its power to change the distribution. The order must be table first.
- Asking for "moderate difficulty". Difficulty is not an executable instruction. Cognitive level and mark allocation are.
- Filler distractors. If two of four options are obviously impossible, the item collapses into a coin toss. Requiring a rationale for each distractor makes this visible immediately.
- Rubrics graded by adjective. "Excellent / good / fair / weak" will not get two teachers to similar marks. Write "identifies at least two pieces of evidence and explains the causal link" instead.
Going further
- A different subject: for Chinese Language reading papers, require each item to state the reading skill level it assesses (retrieval / comprehension / analysis / evaluation). For English Language writing, ask for the task rubric in student-facing language alongside the marking rubric. For Information Technology, require each item to be labelled conceptual or practical, with practical items listing the specific output the marker should look for.
- A different year level: primary rubrics need student-readable language and no more than three levels. Secondary can carry four levels plus sample extracts.
- SEN adaptation: while setting the paper, also ask it to "identify which items can be removed for students with extended time without losing coverage of any learning focus". That is far more accurate than deciding on the morning of the test.
- Importing into an assessment system: if items have to go into an item bank or spreadsheet, do not rely on layout — specify a machine-readable format. See lesson 15.
With the paper set and the rubric written, the next step is the most time-consuming and most sensitive of all: marking and feedback. See lesson 9.
Key takeaways
- Build the table of specifications before the items, or the marks pile up on recall questions — and the table is also the fastest tool a panel head has for vetting a paper.
- Every distractor needs a reason: each wrong option should correspond to a real student misconception rather than making the count up to four.
- Rubric descriptors must be observable performance, never unjudgeable wording like "outstanding performance".
- Reviewing the finished paper in a different role is the cheapest quality control available, and it is particularly good at catching ambiguous and out-of-syllabus items.
FAQ
No. They are a draft. A teacher must check every item for answer, syllabus coverage and wording, especially anything involving figures, regulations or curriculum documents. For formal assessment, have a panel head or a second teacher vet it as well.
The standard is that two teachers marking the same script independently land within one level of each other. Usually four levels with one observable descriptor per dimension is enough; finer than that becomes unusable in practice.
State what students have and have not been taught in the context line, and add "use only methods students have already learned" to the constraints. Together those two sentences cut out most of it, though you still need to scan the paper.