Question setting and rubric prompts
A good paper is derived from learning focuses and cognitive levels, not from whatever comes to mind. Four prompts — specification table, items from the table, rubric, and paper review.
以繁體中文閱讀A week before the test, a Secondary 5 Citizenship and Social Development teacher generates a unit paper on public health policy with AI: twenty items, neatly laid out. Checking against the learning focuses, she finds sixteen test definition recall, nothing on "using data to support an argument" — the centre of the unit — and item 7 uses a statistical chart method the class has not been taught. She writes in the margin: "Wrong distribution, not wrong items."
Her prompt had been "Secondary 5 CS, public health, twenty items, multiple choice and short answer" — never stating which focuses carry how many marks at which cognitive level. This is lesson 3 of module two, the assessment step after worksheets and the one that draws most on professional judgement. It also follows lesson 4: plan first, execute second.
Setting a paper starts with the grid
Professional paper setting decides two things at once: what is assessed (learning focus) and how deeply (cognitive level). Draw those as a grid — focuses down the side, cognitive levels across the top, marks in each cell — and you have a table of specifications.
Without the table, the model fills the paper with recall items because that is what most tests look like in its training data. It does not know where the emphasis of your unit lay. Give it the table and it fills the cells faithfully. You can adjust proportions before writing items, instead of discovering item 7 is out of syllabus after the fact.
The specification table is not an admin form. It is the scale drawing before you build — fix the cells first and items land where they should.
Template 1: the table first
You are an assessment design assistant for a Hong Kong secondary Citizenship and Social Development teacher. We work in two steps. Do step one only.
Task: design a table of specifications for a unit test.
Context: Secondary 5, unit "public health policy". Learning focus: (1) describe the main aims of Hong Kong public health policy; (2) interpret health statistics published by the government; (3) use data to support a personal argument on policy effectiveness; (4) identify limitations of data sources. Six lessons taught, emphasis on (2) and (3). Test 40 marks in 45 minutes.
Output format: a table with the four focuses down the side and three cognitive levels across the top (recall and understanding / application / analysis and evaluation), each cell giving suggested marks and item count; below the table, three bullet points justifying the allocation plus a total-marks check.
Constraints: analysis and evaluation must carry at least 30% of the marks; do not write any items yet; every focus needs at least one item; if a focus is unsuitable for written assessment, say so and explain rather than forcing marks into the row.
When the table arrives, spend a minute checking whether (2) and (3) carry enough marks. Adjust before step two.
Template 2: items from the table
You are a question-setting assistant for a Hong Kong secondary Citizenship and Social Development teacher.
Task: write items according to the confirmed table of specifications below.
Context: [paste revised table]. Students have covered: [list]. Not yet covered: [for example "regression or advanced statistical charts"]. House format: four-option multiple choice, short-answer items with marks shown, data items with a written figure description for the teacher to supply later.
Output format: work cell by cell. Each item gives number, question, focus number, cognitive level, marks and answer. Multiple-choice items add a "distractor rationale" column stating which student misconception each wrong option targets. End with a total-marks check.
Constraints: use only methods already taught; distractors must reflect errors students genuinely make, never filler; for items needing a figure, describe it in words and mark "figure to be supplied by teacher"; mark uncertain data or policy "teacher to verify".
The distractor rationale column is the most valuable part. Forcing the model to give each wrong option a pedagogical reason makes filler options obvious immediately.
Template 3: the rubric
For an extended response, turn levels into observable behaviour:
You are an assessment design assistant for a Hong Kong secondary Citizenship and Social Development teacher.
Task: design a marking rubric for one extended-response question.
Context: Secondary 5. The question asks students to take and justify a position on whether Hong Kong should raise the tax on sugary drinks, in about 350 words, worth 12 marks. Students have studied argument structure and data interpretation, but commonly list points without explaining cause and effect.
Output format: three dimensions (position and arguments, evidence and use of data, counter-argument and conclusion), four levels each, one observable descriptor per cell, maximum marks noted under each dimension; then (a) common borderline judgements and how to resolve them; (b) one synthetic extract at the top level and one mid-range, each with marking rationale; (c) a short student-facing version of the rubric.
Constraints: no unjudgeable wording such as "outstanding" or "deep understanding"; the three dimensions must total exactly 12 marks; sample extracts must be synthetic and must never quote real student work; mark uncertain policy or statistics "teacher to verify".
Template 4: panel-head review
After the paper is written, switch roles — two minutes for the best return in this lesson.
You are an experienced panel head for Hong Kong secondary [subject], vetting a colleague's draft paper.
Task: review the paper below item by item.
Context: [year level], unit "[name]", [marks] marks in [minutes] minutes. Students have covered [list] and have not covered [list]. Specification table: [paste].
Output format: a four-column table — item number, problem type (ambiguous / out of syllabus / wrong answer / marks disproportionate / wording above year level / no problem), the specific issue, suggested fix; after the table, three sentences on whether the paper matches the specification table and whether the time limit is realistic.
Constraints: review item by item, do not rewrite the whole paper; if you cannot determine an answer, mark "teacher to verify"; flag every item that relies on an untaught method.
Paper:
[paste]
Where people go wrong
- Items first, table afterwards. The table becomes a record of what happened and cannot change distribution. Self-check: when you edit the table, would you delete items because of it? If not, items came first and the order is wrong.
- "Moderate difficulty" as an instruction. The model cannot execute it. Replace with cognitive level and mark proportions. Self-check: does the prompt contain a specific percentage or mark allocation?
- Filler distractors to make four options. Two obviously impossible options turn the item into a coin toss. Require a distractor rationale column. Self-check: can you name the misconception behind each wrong option?
- Rubrics graded by adjective. "Excellent / good / fair / weak" spreads two markers apart. Write "cites at least two pieces of evidence and explains the causal link". Self-check: could you mark from the descriptor alone, without a sample answer?
Try it once
Time: 10 to 15 minutes.
What to do: Choose a unit you are assessing soon. Use template 1 to produce the specification table only (no items). Check whether analysis and evaluation carries at least 30%. If not, revise the table, then use template 2 to write only the items for one cell (for example "focus 3, application level, 8 marks"), including distractor rationale.
What to submit: The revised table (or the relevant row) and the complete item(s) for that one cell, with distractor rationale. In two sentences, explain how the item maps to that cell.
How you know it worked: Without the table in front of you, you can still state which focus and cognitive level the item assesses, and you can name the student misconception behind each wrong option — meaning you wrote to the cell, not to "what a test looks like".
With the paper set and the rubric written, the next step is marking and feedback: lesson 9.
What to take from this lesson
- Produce and confirm a table of specifications before writing items, so recall questions do not fill the paper.
- Require a distractor rationale column for multiple-choice items, so each wrong option maps to a real student misconception rather than filler.
- Write rubric levels as observable performance descriptors, not unjudgeable wording like "outstanding performance".
- Run a finished paper through a panel-head review role item by item to catch ambiguity and out-of-syllabus questions.
Questions that keep coming up
Only as a draft. Check every item for answer, syllabus coverage and wording; mark anything involving figures, regulations or curriculum documents "teacher to verify". For formal assessment, have a panel head or second teacher vet it.
The standard is two teachers marking the same script independently landing within one level of each other. Usually three dimensions, four levels each, one observable descriptor per cell is enough.
State what students have and have not covered in context, and add "use only methods already taught" to constraints. Together those two lines cut most of it, though you still need to scan the paper.
Not recommended. A table filled in after the fact only records what happened and cannot change the distribution. The order must be confirm the table, then write items.