Multimodal prompts for images, handwriting and audio

The problem: a photo of a maths page, and the AI reads a 7 as a 1

A Primary 6 maths teacher, hoping to save time checking line by line, photographed a page of a pupil's calculations and asked:

Mark this page of maths homework for me.

The AI cheerfully listed five wrong answers with detailed explanations. Back at the exercise book, two of them were actually correct — the AI had read the pupil's 7 as a 1, and a hastily written multiplication sign as a plus. Worse, it never said "I may have misread this". All five errors arrived in exactly the same confident register.

Copy those five into a comment and hand the book back, and the pupil redoes two problems that were never wrong.

Same photograph, split into two steps:

Role: you are a transcription assistant for maths homework. You are not marking at this stage.
Task: transcribe every calculation the student has written in the image, exactly as written.
Output format: number each item and give the calculation as you read it, plus a section headed "low recognition confidence" listing every digit or symbol you are unsure of and the item it belongs to.
Constraints: transcribe only — do not calculate, do not judge correctness, do not supply steps the student did not write. Where the handwriting is illegible write "illegible" rather than guessing what was intended.

Forty seconds of checking, two corrections, and the confirmed text goes to the marking prompt. This time the list of errors is right.

Why this works

Multimodal models handle images and audio much as they handle text: convert the input into a representation the model can work with, then predict the most plausible output. The catch is that most plausible is not the same as most faithful to the original. Shown an ambiguous handwritten symbol, the model does not stop and say it cannot see clearly; it outputs whichever symbol is most likely in that position. This is what hallucination looks like in images and audio — see where hallucination and bias come from and multimodality, reasoning models and embodied AI.

So the core design of a multimodal prompt is one sentence long: separate "what did you see" from "what do you conclude". Done in a single step, a recognition error is buried under the reasoning that follows — the AI misreads a digit, then writes a perfectly coherent analysis built on it, and the teacher cannot see where it went wrong. Split in two, what the teacher checks is a short transcription list rather than a finished analysis.

The second reality to accept is uneven accuracy. Printed English is best, printed Chinese next, handwritten English after that, and handwritten Chinese is the weakest — more strokes, more similar-looking characters, more variation in everyday student handwriting. On audio, clear standard pronunciation transcribes well, while accented English, primary pupils reading Putonghua and classroom background noise all raise the error rate markedly. This is not one vendor's weakness but the current state of the task, which is why the prompt must demand that uncertainty is flagged.

Template 1: two-step handling of handwritten work

Step one is the transcription prompt above. Only after it is confirmed does this step run.

Role: you are a marking assistant for Primary 6 Mathematics.
Task: mark the calculations below, which a teacher has already checked against the original, and identify which step went wrong.
Output format: a four-column table — item number, the step where the error occurs, error type (conceptual / calculation / copying / cannot determine), one hint for the student.
Constraints: judge only from the text provided and do not reinterpret the image. Do not rewrite the student's full solution, and the hint must not contain the final answer. Where you cannot identify the faulty step, set error type to "cannot determine" and write "teacher to check the original book" in the hint column.
Checked calculations:
[paste the text confirmed in step one]

Template 2: reading a graph or diagram

Most used in Science and General Studies. The point is to separate reading values from interpreting them.

Role: you are a graph interpretation assistant for [Science / General Studies], working with [year level] material.
Task: interpret the [graph / diagram] in the image below so a teacher can judge whether it suits classroom questioning.
Output:
1. "Read from the image" — every value, label and axis detail you can read directly, each with its position (for example, the third data point on the horizontal axis).
2. "Inferred" — anything you concluded rather than read, listed with the basis for each inference.
3. "Not legible" — any part of the image that is blurred, obscured or too low in resolution.
4. Three guiding questions suitable for classroom use.
Constraints: never report a familiar trend or a textbook conclusion as if it were data on this image; anything not read directly goes in section 2. Where a value is uncertain, give a range and write "teacher to verify" rather than a precise figure. Do not answer the graph question for students.

Template 3: feedback on reading aloud

For English or Putonghua reading practice. This is practice feedback, not an assessment score.

Role: you are a feedback assistant for [English / Putonghua] reading practice with [year level] students.
Task: compare the student's recording with the passage below and give specific, practisable feedback.
Output:
1. "Differences heard" — every point where the reading departs from the passage, giving the word in the text and what you heard.
2. "Low recognition confidence" — passages you cannot be sure of because of pronunciation, pace or background noise, with their positions.
3. "Three things to practise" — one sentence each, saying what to work on and how.
Constraints: give no score, grade or percentage. Do not comment on the student's accent itself, only on departures from the text. Anything uncertain goes in section 2 and may never be counted as an error. Keep the tone encouraging and specific, never evaluative language such as "poor pronunciation". Do not re-record or model the whole passage for the student; comment word by word instead.
Passage: [paste the reading material]

Common mistakes

1. Doing it in one step and asking straight for marking. A recognition error arrives dressed in perfectly reasonable analysis. Fix: split transcription from judgement, with a teacher check in between.
2. Not requiring uncertainty to be flagged. The model will not volunteer "I cannot see this clearly". Fix: put a "low recognition confidence" or "not legible" section in the output specification and forbid guessing in the constraints.
3. Treating oral practice feedback as an assessment score. The same recording can score differently on a rerun, and parents will ask about that number. Fix: prohibit scores and grades explicitly, and output only practisable items.
4. Photographing the name and student number. The easiest privacy slip to make. Fix: cover or crop identity fields before shooting, or work inside a school platform with encryption at rest and an audit trail.

Going further

• Visual Arts portfolio feedback: adapt template 2 to describe composition, colour and media, keeping "what is visible" separate from "my interpretation". Add constraints forbidding judgement of the student's creative ability and any suggestion to redo the work.
• Handwritten Chinese Language compositions: run the transcription step first with an added constraint that incorrect characters must be transcribed exactly as written and never silently corrected, or the error statistics become meaningless.
• Science practical record photographs: require readings and units in separate columns, plus a column recording whether a figure came from the instrument display or from the student's handwritten record.
• Differentiating English oral work: for weaker students cut template 3's practice items from three to one and ask for a single model sentence rather than a passage; for stronger students add observations on intonation and linking.
• SEN adaptation: automated reading feedback should not be used as a progress indicator for students with speech and language needs. If it is used at all, add a constraint limiting the model to listing differences with no evaluative judgement, and leave interpretation to the teacher or speech therapist.

Where to go next

Multimodality extends AI to the things that actually exist in a classroom — a page of homework, a graph, a passage read aloud. The cost is one extra layer of uncertainty: the model can mis-see or mis-hear, and it will not tell you. The two-step workflow and a compulsory uncertainty section hold that risk inside manageable bounds, and teacher review closes the rest.

For how oral practice runs in a real timetable see AI oral practice, and for the marking workflow see AI marking and instant feedback. To turn these prompts into a shared panel asset, see building a panel prompt template library.