Chunking, reranking and lost in the middle

A vice-principal pasted the whole twenty-eight-page student handbook into an AI and asked it to list every form requiring a parent's signature.

It listed six. Checking afterwards, she found the handbook actually contains nine, and the three it missed all sat on pages 12 to 18, which is to say the middle of the document.

Her first reaction was that the AI had malfunctioned. In fact this is an effect with a name, a body of research behind it, and a set of remedies.

The effect first: attention is uneven

Imagine asking a colleague to read a long set of minutes and then tell you the action items. He will probably remember the first few and the last few clearly, while the long stretch in the middle becomes a blur.

Language models behave the same way, and researchers named the effect "lost in the middle". The method was to place the same piece of key information at different positions in a long input and see whether the model could find it. The finding: accuracy is highest when the information sits at the beginning or the end and drops markedly when it sits in the middle, even for models explicitly built for long contexts.

Note this is not simply running out of memory. The model genuinely read the whole document. The problem is how unevenly it distributes attention, which relates to the self-attention mechanism covered in embeddings, positional encoding and self-attention.

For a teacher the practical conclusion is blunt: in any long document you paste, the middle is the high-risk zone.

Remedy one: chunking, or cutting a textbook into revision cards

Chunking means cutting a document into passages. Imagine turning a textbook into revision cards, and ask how much goes on one card.

• Cut too small (one sentence per card): each card loses its context. "Must not exceed thirty minutes" — what must not?
• Cut too large (one chapter per card): a single card carries five different topics, so retrieval drags in a great deal of irrelevant text, and irrelevant text is exactly what pushes the key sentence into the middle.
• Cut about right (one complete unit of meaning per card): roughly 300 to 500 words, each covering one thing.

School documents have a large advantage here: they already have structure. Policies have numbered clauses, schemes of work have units, textbooks have subsections, circulars have headed paragraphs. Cutting along those natural boundaries almost always beats cutting at a fixed word count. It is also why a cleanly formatted policy with clear headings performs noticeably better in a knowledge base than a long scanned PDF.

Remedy two: overlap, so a key sentence is not cut in half

Cut strictly at every 400 words and sooner or later a crucial sentence straddles a boundary. "Where a pupil is absent three days running, the class teacher must" ends passage seven, and "notify the student guidance master the same day" begins passage eight. Retrieved separately, neither is complete.

The fix is to let neighbouring passages share about a tenth of their content, so passage eight repeats the last sentence or two of passage seven. The cost is slightly more storage. The return is that sentences never break. This matters especially in education documents, where policy clauses are often long.

Remedy three: reranking, the librarian who re-sorts your results

The first retrieval pass typically finds a dozen or so possibly relevant passages. The one ranked first is not necessarily the most useful.

Imagine searching a library for "Hong Kong water resources" and being handed ten books. You show all ten to the librarian and explain you are teaching Primary 5 General Studies. She flicks through and says: these three are what you want, that one is university hydraulic engineering, that one is about Taiwan.

Reranking is that librarian. Technically it uses a second, more discerning model to compare each retrieved passage against the question, reorder them, and pass only the best three or four to the main model.

Why does it matter so much? Because it directly shortens the input. Cut fifteen passages down to four and the high-risk middle all but disappears. The most effective defence against lost in the middle is not asking the model to concentrate harder, but giving it less to read.

Remedy four: put the most important instruction last

This one is free and available to every teacher today.

Since attention is highest at the end, a prompt should be structured like this:

1. background and material first (this is where the long part goes);
2. then the task;
3. and only then the most important constraints and format requirements.

Many people write their requirements at the top and then paste a wall of text, which pushes those requirements straight into the low-attention zone. Move "do not invent anything, write teacher to verify where unsure, keep each item under 25 words" to the final line of the prompt and the difference is often immediately visible.

An experiment you can verify yourself

This takes five minutes and is thoroughly convincing. Paste the following into any AI tool, filling in about twenty passages of your own padding:

Below are twenty numbered extracts from our school's administrative circulars. Tell me which extract mentions a parent's signature being required, quoting the extract number and the original sentence.
<<<
[passage 1] (content unrelated to signatures)
[passage 2] (content unrelated to signatures)
... (continue to passage 10)
[passage 10] Pupils taking part in external competitions must return a consent form signed by a parent no later than five working days before the event.
... (continue with unrelated content to passage 20)
[passage 20] (content unrelated to signatures)
>>>
Final requirement: if the answer is not in the extracts, reply "not mentioned" and do not speculate.

Run it twice, once with the key passage at number 10 (the middle) and once at number 19 (near the end). Same model, same twenty passages, and yet at the end it is found almost every time while in the exact middle it is missed noticeably more often. This is not superstition. It reproduces.

An example from the classroom

A secondary panel head had to consolidate her panel guidelines for the new school year. Her first attempt was to paste all three old documents, forty pages in total, and ask the AI to merge them into one.

The resulting draft omitted a change to the marking rubric that sat in the middle of the old documents, and it happened to be the item the whole panel had spent longest debating the previous year. She nearly issued it.

Her second attempt used no technical tools whatsoever:

First, she split the forty pages along their natural headings into fourteen sections and handled one section at a time. Second, she required the AI to quote original sentences rather than summarise. Third, she put the line "do not merge statements from different documents, and where two documents conflict, list both and label the source" as the final line of every prompt. Fourth, only at the end did she use one prompt to assemble the fourteen results into a table of contents.

The second draft omitted nothing, and because every point carried an original quotation she could answer "where does this clause come from" live at the panel meeting. The whole method was simply cutting one large job into fourteen small ones, which is chunking in its teaching form.

What this means for your classroom

1. Do not paste a whole document at once. Working section by section takes more clicks, but the cost of missing something matters far more, especially in policy and assessment documents.
2. Put the most important sentence last. A free change of habit with immediate effect, applicable in every AI tool.
3. Ask for quotations, not summaries. A summary cannot be checked, a quotation can. And when it cannot produce the quotation, you know it never read the passage.
4. Document formatting affects AI performance. Clear headings and clause numbers help retrieval as well as human readers, which is a fresh argument for persuading colleagues to tidy their files.
5. Long context is not a cure-all. The context length a vendor advertises is a ceiling on what can be read in, not a guarantee of what is read accurately, and long inputs are priced separately. See context caching and cost control.

What comes next

If re-pasting the same document is expensive, is there a way to make the second time cheaper? There is, and the principle is surprisingly simple. See context caching and cost control. To revisit the retrieval pipeline as a whole, return to the school knowledge base and retrieval-augmented generation. To understand why a model invents content it skipped, see where hallucination and bias come from.