Chunking, reranking and lost in the middle

At the start of term an administrative panel head pasted an entire parent-information pack, twenty-two PDF pages, into an AI and asked: "Which activities require a parent's electronic signature?" The AI listed five. She checked manually and found three missing, all on pages 9 to 14, the dead centre of the file. She said: "I thought long context meant it remembered everything."

The previous lesson covered retrieval before answering. This lesson covers why answers still go wrong afterwards: the middle of a long input is a high-risk zone, and how chunking, reranking and prompt order address that.

---

Middle-section amnesia when marking

Imagine asking a colleague to mark an extremely long assignment in ten minutes. They will probably remember the first page's points and the conclusion on the last page clearly, while the middle ten pages blur. They did read it. Attention was just uneven.

Language models behave similarly. Researchers placed the same key fact at different positions in a long input and found accuracy highest at the beginning or end and markedly lower in the middle, even for models advertised with million-token context. The effect is called lost in the middle; the original paper is in sources.

The practical conclusion for teachers is blunt: in any long document you paste at once, the middle pages are the least reliable. The mechanism relates to self-attention, covered in embeddings, positional encoding and self-attention.

The most effective defence against lost in the middle is usually not asking the model to concentrate harder. It is not giving it that many pages at once.

Chunk size: mark only the assigned pages

Chunking decides how much text sits in each cell of the knowledge base. Think of marking: do not ask a colleague to swallow twenty-two pages in one go. Ask them to mark three assigned pages and quote original sentences.

• Too small (one sentence per chunk): context disappears. "Must not exceed thirty minutes" — must not what?
• Too large (one chapter per chunk): five topics in one cell, retrieval drags irrelevant lines in and pushes the key clause into the middle.
• About right (one complete unit of meaning): roughly 300 to 500 words, one thing per chunk.

School documents already have boundaries: policies by clause number, schemes by unit, textbooks by subsection, circulars by heading. Cutting along structure almost always beats cutting every 400 words. A clean Word file with headings often beats a long scanned PDF in a knowledge base.

Pastoral circular versus fixed word count

The same "notify when absent three days running" rule, cut every 400 words, might leave "the class teacher must on the same day" at the end of chunk seven and "notify the student guidance master" at the start of chunk eight. Retrieved separately, neither chunk is complete. Cutting at the circular subheading "Leave procedures | three days or more" keeps the logic in one cell.

Overlap a sentence or two to prevent cliffs

Let neighbouring chunks share about a tenth of their content: the next chunk repeats the last sentence or two of the previous one. Storage cost rises slightly. The return is that key lines are not sliced exactly on a boundary. Policy clauses are often long; this matters especially in admin documents.

Reranking: from fifteen passages down to four

The first retrieval pass often returns a dozen possibly relevant passages. The one ranked first is not necessarily the best match, especially when many documents share words like "assessment", "submission" and "deadline".

Reranking uses a second, more discerning model to compare each passage against the question, reorder them, and pass only the best three or four to the main model. It is like a panel head opening fifteen old guideline folders and saying: "These four answer your question; the rest belong to other subjects."

Cutting fifteen passages down to four shortens the input and the high-risk middle all but disappears. That is more practical than chasing a larger context window.

Put critical instructions on the final line

Even when pasting by hand, the same layout rule applies, and it aligns with caching in the next lesson:

1. long material first (extracts, pupil work, full teaching text);
2. then the task;
3. finally the most important off-limits rules (do not invent, say not mentioned if absent, word limits).

Many people write requirements at the top and paste twenty circular extracts below, pushing off-limits rules into the low-attention zone. Moving "if the answer is not in the extracts, reply not mentioned" to the final line often shows an immediate difference.

Where people go wrong

• The drift: seeing "supports 1M tokens" and pasting an entire PDF expecting a complete summary. Self-check: do missed details cluster in the middle? Try six section-by-section prompts and compare omissions.

• The drift: asking for summary only, not original quotations. Self-check: summaries cannot be checked. Require "quote the original sentence and label the passage number". If it cannot quote, it probably never read it.

• The drift: uploading scanned PDFs without OCR or headings straight into the knowledge base. Self-check: do retrieval logs show "plausible but wrong topic" passages often? If so, tidy formatting before changing models.

Try it once

Time: about 12 minutes. Materials: any AI tool; prepare about fifteen short passages of your own (two or three sentences each, mostly unrelated admin text). In one passage include: "Pupils in external competitions must return a consent form with a parent's electronic signature no later than five working days before the event."

What to do: use the template below, placing the key passage at number 8 (middle) and number 14 (near the end) on separate runs:

Below are fifteen numbered administrative extracts from our school. Which extract mentions a parent's electronic signature? Quote the number and the original sentence.
<<<
[passage 1] ...
...
[passage 15] ...
>>>
If the answer is not in the extracts, reply "not mentioned" and do not speculate.

What to hand in: a comparison of both runs, which position was easier, and the model name if both succeeded.

How you know it worked: you can explain lost in the middle to a colleague and describe how "chunked processing, original quotations and off-limits last" reinforce each other.

Next lesson: context caching and cost control. Previous lesson: the school knowledge base and retrieval-augmented generation.