How do you choose top-k, and what is the "lost in the middle" problem?

30-second answerSay your answer out loud first, then reveal.

Top-k is how many chunks you pass to the LLM. Too few and you miss needed information (low recall). Too many and you add noise, cost and latency, and the model may get distracted. Research ("Lost in the Middle", Liu et al. 2023) showed LLMs use information at the beginning and end of long contexts better than information in the middle. So retrieve broadly, rerank, pass a focused set, and put the most relevant chunks first.

Choosing k

Retrieval k vs context k: retrieve 50–100 candidates for reranking, but pass only 3–10 chunks to the LLM.
Set it from data: plot recall@k on your eval set. If recall@5 = 0.82 and recall@10 = 0.90, decide whether 8 extra points of recall are worth double the context.
Question type: factoid questions need few chunks; summaries and comparisons need many.
Dynamic k: use a relevance-score threshold (only include chunks above a reranker score), with min/max bounds.

Mitigating lost-in-the-middle

Rerank so the best chunks come first.
Reorder: place the strongest chunks at the start and end of the context.
Compress: remove irrelevant sentences from chunks (contextual compression).
Fewer, better chunks instead of stuffing.

Modern long-context models have improved on this, but distraction from irrelevant context is still a real effect. Measure it on your own data rather than assuming.

Common mistakes

"Just send 50 chunks; the model has a 1M-token window." That costs more, adds latency, and often lowers answer precision.

PreviousWhat is reranking? Explain bi-encoders vs cross-encoders.

This is what real progress feels like.

How do you choose top-k, and what is the "lost in the middle" problem?

Choosing k

Mitigating lost-in-the-middle

Common mistakes

Related