What are the main reasons a RAG system gives wrong or hallucinated answers?

30-second answerSay your answer out loud first, then reveal.

Failures happen at every stage. The answer isn't in the corpus (missing content), parsing broke the document, chunking split the answer, retrieval missed it (vocabulary mismatch, poor embeddings, wrong filters), it was retrieved but ranked too low, the context was too noisy, the LLM ignored or misread the context, or the sources conflict or are outdated. Diagnose by tracing each stage, not by changing the prompt first.

Failure points

Adapted from "Seven Failure Points When Engineering a RAG System", Barnett et al. 2024:

Missing content: the answer isn't in the knowledge base. The system should abstain, but often makes something up.
Parsing errors: tables flattened into gibberish, multi-column PDFs interleaved, scanned pages with no text.
Bad chunking: the answer is split across chunks, or a chunk lacks context ("it" refers to something in the previous chunk).
Missed retrieval: query wording differs from the document; acronyms; the embedding model is weak on the domain; filters exclude the right doc.
Low ranking: relevant chunk at position 25, but only the top 5 go to the LLM.
Context noise / overload: relevant information drowned among irrelevant chunks.
Generation errors: the model ignores the context, mixes in its parametric knowledge, mis-reads numbers, or answers a different question.
Conflicting or outdated sources: old and new policy versions both retrieved.
Wrong format / incomplete answers: the question needed a list of all items, but retrieval returned some of them.

Debugging approach

For a failing question, check in order: is the answer in the corpus? → in the parsed text? → in a chunk? → in the retrieved top 50? → in the top k after reranking? → used correctly by the LLM? The first "no" is your bug.

PreviousHow do you evaluate a RAG system?

This is what real progress feels like.

What are the main reasons a RAG system gives wrong or hallucinated answers?

Failure points

Debugging approach

Related