Retrieval-Augmented Generation: The Failure Modes Nobody Demos

RAG demos are easy; RAG in production isn't

Retrieval-augmented generation looks solved in a demo: ask a question, get an answer grounded in your docs. Then it meets real queries and real content, and the RAG failure modes show up — most of them in the retrieval and the content, not the model.

1. Weak retrieval, confident answer

The retriever returns the closest passages even when none of them actually answer the question. The model then writes a fluent, confident answer from thin context. The fix is a relevance floor — if the best match is weak, the system says "I don't have a good answer" and escalates, rather than stretching a poor passage into a definitive reply.

2. Stale content

The index was built once and the source moved on. A support agent cites last quarter's returns policy; a search answers with a discontinued spec. Whatever RAG grounds on needs a refresh path — on our semantic-search engagement the vector index is kept current against a live supplier feed through an async re-embed pipeline, precisely so the answers don't drift stale.

3. Chunking that splits the answer

The content is chunked by fixed character count, and the sentence that answers the question is half in one chunk and half in the next, so neither retrieves well. Chunk by topic — one coherent rule per chunk — and keep each chunk self-contained enough to make sense alone.

4. The consistency trap

RAG grounds the general facts (policy, specs), but the query often needs those cross-referenced against specific live data — this order's return window, this account's status. If the two live in separate systems, there's a window where they disagree. Keeping them in one store lets the check happen in a single transaction.

5. Losing the citation

The answer is right but the system can't say where it came from. For anything a person has to trust or defend, retrieval has to carry the source through to the output, so "according to which document" is always answerable.

6. Garbage sources

Retrieval can be perfect and the answer still bad if the source says "returns may be accepted at our discretion." RAG surfaces your content gaps — it doesn't fix them.

7. No refresh path for the index

Related to stale content but worse: the index is built once at launch and there's no pipeline to update it. New documents never enter, edited ones keep their old vector. Within weeks the retrieval is quietly out of date. A production RAG system needs an event-driven or scheduled re-index, and it's not an optional enhancement.

How to catch these before users do

• An evaluation set of hard queries — the ambiguous ones, the near-misses, the things you don't have — run before and after every change.
• Check every claim against a source in a sample of real transcripts. Statements with no traceable source are the failures.
• Track the unsupported-statement rate over time. It should sit near zero by design; a rising number is a grounding regression.

The projects in our portfolio that use retrieval — semantic search, a support agent grounded in help-centre content — all run this loop, because the failure modes above don't show up in a demo and do show up in production.

Where this stops being right

• A tiny, stable FAQ may not need retrieval at all — inline the content in the prompt.
• A purely transactional feature (status lookups only) barely touches a knowledge base; don't over-build the retrieval layer.
• A very large or multilingual corpus changes the chunking and freshness design — these failure modes get harder, not easier.

FAQ

What's the most common RAG failure? Weak retrieval grounding a confident wrong answer. A relevance floor that triggers "I don't know" is the main defence.

Why do RAG answers go stale? The index isn't refreshed when the source content changes. RAG needs a re-index path, not a one-time build.

Does RAG fix bad documentation? No — it exposes it. A vague policy produces a vague grounded answer; the fix is upstream, in the content. Treat the first weeks of transcripts as a documentation audit: the questions the system hedges on are the questions your source material doesn't actually answer.

ISTRALLEN builds retrieval with a relevance floor, topic-aware chunking, and a live refresh path — see what we do.