Common pitfalls and architectural solutions for robust corporate RAG.
Most teams that implement RAG do it because they understand, at least conceptually, why it’s the right architecture for grounding AI in proprietary data. They’ve read the explanations. They’ve seen the demos. They build the pipeline, test it on a representative set of queries, get encouraging results, and ship.
Then production happens.
The queries are messier than the test set. The knowledge base has inconsistencies nobody noticed during development. The chunking strategy that worked for the main document type doesn’t work for the edge cases. The retrieval returns plausible-looking results that are subtly wrong for a non-obvious reason. And the model, working with those subtly wrong inputs, produces confident answers that are harder to catch than obvious errors.
This post is about the specific failure modes that show up in production RAG systems — not theoretical ones, but the patterns that appear repeatedly when you look honestly at where these systems break. And more importantly, what to do about each of them before they reach users.
Failure Mode 1: Retrieval That Looks Right But Isn’t
This is the hardest failure to detect and the most consequential. The retrieval step returns documents. They’re about the right topic. They look relevant. The model uses them and produces an answer. The answer is wrong.
What happened? The retrieved documents were topically adjacent to the query but didn’t actually contain the relevant information. The embedding similarity captured the domain — the documents were about the right subject area — but missed the specific content the query needed. The model then tried to answer using context that was close but not correct, and filled the gap with plausible-sounding inference.
This is the hallucination failure mode in RAG systems: not the model inventing from nothing, but the model extrapolating from insufficient retrieved context. It’s harder to catch than pure confabulation because the retrieved documents at least exist and are related to the topic.
What to do about it: Evaluate retrieval separately from generation. Before assessing whether the model’s answer is correct, assess whether the retrieved documents actually contain the information needed to answer the query correctly. This requires a test set of queries with known-good answers and known-good source documents. If retrieval is failing — returning topically related but informationally insufficient documents — no amount of generation-layer tuning will fix the underlying problem.
Failure Mode 2: Chunking That Destroys Context
Documents are split into chunks before embedding. The chunk size and the splitting strategy determine what the retrieval system actually has available to return. And bad chunking produces chunks that don’t mean anything coherently on their own.
A table of contents fragment that lists section headings without content. A paragraph that references “as described above” where “above” was cut off. The conclusion of an argument without the premises. A step-by-step process where steps 3 through 5 are in one chunk and the critical step 6 is in the next, with no guarantee they’ll be retrieved together.
The model receives these contextually incomplete fragments and has to work with them. Sometimes it handles this gracefully — recognizing that context is missing and hedging appropriately. Often it doesn’t. It tries to answer with what it has, and what it has is an incomplete picture.
What to do about it: Chunk with awareness of document structure, not just character count. Headers and section boundaries matter. Paragraphs that reference earlier content should carry that earlier content or a summary of it. Evaluate chunks by reading them in isolation — if a chunk doesn’t make sense without knowing what came before it, it will cause retrieval problems. Consider overlapping windows that carry context across chunk boundaries, and test whether this improves retrieval quality for your specific content types.
Failure Mode 3: The Knowledge Base That Lies
This one is organizational as much as technical. Enterprise knowledge bases contain documents that were accurate when written and have since been superseded, revised, or quietly made irrelevant by policy changes nobody updated the documentation for.
RAG retrieves based on semantic relevance, not on document currency. An outdated policy document that’s highly relevant to a query will be retrieved and used. The model doesn’t know it’s outdated. The user gets an answer that was correct two years ago and is wrong today.
The more insidious version: two documents in the knowledge base that contradict each other — one current, one outdated, both appearing authoritative. The model retrieves both, tries to reconcile them, and produces an answer that’s a confused blend of the two.
What to do about it: Metadata filtering is your first line of defense. If documents have creation or last-modified dates, use them. Set recency weights or hard cutoffs where appropriate. Build a content governance process — not a technical solution but an organizational one — that ensures the knowledge base is actively maintained. Treat stale content as a reliability bug, because in a RAG system it is one. And when you know the knowledge base may contain contradictions, build explicit conflict detection into the retrieval layer.
Failure Mode 4: The Query the System Was Never Designed For
Your test set shaped your development decisions. Chunk size, retrieval parameters, prompt structure — all optimized, consciously or not, for the queries you tested with. Production users will ask things you didn’t anticipate, and the system will handle them with whatever properties emerged from your development-time choices.
Short queries perform differently than long ones. Highly specific queries that should return a single authoritative document compete with topically similar documents that aren’t actually relevant. Vague or ambiguous queries produce retrieval results that reflect the ambiguity. Non-English queries in a predominantly English knowledge base produce unreliable embeddings. Questions that require synthesizing information across multiple documents stress retrieval systems designed for single-document answers.
What to do about it: Expand your test set deliberately before shipping. Include short queries, long queries, ambiguous queries, cross-document synthesis queries, and queries that have no good answer in the knowledge base. The last category is critical: what does your system do when it can’t retrieve anything relevant? If the answer is “generate something anyway,” that’s a failure mode. The correct behavior is to recognize low-confidence retrieval and respond accordingly.
Failure Mode 5: Confidence Without Calibration
Language models generate text that sounds confident. This is a feature in consumer applications and a liability in enterprise ones. A model that can’t distinguish between “I have strong retrieval context for this” and “I have weak retrieval context for this” will answer both types of query with the same confident tone — and users will treat both answers with the same trust.
In regulated environments, this matters enormously. A user who acts on a confidently stated wrong answer can cause real harm — financial, legal, operational. The system’s failure to signal uncertainty is itself a failure mode.
What to do about it: Build confidence signaling into the prompt design. Instruct the model explicitly to qualify answers when retrieved context is limited or when the query touches areas outside the knowledge base’s scope. Implement retrieval confidence scores — most vector databases return distance metrics that can proxy for retrieval confidence — and use them to modify how the model is prompted. Make “I don’t have reliable information on this” a first-class output, not an edge case. And evaluate your system’s calibration: when it expresses uncertainty, is it actually uncertain? When it expresses confidence, is it actually correct?
Failure Mode 6: The Invisible Regression
You update the knowledge base. You refine the chunking strategy. You upgrade to a newer embedding model. You switch to a different LLM version. Any of these changes can shift the system’s behavior in ways that aren’t obvious from spot-checking a few queries.
Without a systematic evaluation framework, these regressions are invisible until users encounter them. And because RAG systems fail by producing wrong-but-plausible answers rather than errors, the signal that something has degraded is often user complaints or qualitative feedback — slow, noisy, and hard to act on.
What to do about it: Treat your evaluation set as a regression suite. Before any significant change to the system — retrieval parameters, chunking strategy, embedding model, LLM version — run the full evaluation suite and compare results to baseline. Automate this where possible. Set thresholds that trigger review before a change goes live. This is basic software engineering discipline applied to AI systems, and it’s the difference between knowing your system is working and hoping it is.
The Common Thread
Every one of these failure modes is detectable before it reaches production, if you have the evaluation infrastructure to detect it. That infrastructure is the prerequisite for everything else: retrieval quality improvements, chunking refinements, confidence calibration, regression detection. Without it, you’re working blind.
Building the evaluation framework before optimizing the pipeline is the right order of operations. It’s almost never the order teams actually follow, because the pipeline is visible and exciting and the evaluation framework is unglamorous. But the unglamorous work is what makes the difference between a system that impresses in development and one that can actually be trusted in production.
Reliable enterprise AI isn’t a property you add after the fact. It’s the result of engineering decisions made before the first line of integration code — and enforced consistently from there.
Next: the EU AI Act and what it actually requires from engineering teams building AI systems in Europe — not as a compliance checklist, but as a useful framework for the reliability work you should be doing anyway.
Some information may be outdated
