Retrieval-augmented generation now powers an estimated 60% of production AI applications. It is also the source of a great deal of misdirected debugging effort.
When a RAG system gives a bad answer, the instinct is to blame the model. Teams switch to a larger model, rewrite the prompt, adjust the temperature. Industry analysis in 2026 consistently finds that when RAG fails, the failure point is retrieval about 73% of the time, not generation.
Put plainly: the model usually answered correctly given what it was handed. It was handed the wrong context.
Why this is easy to miss
Generation failures are visible. You can read the answer and see that it is wrong. Retrieval failures are invisible unless you deliberately look — the model receives three irrelevant chunks, does its best, and produces something fluent and incorrect. The output looks like a reasoning failure, so that is where the debugging goes.
If you are not logging the retrieved chunks alongside every answer, you are debugging blind.
The single highest-leverage change most teams can make is to log retrieval results with each response. It converts an opaque quality problem into a visible one within a day.
Start with hybrid retrieval
Pure vector search fails on a predictable class of queries: exact identifiers, product codes, rare proper nouns, and precise terminology. Embeddings capture semantic similarity, and semantic similarity is not what you want when someone searches for an error code.
Hybrid retrieval — dense vectors plus BM25 keyword matching, combined and reranked — resolves most of this. It is more moving parts than pure vector search and considerably more robust. Current practice treats it as the sensible default rather than an optimisation.
Chunking decides your ceiling
Chunking is the least glamorous decision in the pipeline and one of the most consequential. Too small and you fragment the context needed to answer. Too large and you dilute the embedding until retrieval degrades.
Current practice starts with recursive chunking at 300–500 tokens with 10–15% overlap, then adds a short contextual summary to each chunk so it carries some awareness of its surrounding document. The summary step is frequently skipped and frequently the thing that fixes retrieval quality.
Treat these as starting points to be measured against your own corpus, not as settled answers. Chunking that works for support tickets will not work for contracts.
Route by complexity
Not every query needs the full pipeline. The strongest production systems use adaptive routing — matching query complexity to pipeline complexity, so simple lookups stay cheap and fast while genuinely hard questions get multi-step retrieval or graph traversal.
This matters for cost and latency in roughly equal measure. Running every query through your most expensive path is how RAG systems become too slow and too costly to keep in production.
Measure faithfulness, not vibes
Faithfulness measures whether an answer is actually grounded in the retrieved context or whether the model filled gaps from its parameters. In regulated environments, faithfulness above 0.85 is increasingly treated as a baseline for deployment.
Retrieval quality and generation quality need separate measurement. Aggregate scores hide which half is failing, which is precisely the information you need.
A practical order of operations
Log retrieved chunks with every answer. Build an evaluation set of real questions with known-correct sources. Measure retrieval quality independently — did the right chunk make it into the top results? Fix chunking and add hybrid search plus reranking before touching the model. Add routing once you know your query distribution.
Most teams do this in reverse, starting with model selection. It is the most expensive ordering available.
Better retrieval beats a better model, at a fraction of the cost.