Why most RAG pipelines fail in production

Introduction

The demo works. The stakeholders clap. Three weeks after launch, nobody is using the assistant, and support is quietly answering tickets by hand again. This is the most common trajectory for RAG projects, and the cause is almost never the model.

It's retrieval. A demo runs on ten curated questions against a clean corpus. Production runs on thousands of badly-phrased questions against documentation that contradicts itself, duplicates itself, and went stale two quarters ago. If the right passage doesn't come back in the top results, the best model in the world will confidently summarize the wrong one.

The Three Failure Modes

Nearly every failed RAG deployment we've audited traces back to one of three causes.

1. Stale Corpus

The docs changed, the index didn't. Answers were right in March. Ingestion has to be a pipeline with change detection — not a one-time script someone ran before launch.

2. Chunking That Ignores Structure

Split a table from its header or a step from its warning, and retrieval returns fragments that are individually true and collectively wrong. Chunking has to respect the document's structure, not a fixed token count.

3. No Evaluation Harness

If you can't score accuracy and groundedness on every deploy, you find regressions the way your customers do — in production.

"A RAG system is a search engine with a language model on top. Fix the search engine first."

What Working Looks Like

Every RAG system we ship has the same skeleton. None of it is glamorous. All of it is the difference between an assistant people trust and one they route around.

✓Automated ingestion with change detection, so the index never goes stale

✓Structure-aware chunking that keeps tables, steps, and warnings intact

✓Hybrid retrieval with reranking, because embeddings alone miss exact matches

✓Citations on every answer, so users can verify instead of guessing

✓A fixed evaluation set that gates deploys — no regression ships silently

"Demos are graded on the best answer. Production is graded on the worst one."