RAG in Production: Grounding, Evaluation, and the Unglamorous Parts

Why RAG demos so well and deploys so badly

Retrieval-augmented generation is straightforward to prototype. Embed some documents, search them, pass the results to a model, and the answers are impressive within an afternoon.

The demo works because the questions were chosen and the corpus was small and clean. Production supplies neither. Real users ask about things the documents cover partially, or contradict each other on, or do not cover at all, and the difference between a useful assistant and a liability is entirely in how those cases are handled.

Chunking decides what retrieval can find

Retrieval can only return what chunking preserved. Split a document at fixed character counts and you will cut a table from its header, a clause from its definition, a step from its procedure.

• Split on structure. Headings and sections, not character counts.
• Carry context down. Each chunk should retain the document title and section path so it makes sense in isolation.
• Keep tables whole. A fragment of a table is worse than no table.
• Overlap deliberately. Enough to preserve meaning across a boundary, not so much that the index fills with duplicates.

Retrieval is more than vector similarity

Pure vector search finds semantic neighbours and misses exact terms (product codes, policy numbers, surnames) where the literal string is what matters.

Production retrieval usually combines both: a vector search for meaning, a keyword search for precision, and a reranking pass over the merged candidates to order them by actual relevance rather than raw similarity. The reranker is often the single highest-value addition, because it fixes the case where the right document was retrieved but ranked fourth.

Evaluation: the part that decides whether to ship

A RAG system has two failure modes and they need separate measurement. Retrieval can fail to find the right source, or generation can misuse a source it did find.

• Retrieval quality. For a set of real questions with known correct sources, is the right document in the returned set?
• Answer faithfulness. Is every claim in the answer supported by the retrieved text, or has the model filled a gap?
• Refusal behaviour. When the corpus genuinely lacks the answer, does the system say so?

Building that set from real questions is the least glamorous part of the work and the one that most determines whether the system is trusted.

Citation and the honest 'I do not know'

Every answer should carry its sources, visible and clickable. This is not a compliance nicety. It is what lets a reader verify in seconds instead of trusting blindly, and it changes the failure mode from invisible to obvious.

Equally, a system that cannot answer should say so. A confident fabrication costs more than a refusal, because the refusal prompts a human to look while the fabrication does not.

Keeping it current

A knowledge assistant that quotes last year's policy is worse than no assistant, because it is authoritative and wrong. Production RAG needs a refresh path: change detection at source, re-embedding on update, removal when a document is retired, and freshness visible in the answer so a reader can judge for themselves.

Key takeaways

• Chunk on structure and carry context into each chunk.
• Combine vector and keyword retrieval, then rerank.
• Filter by permission inside retrieval, never after it.
• Measure retrieval and faithfulness separately.
• Cite sources, refuse honestly, and keep the index fresh.

Frequently asked questions

What is retrieval-augmented generation?

RAG connects a language model to a searchable body of your own documents. Instead of answering from general training, the system retrieves relevant passages and generates an answer grounded in them, which makes the output traceable to a source.

Why do RAG systems give wrong answers?

Usually because retrieval failed to surface the right passage, chunking separated content from the context that gave it meaning, or the model filled a gap the sources did not cover. These are separate failures and need to be measured separately.

Is a vector database enough for RAG?

Rarely on its own. Vector search finds semantic similarity but misses exact identifiers like product codes or policy numbers, so production systems normally combine it with keyword search and a reranking step.