Retrieval-augmented generation — RAG — is the workhorse behind most useful enterprise AI: an assistant that answers from your own documents instead of its training data. In a demo it looks trivial. In production it is where most projects quietly stall. The gap is almost never the language model. It is the retrieval, the chunking, the evaluation and the honesty about what the system does not know.
The demo trap
A first RAG prototype is deceptively easy: embed a few documents, drop them in a vector store, retrieve the top matches and let the model answer. It works beautifully on the ten questions you tried. Then it meets real users, real documents and real edge cases, and accuracy falls off a cliff. The reason is that a demo optimises for the happy path, while production is almost entirely edge cases.
The teams that succeed treat retrieval as a first-class engineering problem, not a wrapper around an API. They measure it, they break it deliberately, and they design for the questions that have no good answer in the corpus.
Chunking is a design decision, not a default
How you split documents into retrievable pieces quietly determines the ceiling of the whole system. Chunks that are too large dilute the relevant sentence in noise; chunks that are too small lose the context that made them meaningful. Fixed-size splitting is convenient and usually wrong. Better results come from respecting the document's own structure — sections, headings, clauses — so each chunk is a coherent unit of meaning.
There is no universal chunk size. A legal contract, a product manual and a chat transcript each want different treatment. This is worth real experimentation, because everything downstream inherits the quality of this step.
Retrieval quality is measurable — so measure it
The single biggest difference between a demo and a dependable system is that the latter has an evaluation set. Assemble a fixed list of real questions with known-good answers and known-relevant source passages. Then you can ask concrete questions: does the right passage appear in the top results? How often does the system retrieve nothing useful? Without this, every change is a guess and every improvement is a matter of opinion.
We treat this evaluation set as a regression suite. Every change to chunking, embeddings or prompts is scored against it before it ships, so quality never silently degrades — the classic failure mode where a "small improvement" quietly breaks a class of questions nobody re-tested.
Grounding and citations are non-negotiable
A production assistant must answer only from retrieved content and cite where each claim came from. This does two things: it lets users verify answers, and it makes hallucinations obvious rather than hidden. If the model cannot ground an answer in the retrieved passages, the correct behaviour is to say so — not to fill the gap with a confident guess.
Designing for "I don't know" is what earns trust. Users forgive a system that admits uncertainty far more readily than one that is confidently wrong, and in regulated or high-stakes settings, a wrong confident answer is a genuine liability.
The unglamorous checklist
The systems that hold up share a set of unglamorous properties: structure-aware chunking, a real evaluation set used as a regression gate, strict grounding with citations, explicit handling of out-of-scope questions, and monitoring once live so that a drop in retrieval quality is caught early. None of this is exotic. All of it is the difference between a compelling demo and something a team can actually depend on every day.
Metadata and filtering matter more than you expect
Pure semantic similarity is rarely enough on its own. Real corpora carry structure that the question implicitly relies on: a date, a document type, a department, a version. A user asking about "the current refund policy" does not want a semantically similar passage from a policy that was retired two years ago. Attaching metadata to each chunk and filtering on it before or alongside the semantic search removes a whole class of confidently-wrong answers that no amount of better embeddings would fix.
This is one of the most common upgrades we make to a struggling system. The retrieval was not bad in a vector sense; it was retrieving the right topic from the wrong context. A little structure goes a long way.
Latency and cost are product features
A technically excellent assistant that takes eight seconds to answer will lose its users to the old way of doing things. Retrieval adds steps, and each step adds time and cost. Caching frequent queries, retrieving fewer but better chunks, and choosing model sizes deliberately for each part of the pipeline all matter. These are not afterthoughts to bolt on at the end; they shape whether people actually adopt the system, which is the only measure of success that counts.
Where to start
If you already have a RAG prototype that impresses in demos but frustrates real users, the fix is rarely a better model. Start by building the evaluation set — even fifty good question-answer pairs will expose where retrieval is failing. Layer in metadata filtering, watch your latency, and treat grounding as mandatory. From there, the improvements are concrete, measurable and, usually, faster than expected.