RAG (retrieval-augmented generation) grounds a language model in your own data by fetching the most relevant, source-cited passages at query time and giving them to the model as context. Done right — clean chunking, hybrid keyword-plus-vector search, a reranking pass, inline citations and permission-aware retrieval — it removes most 'hallucination' without exposing data users shouldn't see.
RAG is a retrieval problem wearing a generation costume
Everyone obsesses over the model. But in a RAG system the model is the easy part — a competent writer that can only be as good as the paragraphs you hand it. If retrieval surfaces the wrong chunk, the smartest model on earth will fluently summarize the wrong thing. So here's the uncomfortable truth behind most "the AI is hallucinating" complaints: they are usually retrieval failures in disguise. Fix retrieval first, and the generation quietly takes care of itself.
Chunking is a product decision, not a default
The default tutorial splits every document into fixed 512-token windows and moves on. That's where quality goes to die. A fixed window happily cuts a table in half, severs a clause from its heading, and strands a number from the sentence that gives it meaning. Chunk along the document's real structure instead — by section, heading, list, or table — so each chunk is a self-contained idea. Add a little overlap so context isn't lost at the seams, and attach metadata to every chunk:
- Source and section — so you can cite it and a user can find it.
- Date or version — so stale content can be down-ranked or filtered.
- Access level — so retrieval can respect who's allowed to see it (more on that below).
Good chunking is unglamorous, and it is roughly 80% of your result.
Embeddings get you close; they don't get you right
Vector search is superb at semantic similarity and merely okay at precision. Ask about "error code 5012" and pure embeddings may hand back passages that are broadly about errors while missing the exact string a user needs. The fix is hybrid retrieval: run keyword (BM25) search and vector search together, then merge the results so you get both exact matches and conceptual ones. Then add a reranker — a cross-encoder that reads the query and each candidate together and reorders them by true relevance.
Retrieve widely, then narrow hard
A reliable pattern: cast a wide net (retrieve 20–50 candidates via hybrid search), rerank them, then pass only the top handful into the model's limited context window. Stuffing more raw chunks into the prompt is not better — it dilutes the signal, inflates cost and latency, and buries the answer the model needs under nine it doesn't. Retrieve broadly, filter ruthlessly, generate from the best.
Citations, or it didn't happen
Every claim an answer makes should point back to the source chunk it came from. This isn't a nicety — it's the mechanism that makes the whole system trustworthy and debuggable. Inline citations let a user verify an answer in one click, let you diagnose exactly which retrieved passage led the model astray, and quietly change the psychology of the tool from "mysterious oracle" to "fast research assistant." If your RAG app can't show its work, users are right not to trust it.
If the answer isn't in the retrieved context, the correct output is "I don't know" — not a fluent guess.
Grounding is how you stop the model inventing things
Hallucination isn't mystical; it's simply what a model does when you ask it to answer beyond its evidence. You reduce it structurally, not by pleading:
- Constrain the model to the context. Instruct it to answer only from the retrieved passages and to say plainly when they're insufficient.
- Give it an escape hatch. An explicit "I don't have enough information to answer that" path — and reward using it in your evals rather than penalizing it.
- Narrow the scope. A focused, well-grounded assistant beats a know-it-all that bluffs.
- Measure it. Track how often answers are actually supported by their citations, and treat unsupported claims as bugs, not quirks.
Retrieval must respect who is asking
This is the one that ends careers when it's skipped. In a real organization, not everyone may see everything — and permissions cannot be an afterthought bolted on at the end. The retrieval layer has to be permission-aware: filter candidates by the caller's access rights before they ever reach the model, so a document the user can't open can never leak into an answer. Enforce access at query time, per user — not just at indexing time — and keep sensitive fields out of the index unless they're properly scoped. A RAG system that ignores permissions isn't a knowledge base; it's a data-exfiltration tool with a friendly chat UI.
You can see grounded retrieval working in practice in our live demos, where a document assistant answers strictly from a provided corpus, cites its sources inline, and openly admits when the answer isn't there.
- Most "hallucination" is really bad retrieval — invest there before you touch the model.
- Chunk by document structure, not fixed token counts; carry metadata on every chunk.
- Combine keyword + vector search and add a reranker; retrieve widely, then narrow.
- Cite every claim so answers are verifiable and failures are debuggable.
- Ground the model in the retrieved context and give it a real "I don't know" path.
- Make retrieval permission-aware and enforced per user at query time — not just at indexing.
Frequently asked questions
What is retrieval-augmented generation (RAG)?
RAG grounds a language model in your own data by retrieving the most relevant passages at query time and giving them to the model as context to answer from. Instead of relying on what the model memorized in training, it reads your documents, cites its sources, and stays current as your content changes.
Why do LLMs hallucinate, and does RAG fix it?
A model hallucinates when it's asked to answer beyond its evidence, so it fluently fills the gap. RAG reduces this by constraining the model to retrieved passages and giving it an explicit 'I don't know' path. But most hallucination is really a retrieval failure — fix retrieval first, and generation largely takes care of itself.
What is the difference between keyword, vector and hybrid search?
Keyword (BM25) search matches exact terms, so it nails specific strings like an error code but misses synonyms. Vector search matches meaning, so it finds conceptually related passages but can miss the exact match. Hybrid search runs both and merges the results, then a reranker reorders the candidates by true relevance — giving you precision and recall.
How do you keep RAG from leaking data a user shouldn't see?
Make retrieval permission-aware. Filter candidate passages by the caller's access rights at query time, per user — before anything reaches the model — so a document someone can't open never lands in their answer. Enforce access at retrieval, not just at indexing, and keep sensitive fields out of the index unless they're properly scoped.