A raw LLM is like a brilliant friend who read the entire internet two years ago and remembers most of it, *vaguely*, with total confidence. Ask about your company's refund policy and they'll invent one on the spot, fluently, with the calm authority of a man who has never been right about anything.
That's not lying. That's just what next-token prediction does when it has no facts in front of it. It's filling the gap with the most statistically plausible words, and plausible is not the same as true. RAG is how you put the facts in front of it.
The open-book trick
Retrieval-augmented generation is the open-book exam version of the same friend. Before they write a single word, you slide the relevant pages across the desk. Now they're not *recalling* — they're *reading*. That one change kills most hallucinations and lets every answer point at a source.
Don't make the model memorize your data. Hand it the right page at the right moment.
This matters because the alternative — fine-tuning a model on your data — is slow, expensive, and goes stale the moment your data changes. RAG updates the instant you add a document. New policy at 9am? The system knows it at 9:01. No retraining, no GPU bill, no waiting.
The whole thing in three moves
People wrap RAG in a lot of mystique. It's genuinely just three steps:
- 01Index — chop your documents into chunks, turn each into an embedding (a vector that captures meaning), and store them in a vector database.
- 02Retrieve — when a question comes in, embed it too, and pull the chunks closest in meaning.
- 03Generate — stuff those chunks into the prompt and ask the model to answer *using them*.
// 1. index (offline, once)
for (const doc of docs) {
for (const chunk of split(doc)) {
await db.upsert({ id: chunk.id, vector: await embed(chunk.text), text: chunk.text });
}
}
// 2 + 3. retrieve, then generate (per query)
const q = await embed(question);
const hits = await db.search(q, { topK: 6 });
const context = hits.map(h => h.text).join('\n---\n');
const answer = await llm.complete(
`Answer using ONLY the context. If it's not there, say you don't know.\n\n` +
`Context:\n${context}\n\nQuestion: ${question}`
);That's a working RAG system. It'll demo beautifully. And then you'll put it in front of real users and it'll fall apart — not because of the model, but because of step 2.
Where every RAG system goes to die
Everyone's first RAG demo works. Everyone's first RAG *product* disappoints. The gap is almost always retrieval. If you fetch the wrong chunks, even a genius model gives you a confident, beautifully-worded wrong answer. Garbage in, eloquent garbage out.
1. Chunk like you mean it
Naive fixed-size chunking will happily slice a sentence — or a critical caveat — clean in half. Imagine cutting "You are eligible for a refund unless the item was on final sale" right after "refund." Now your bot is promising money it shouldn't. Chunk on natural boundaries (headings, paragraphs) and add a little overlap so context doesn't fall off a cliff between chunks.
// don't: blind 500-char slices that guillotine sentences
// do: split on structure, keep a sliding overlap
function chunk(text: string, size = 800, overlap = 120) {
const paras = text.split(/\n{2,}/);
const out: string[] = [];
let buf = '';
for (const p of paras) {
if ((buf + p).length > size) { out.push(buf); buf = buf.slice(-overlap); }
buf += p + '\n\n';
}
if (buf.trim()) out.push(buf);
return out;
}Tip
Attach metadata to every chunk — source, title, section, date. You'll want it for citations, for filtering ("only 2026 docs"), and for debugging why a wrong chunk got retrieved.
2. Hybrid search beats pure vectors
Vector search is great at *meaning* and weirdly bad at *exact things* — error codes, product SKUs, names. Ask for "E-4021" and pure semantic search returns five philosophically-adjacent paragraphs about errors in general. Keyword search is the opposite: literal but blind to meaning. Run both and merge the results and you get the best of each.
const [dense, sparse] = await Promise.all([
vectorSearch(await embed(q), { topK: 20 }), // meaning
keywordSearch(q, { topK: 20 }), // exact terms
]);
// reciprocal rank fusion: reward docs that rank high in EITHER list
const scores = new Map<string, number>();
for (const list of [dense, sparse])
list.forEach((d, i) => scores.set(d.id, (scores.get(d.id) ?? 0) + 1 / (i + 60)));
const merged = [...scores.entries()].sort((a, b) => b[1] - a[1]).slice(0, 10);3. Rerank before you trust the top-k
Your first retrieval pass is fast and approximate. Take the top ~20 candidates and run them through a reranker — a model whose entire job is to score *real* relevance for a specific query — then keep the best 5. This single step is the cheapest, biggest quality win in most RAG stacks. It's the difference between "close enough" and "exactly right."
Note
Mental model: retrieval is recall (cast a wide net), reranking is precision (sort the catch). You need both, in that order.
Prompting the generation step
Retrieval gets the right pages; the prompt decides whether the model actually behaves. Two instructions earn their keep: force grounding ("use only the context") and force humility ("if it's not there, say so"). Otherwise the model will helpfully blend the context with its own half-memories and you're back to confident nonsense.
const system = `You answer strictly from the provided context.
Rules:
- If the answer isn't in the context, say "I don't have that information."
- Cite the source title for each claim, like [Refund Policy].
- Do not use outside knowledge.`;The unglamorous secret nobody posts about
Build an evaluation set. A few dozen real questions with known-good answers. Score every change against it. "Feels better" is a vibe, not a metric. "Recall went from 71% to 89% after we added reranking" is a number you can defend in a meeting and to yourself at 2am.
const cases = [
{ q: 'how do I reset my password?', mustCite: 'auth/passwords.md' },
{ q: 'what is error E-4021?', mustCite: 'errors/payments.md' },
];
let hits = 0;
for (const c of cases) {
const got = await retrieve(c.q);
if (got.some(d => d.source === c.mustCite)) hits++;
}
console.log(`recall@k: ${(hits / cases.length * 100).toFixed(0)}%`);Common ways it still goes wrong
- Stale index — you updated the docs but forgot to re-index. The bot confidently quotes last quarter.
- Context overflow — you stuffed 20 chunks in and the model lost the plot. Fewer, better chunks win.
- No citations — users can't trust what they can't verify. Always show sources.
- One giant chunk per doc — you embedded a 50-page PDF as one vector, so it matches everything and means nothing.
That little eval harness plus disciplined retrieval is the entire difference between a RAG demo that gets applause and a RAG product that gets renewed. The model is the easy part. The plumbing is the job. Build the plumbing well and the magic takes care of itself.
Key takeaways
- 01RAG grounds an LLM in your data so it answers from sources, not vibes.
- 02Retrieval quality — chunking, hybrid search, reranking — matters more than model size.
- 03Build an evaluation set so 'feels better' becomes a number you can defend.
FAQ
Related reading