I wrote the hungry-person's guide to RAG a while back — restaurant menu, kitchen, the whole bit. That post ends where every tutorial ends: the demo works, everyone claps. This post starts where production starts: a user asks a reasonable question and the system answers with total confidence and total nonsense.
The instinct is always the same: the model is dumb, swap the model. But in almost every RAG failure I've debugged, the model did a fine job answering from what it was handed. The problem was upstream — the pipeline handed it the wrong pages. You don't fix that with a bigger model. You fix it with a debugging order.
Suspect #1 — chunking cut the answer in half
Fixed-size chunking is a bread slicer, and sometimes the answer is a meatball. Slice every 500 tokens and you will eventually cut through the exact sentence that mattered. The retrieval then finds half the answer, the model improvises the other half, and the improvised half is where the hallucination lives.
- A table sliced mid-row — the model sees column headers but not the row the user asked about, so it invents one.
- A heading separated from its body — 'Refund policy' lands in chunk 12, the actual policy in chunk 13.
- A code example split down the middle — the setup retrieved, the punchline lost.
- Pronouns orphaned from their subject — a chunk full of 'it must be configured' with no surviving mention of what 'it' is.
The fix is structure-aware chunking: split on headings and semantic boundaries, keep tables and code blocks whole, add overlap so boundary sentences exist in both neighbors, and prepend each chunk with its document title and section path. A chunk should make sense read alone by a stranger — because that's exactly how the model reads it.
Suspect #2 — the right chunk never showed up
Here's the question that saves you weeks: *for the questions users actually ask, does the chunk containing the answer appear in the top-k at all?* That number is your retrieval hit rate, and until you know it, you're debugging blind. Prompt tweaks can't help a model answer from a document it never received.
// the cheapest eval you'll ever build, and the most useful
let hits = 0;
for (const q of goldenSet) {
const retrieved = await retrieve(q.question, { k: 5 });
if (retrieved.some((c) => q.answerChunkIds.includes(c.id))) hits++;
}
console.log(`hit rate @5: ${((hits / goldenSet.length) * 100).toFixed(1)}%`);
// below ~90%? stop tuning prompts. your problem is retrieval.Tip
Build the golden set from real logged questions, not questions you invented. You will phrase things the way your documents do; your users won't. That gap is exactly what you need the eval to catch.
Suspect #3 — right chunk, wrong position
Embedding search is a great scout and a mediocre judge. Bi-encoders compress every document into one vector before the question ever arrives, so they find the right *neighborhood* but fumble the final ordering. The fix is the step most pipelines skip: retrieve wide (30–50 candidates), then let a cross-encoder reranker read each candidate *with* the query and re-sort. When we added exactly this step to the retrieval pipeline in my work section, relevance went up 3x. Same corpus, same embeddings, same model — the good chunk just finally made it into the window.
Suspect #4 — the question doesn't look like the documents
Users write questions: 'why did my payout bounce?' Documents write statements: 'Settlement failures occur when the beneficiary account fails verification.' Those two strings share almost no vocabulary, and vector similarity can only stretch so far. Query rewriting closes the gap — expand the user's question into the dialect your corpus speaks, or generate a hypothetical answer and search with *that*. It's one cheap LLM call before retrieval, and it regularly rescues the long tail.
The suspect everyone arrests first
Fine-tuning embeddings is real and occasionally the answer — when your domain's vocabulary is so specialized that generic embeddings genuinely can't tell your concepts apart. But it's the last resort, not the first move, because it's the most expensive fix with the slowest feedback loop. Nine times out of ten the actual problem was messy source data: outdated docs, duplicate pages disagreeing with each other, or chunks nobody ever spot-checked.
RAG failures are supply-chain failures. The model is the chef at the end of the line — before blaming the cooking, check what the trucks delivered.
The debugging order, on one napkin
- 01Read ten retrieved chunks raw. Would a smart human answer correctly from these? If no — pipeline problem, not model problem.
- 02Measure hit rate on a golden set of real questions. Below ~90%: fix chunking and retrieval first.
- 03Hit rate fine but answers still off? Add a reranker — position matters as much as presence.
- 04Rewrite queries so questions speak the corpus's language.
- 05Only then consider fine-tuned embeddings or a bigger model — and check the source data one more time first.
Confidently wrong is the worst failure mode software can have, because it looks exactly like success. Measure the pipeline, and it stops being a mystery and starts being a checklist.
Key takeaways
- 01Debug RAG in pipeline order — chunking, retrieval, ranking, query shape. The model is the last suspect.
- 02Retrieval hit rate on a golden set of real user questions is the single most useful RAG metric.
- 03Retrieve wide, then rerank with a cross-encoder. Position in the context window matters as much as presence.
FAQ
Related reading