RAG Evaluation: How to Tell If Your Retrieval Is Working

To tell if your RAG system's retrieval is actually working, measure two things separately: retrieval quality (did the system fetch the passages that contain the answer?) and answer quality (did the model use them faithfully?). Most RAG failures are retrieval failures, so if you only look at the final answer you cannot tell whether the model made something up or was simply never handed the right text. Build a fixed test set of real questions with known correct sources, and score both layers. Last updated July 2026.

Retrieval-augmented generation reduces hallucinations by grounding answers in your documents, often by 40 percent or more, but it does not remove them. When a RAG chatbot gives a confidently wrong answer, the instinct is to blame the model. Usually the model is not the problem. It answered faithfully from whatever it was given, and it was given the wrong passages, or no useful passage at all. You cannot fix that by swapping models. You have to measure the retrieval layer on its own.

Why you must measure retrieval and generation separately

A RAG answer can be wrong for two very different reasons. Either the retriever failed to find the passage that contains the answer, or the retriever found it and the model ignored or misread it. These need opposite fixes. A retrieval failure is solved with better chunking, hybrid search, or re-ranking. A generation failure is solved with better prompting or a stronger model. If you only score the final answer, you are guessing which one you have, and teams routinely spend weeks tuning the model when the real problem was that the right document never made it into the context. Separating the layers this way is also the most effective lever you have for reducing LLM hallucinations, because a model that never received the right passage will fill the gap itself.

Build a fixed evaluation set first

Before you tune anything, assemble a test set of real questions users actually ask, and for each one record the correct answer and which source document or passage it should come from. Thirty to fifty questions is enough to start, as long as they are representative, including the ambiguous and rare ones, not just the easy demo questions. Hold this set back so it stays a fair measure. Every change you make to chunking, embeddings or prompts gets scored against the same set, so you can see whether you improved things or just moved the failures around. This is the retrieval-specific case of the wider discipline in how to evaluate an AI agent before you ship it, and the quality of the source documents you are indexing is worth checking against a data readiness checklist first.

The retrieval metrics that matter

For the retrieval layer, the core question is whether the correct passage was fetched at all, and how highly it ranked. A few practical metrics:

If hit rate is low, the answer was never available to the model, and no amount of prompt tuning will save it. That points you straight at chunking, the embedding model, or adding keyword search alongside semantic search. If hit rate is high but the answers are still wrong, the problem has moved downstream to generation.

The generation metrics that matter

Once you know the right passages are being retrieved, score whether the model used them honestly. The key measure is faithfulness: does every claim in the answer trace back to the retrieved context, or did the model add something that is not there? The second is answer relevance: did it actually address the question. A common and useful pattern in 2026 is to use a separate model as a judge to score faithfulness against the retrieved passages, calibrated against a sample you have checked by hand so you trust the judge before you trust its scores.

Why a bigger model is rarely the fix

When a RAG system underperforms, upgrading the model is the tempting move because it is the easiest. It rarely helps, because the dominant failure mode is retrieval, and a smarter model cannot answer from a passage it never received. The teams that get RAG working treat retrieval quality as the main event: they fix chunking so passages are self-contained, add hybrid and re-ranked search so the right chunk surfaces, and only then look at the model. This is the same reason the choice between grounding a model and changing its weights matters less than people think, a trade-off we cover in RAG vs fine-tuning.

Keep measuring after launch

Retrieval quality drifts. As you add documents, change sources, or your users start asking new kinds of questions, a system that scored well at launch can quietly degrade. Production RAG needs monitoring that watches answer quality over time and alerts you when it slips, the same way you would get notified the moment a service starts failing rather than finding out from an angry customer. Treat evaluation as a permanent part of the system, not a one-time gate, and your RAG app stays trustworthy as it grows.

The short version: separate the two layers, build a fixed test set, and score retrieval before you touch the model. Do that and you stop guessing why your assistant is wrong. If you want that discipline built in from the start, our RAG development services page explains how a vetted engineer bakes evaluation into the build rather than bolting it on after users complain.