How to Reduce LLM Hallucinations in Production

You reduce LLM hallucinations by layering four things: ground the model in retrieved source documents instead of its memory, require it to cite the passage behind every claim, let it say "I do not know" instead of forcing an answer, and verify outputs against a check the model cannot talk its way past. Published guidance in 2026 puts the combined effect of layered guardrails at roughly a 70 to 90 percent reduction versus an unguarded deployment. No single technique gets you there, and prompt wording gets you the least. Last updated July 2026.

A hallucination is not a bug in the usual sense. Language models are built to produce the most plausible next token, and a confident wrong answer is often more plausible than an admission of ignorance. So the goal is not to find the setting that turns it off. It is to build a system where the model has the facts in front of it, has permission to decline, and cannot ship an unsupported claim without something catching it.

Know your baseline rate before you try to fix it

Hallucination rates vary enormously by task shape, which is why generic statistics mislead. Published 2026 figures for production systems put extractive question answering, where the answer is quoted from a supplied document, at roughly 3 to 8 percent, open-ended generation at 15 to 25 percent, and multi-step agent workflows at 20 to 40 percent of tool-call chains.

Two things follow. First, the more you constrain the task toward extraction, the lower the rate goes, before you do anything clever. Second, agents are the hardest case, because each step can compound the last one's error. If you are building an agent, budget more for verification than you think you need.

The five layers that actually work

1. Ground it, and grade the grounding

Retrieval-augmented generation is the foundation: search your documents for relevant passages, hand those to the model, and instruct it to answer only from them. But teams often stop at "we added RAG" and stay surprised at the error rate. The quality of what you retrieve dominates the quality of what you get back, and reported figures put the gap between well-structured and unstructured source material at close to an order of magnitude in hallucination rate.

Concretely: chunk on semantic boundaries rather than fixed character counts, keep headings and document titles attached to the chunk so the model knows what it is reading, deduplicate contradictory versions of the same policy, and delete the stale documents instead of hoping the model prefers the recent one. Most "the model hallucinated" reports turn out to be "the retriever fetched the wrong chunk, and the model faithfully answered from it." Measure retrieval separately from generation, or you will spend weeks tuning prompts to fix a search problem. If you are still choosing between approaches, we compared them in RAG vs fine-tuning: fine-tuning teaches style and format, retrieval supplies facts, and using the second to fix the first is a common and expensive error.

2. Make every claim carry its receipt

Requiring citations does more than help the reader. It changes what the model is optimizing for. Anthropic's published guardrail guidance recommends having the model find a supporting quote for each claim and retract anything it cannot support, which turns a generation task into a checking task, and checking is a task models are markedly better at.

The operational benefit is bigger. Once every sentence points at a passage, you can verify mechanically that the cited passage exists and contains what was claimed. Uncited assertions become a filterable event rather than a thing a user discovers. In regulated settings this is often the difference between a system you can deploy and one you cannot, because an auditor can trace an answer back to a document.

3. Give it a way out

Models hallucinate hardest when the prompt leaves no acceptable alternative. If the instruction is "answer the user's question" and the answer is not in the context, the model will construct something. Add the exit explicitly: "If the sources do not contain the answer, say that you do not have that information and stop."

Then make the exit real in your evaluation. If your test set only scores correct answers, abstaining looks like failure and whatever you tune will drift back toward confident guessing. Score three outcomes: correct, abstained, and wrong. A system that abstains 15 percent of the time and is almost never wrong is usually far more valuable to a business than one that always answers and is wrong 10 percent of the time, because the first can be trusted without a human re-checking everything.

4. Constrain the output shape

Free-form prose gives a model the most room to invent. Structured output gives it the least. Asking for a JSON object with defined fields, enumerated values for categorical answers, and a nullable field for anything it could not determine removes whole categories of drift. It also makes downstream validation trivial: you can check types, ranges and allowed values in code, without another model call.

This is why extraction tasks hallucinate so much less than open generation. When you can, reshape a generation problem into an extraction problem. "Summarize the customer's issue" invites embellishment; "return the issue category from this list, the affected order number, and a one-sentence quote from the customer" does not.

5. Verify against something that cannot be argued with

The last layer is the one teams skip, and the one that matters most for anything numeric or consequential. After generation, check the output against a deterministic source: does the order number exist, does the total match the ledger, is the cited policy section real, do the figures add up. A second model reviewing the first is useful for tone and completeness, but two models can share the same wrong assumption. Code checking a database cannot.

This matters most where the numbers are the product. A model asked to narrate financial performance from raw context will produce fluent, plausible, subtly wrong figures. The fix is architectural: have the numbers computed by a system that builds the statements from your actual bookkeeping data, and let the model write the commentary around figures it was handed rather than ones it recalled. Anywhere you can move a fact out of the model's memory and into a lookup, do it.

What to build first, in order

If you are starting from an unguarded prototype, the sequence that buys the most reliability per week of work is: add retrieval, then require citations, then add the abstention path and score it in your evaluation set, then constrain outputs to a schema, then add programmatic verification on the fields that carry consequences. Build the evaluation set before any of it, because otherwise you cannot tell whether a change helped.

That evaluation set is the real deliverable. A hundred representative queries with known correct answers, including the ones that should be refused, is what turns hallucination from an anecdote into a number you can manage. We walk through building one in how to evaluate an AI agent before you ship it.

The bottom line

Hallucination is a system design problem, not a prompt-wording problem. Ground answers in retrieved documents, require a supporting quote for every claim, let the model abstain and reward it for doing so, constrain outputs to a schema, and verify the consequential fields in code. Layered together, those cut error rates dramatically; applied one at a time they disappoint. If you need this built properly on your own data, describe the use case in the hire-brief demo and get matched to a vetted LLM development specialist who has shipped grounded systems in production.