Hallucination: the $5,000 lesson, and what actually reduces it.
Lawyers were just sanctioned for filing ChatGPT's invented citations. Notes from production on the stack of controls that actually reduces hallucination.
⚠️ Hallucination: the $5,000 lesson, and what actually reduces it.
This week a federal judge sanctioned two lawyers $5,000 for filing a brief containing six court cases that ChatGPT invented — complete with case names, quotes and citations. When challenged, the model had assured them the cases "indeed exist." They filed that assurance too.
Everyone building with LLMs right now should read that story twice, because it is a perfect specimen of the failure mode: not a wrong answer, but a confidently formatted, verification-resistant wrong answer. Having spent this year putting grounded bots into production, here is my working list of what reduces hallucination — in increasing order of how much it actually matters:
1. Temperature — the first dial, and the most overrated. Temperature controls sampling randomness. Turn it down toward zero for factual tasks and outputs become more deterministic and less prone to creative drift — genuinely useful, always do it. But understand what it does not do: a model at temperature zero will still fabricate, just consistently. Those six fake cases were not a randomness problem; the model does not know what it does not know, at any temperature. Temperature reduces variance, not fiction.
2. Grounding — the biggest single lever. Retrieve the relevant documents first, then instruct the model to answer only from what was provided. This converts the task from "recall from your training" (where hallucination lives) to "read and synthesise this text" (where models are genuinely strong). Most of our hallucination reduction came from this one architectural decision.
3. Mandatory citations — trust through checkability. Every claim links to its source chunk; no source, no answer. This does two things: it gives the reviewer a one-click verification path, and — in our experience — the citation requirement itself disciplines the output, because unsupported claims have nowhere to point.
4. Permission to say "I don't know." Models are tuned to be helpful, and unhelpfully so: asked a question outside the provided context, the default behaviour is to attempt an answer. Explicitly instructing — if the sources do not contain the answer, say so — and treating "I don't know" as a successful outcome in your metrics changes behaviour more than any clever prompt.
5. Scope design — prefer extraction over generation. "Summarise what this document says about penalties" hallucinates far less than "what are the penalties?" The narrower the creative license, the smaller the room for invention. Design tasks so the model transforms text you gave it rather than produces text from memory.
6. A verification pass. For anything consequential, a second step — same model or another — checks the draft against the sources before a human sees it. Cheap, catches the embarrassing fraction.
7. A human gate where the cost of wrong is high. The lawyers' real failure was not using ChatGPT; it was skipping the gatekeeping their profession already required. Every domain has its equivalent. The gate is not a concession that AI failed — it is part of the system design, sized to the blast radius.
The uncomfortable summary: hallucination is not a bug awaiting a patch. It is inherent to how these models work, and it is managed, layer by layer, the way engineering has always managed unreliable components. Anyone selling you a hallucination-free model is doing to you what ChatGPT did to those lawyers.
One question before your next AI feature ships: when it is wrong — and it will be — what in your design makes the wrongness visible before it becomes a filing, a quote, or a commitment? If the answer is "the user will notice," you have already met the lawyers.
Last updated