Module 1 · Lesson 2 of 5
Where AI Fails — And How to Catch It
GuideHerd Academy
Subscribe to access this lesson.
GuideHerd Academy gives your whole firm access to AI workflow training, monthly updates, and a prompt and template library.
The problem with AI errors
AI failures do not look like failures. A system that returned broken output or obviously wrong answers would be easy to catch. The problem is that AI tools produce polished, well-structured, confident-sounding text even when the underlying content is wrong, incomplete, or fabricated.
This lesson covers the specific failure modes you need to know — not to make you distrust AI tools, but to make you a better reviewer of their output.
Failure mode 1: Made-up facts
The technical term is hallucination. The model generates text that sounds factually accurate but is not. This includes invented case citations, made-up statistics, fictional company policies, and plausible-sounding regulation references that do not exist.
The hallucination problem is worst in areas that require precise factual accuracy — legal citations, medical references, financial figures, regulatory details. These are also areas where errors have real consequences.
How to catch it: Verify any specific fact, figure, citation, or reference independently before using it. Do not assume that because a citation looks real, it is real.
Failure mode 2: Missing context
The model only knows what you tell it and what it learned during training. It does not know your client's specific situation, your firm's policies, or the details of a case unless you provide them. When context is missing, the model fills the gap with plausible-sounding general content — which may be completely wrong for your situation.
How to catch it: Ask yourself: did I give the model enough context to answer this correctly? If the answer is no, the output needs more scrutiny — or the prompt needs to be redone with the relevant details included.
Failure mode 3: Overconfidence
AI tools do not reliably express uncertainty. A model that is 60% confident and a model that is 99% confident may produce output that reads exactly the same way. The hedging language ("this may vary by jurisdiction," "consult a professional") is often boilerplate that does not reflect genuine uncertainty about the specific claim being made.
How to catch it: Treat confident AI output the same way you treat uncertain AI output when the stakes are high. Confidence in the writing is not evidence of accuracy in the content.
Failure mode 4: Bad assumptions
When a prompt is ambiguous, the model picks an interpretation and runs with it — without telling you it made a choice. You asked about "the client" and the model assumed which client you meant. You asked about "the contract" and the model assumed which version. The output may be perfectly correct for the interpretation the model chose, and wrong for the one you intended.
How to catch it: Read the output carefully to confirm the model addressed what you actually meant. If the prompt was ambiguous, it probably needs to be more specific.
Failure mode 5: Outdated information
Language models have a training cutoff. Anything that changed after that date — new regulations, updated case law, revised policies — is not in the model's training data unless you provide it. Tools with web access reduce this problem but do not eliminate it.
How to catch it: For anything time-sensitive, check when the model's training data ends. Provide current documents rather than relying on the model's internal knowledge of current events or regulations.
Failure mode 6: Source mismatch
When a model produces a summary or analysis of a document you provided, it may occasionally summarize things not in the document, or attribute claims to the wrong section. The summary reads as though it came directly from the source — but the correspondence is imperfect.
How to catch it: When the output is intended to reflect a specific document, spot-check key claims against the original source. Do not assume that because you provided the document, the output is a faithful representation of it.
Failure mode 7: Boundary crossing
AI tools sometimes do more than you asked. You asked for a summary and got a recommendation. You asked for a draft and got a judgment call presented as fact. You asked for a list of options and got a recommended course of action. The model is trying to be helpful, but it may have crossed into territory that requires professional judgment.
How to catch it: Check whether the output stayed within the scope of what you asked. If the model drew a conclusion, made a recommendation, or expressed an opinion you didn't request, that part needs separate evaluation.
The right posture
None of this means AI tools aren't useful. It means AI output is draft material that requires review — not a finished product that can be passed along without checking. The failure modes above are not rare edge cases. They happen regularly, and they tend to happen in ways that are easy to miss if you are not looking for them.
The next lesson covers how to structure the review process so these failures get caught before they matter.