Sooner or later an AI tool will tell you something false, in the same calm and confident tone it uses for everything else. No hedging, no warning.
This surprises people, and it puts some off entirely. It should not. It is a predictable consequence of how these tools work, and once you know the shape of it you can spot the risky answers in seconds.
It is not lying, and it is not broken
An AI language tool has one core skill: predicting what text usually comes next.
That is not a simplification for beginners — it is genuinely the mechanism. It has read an enormous amount of writing and become extremely good at continuing a passage plausibly.
Now consider what that means for a question like "which section of the Housing Act covers this?"
There is no lookup happening. Nothing is being retrieved from a database of law. It is producing the kind of text that usually follows that question — and text that follows that question usually contains a section number stated confidently. So you get a section number, stated confidently.
It is not reporting a fact. It is producing the shape of an answer.
Sometimes the shape and the fact coincide, because the true answer appeared often enough in what it read. Sometimes they do not. From the outside, both look identical.
This is usually called a hallucination, which is a poor name — it suggests a malfunction. It is the same process working the same way. There is simply no separate step that checks whether the output is true.
Why it never sounds unsure
The confidence catches people out more than the errors do.
Text that hedges — "I think", "possibly", "you should check" — is a small fraction of the writing these tools learned from. Confident explanation is the overwhelming majority. So confident explanation is what they produce, regardless of whether the underlying content is solid.
The tone carries no information about reliability. A completely invented citation reads exactly like a correct one.
The answers most likely to be wrong
This is the useful part. Errors are not evenly distributed, and the risky categories are easy to recognise.
High risk — verify every time:
- Numbers, dates, prices, statistics
- Names of people, and who said what
- Citations, case law, section numbers, page references
- URLs and links, which are frequently invented outright
- Anything recent, especially since the tool was last updated
- Very specific details about small organisations, local rules, or niche products
Low risk — usually fine:
- Rewriting, shortening or restyling text you provided
- Summarising a document you pasted in
- Explaining a general, well-established concept
- Brainstorming, naming, structuring
- Grammar and phrasing
The pattern is the same one that governs everything else about these tools. When you supply the material, it is reliable. When it has to supply the material, it is not.
If you notice, every high-risk item is a fact it had to produce from nothing. Every low-risk item is a transformation of something you gave it.
A two-minute check
You do not need to fact-check everything. You need to check the parts that would matter if they were wrong.
- Ask what came from you. Did you paste the source, or is it recalling? Recall is where errors live.
- Pull out the checkable specifics. Every number, name, date and citation. Those are your check-list.
- Verify the ones with consequences. A statistic in a work presentation, yes. An adjective in a birthday message, no.
- Click every link. Do not trust a URL because it looks right. Invented links are one of the most common failures, and they are the easiest to catch.
- Ask it to mark its own uncertainty. "Which parts of that are you least confident about?" is imperfect — it is another prediction — but it surfaces the weakest claims more often than not.
Two useful habits on top of that: ask the same question in a fresh conversation and see whether the answer holds, and tell it explicitly not to add anything you did not provide.
What this means in practice
None of this makes AI tools unusable. It makes them a particular kind of useful.
They are excellent at the work of shaping, explaining and drafting, where you brought the substance. They are unreliable as a source of record, where they must produce the substance. Most people's bad experiences come from using them as the second when they were only ever good at the first.
Use them as a fast assistant whose drafts you read before sending, and they will save you real time. Use them as an encyclopedia and they will eventually embarrass you.
The good news is that the failure is predictable, which means it is manageable. That is not true of most things that go wrong with software.
If you want to get more out of the reliable half, how to write a prompt that actually works covers the specifics — and what AI can actually do for you is a broader tour of where these tools earn their keep.