Nine years ago, I wrote a post titled “Search Engines are not foolproof.” The complaint was simple: you type into one box, you press Enter, and you assume the machine is doing something like looking for the truth. It isn’t. It’s looking for relevance. Relevance is not truth.
At the end of that post, I took a swipe at the companies building all of this and asked: “Will those technologies also be so sensitive to an omission of ‘THE’ from your question?”
I want to come back to that because the box talks back now.
The box answers you now.
The change since 2017 is that you no longer get 10 links. You get a paragraph. It’s in complete sentences, well organised, answers the question you actually asked rather than the one your keywords accidentally asked, and sounds like it knows.
That last part is the whole problem.
What the thing actually is
Very simplified — and I mean very: a large language model is a text continuation machine.
It was trained by being shown an enormous quantity of text and asked, over and over, to guess what came next. Not “is this true.” What comes next? Do that enough times, and you end up with something that has absorbed the statistical shape of human writing: which words tend to follow which words, what a confident explanation looks like, how a paragraph about medieval history is put together, how a C# method is usually written.
When you ask it a question, it is not looking anything up. There is no fact database in there to consult. It is generating the most plausible continuation of your text, one piece at a time.
That’s it. That’s the trick. It is genuinely remarkable how far that gets you — and it tells you exactly where it breaks.
Plausible is not true
In 2017, the line was: Google doesn’t care what’s true; it cannot possibly care what’s true; it cares about relevance.
The line today is nearly identical. A language model doesn’t care what’s true. It cares about what’s plausible. And here’s the trouble: a plausible sentence and a true sentence are made of the same material. That’s why plausibility works as a proxy most of the time, and it’s why you cannot tell from the outside when it stops working.
When Google failed me in 2017, I could see it fail. The dinosaur answer came from a creationist site and said so, right there in the URL. The failure had a domain name attached to it.
When a model fails, it fails in the same calm, articulate, nicely formatted prose it uses when it’s right. The invented court case is set in the same font as the real one. The fabricated citation is formatted correctly, down to the page numbers. The confidence you are reading is a property of the writing style, not of the accuracy — and there’s no second result underneath it to compare against. In that one specific respect, this is worse than 2017, not better.
Your bias, faithfully completed.
The old post argued that a search engine can’t correct the bias in your query. Search “is milk bad,” and you get results saying milk is bad. Search “what should we eat”, and you get somewhere better.
A model doesn’t merely fail to correct your bias. It finishes your sentence for you. Ask why X is bad, and the most plausible continuation of “why is X bad” is a well-organised list of reasons X is bad. It isn’t lying, and it isn’t conspiring. It’s doing the only thing it does: producing text that fits.
So the 2017 advice — correct for bias in your query — applies more directly now than it did then. Ask what the evidence says. Don’t ask it to agree with you, because it is very, very good at agreeing with you.
One more thing: most of these tools now search the web before answering, and it does help. But look at what that actually does. It doesn’t remove the 2017 problem; it buries it. Now you have unreliable search results (problem one) summarised confidently by a plausibility engine (problem two), with the seams sanded off. Click the citations. Sometimes they don’t say what the summary says they say.
So, what should we do? What are the Conclusions?
Roughly what they were, which is sort of the point of writing this nine years later.
- Take everything the model gives you with a grain of salt. Fluency is not evidence.
- Correct for bias in your query. Ask what the evidence says, not for confirmation of what you walked in with.
- Check anything specific — names, numbers, dates, quotes, citations, court cases, API methods. Specifics are where it breaks, and it breaks silently.
- Match the checking to the stakes. Brainstorming? Don’t bother. Medical, legal, financial, or going into production? Verify every claim individually.
- Use it for what it’s actually good at: drafting, restructuring, explaining a concept you’ll recognise as right or wrong the moment you read it. Those play to plausibility instead of against it.
- Proportion your beliefs to the evidence. Think critically. Remain skeptical. Look for legitimate sources.
- Rinse and repeat.
None of this makes these tools evil, a conspiracy, or useless. I used one to help write this post, which is either the joke or the point. It makes them tools with a particular failure mode, like everything else.
In 2017, the failure mode was loud and wore a bad domain name. Now it’s quiet, and it writes better than I do.
Comments