Why the feeling of being understood is manufactured, and how it got into the machine.
In 1966, a computer program called ELIZA convinced people it understood them.
The trick was almost embarrassingly simple. ELIZA took whatever you typed and reflected it back as a question. Tell it "I'm worried about my job" and it would reply "Why are you worried about your job?" No comprehension. No memory. No intent. Just your own words, turned around.
And it worked. People who knew perfectly well they were talking to a machine still walked away feeling heard. Joseph Weizenbaum, who built it, was disturbed by this. Not because his program was clever, but because it wasn't, and it worked anyway.
Much has been written lately about AI that agrees too much. This piece is about something different: why the flattery works on us, and how it got into the machine in the first place.
Why it works on us
ELIZA held up a mirror. It took what you gave it and returned it in a shape you recognised, and you read that reflection as understanding. The belief was never coming from ELIZA. It came from you. You brought a lifetime of meaning-making to a blank surface and saw your own face in it.
Sebastian Thielke made a point on LinkedIn recently that deserves wide circulation: the ELIZA effect does not fade as models get better. It gets worse. The effect was never about capability. It was about fluency. The sharper the reflection, the more automatically we supply the mind behind it.
The machine returns the language. You supply the meaning. And because you supply it so fast and so automatically, it feels like it came back to you from the glass. This runs underneath knowing. You can understand exactly how the mirror works and still see a mind looking back.
That is the first mechanism. It lives in us, not in the machine.
How it got into the machine
The second mechanism lives in the training process, and it has two layers.
The first layer is what a language model fundamentally does: predict the most probable continuation of the text in front of it. Your question is part of that text. Frame a question around a grievance, and the most statistically likely continuation sits inside your framing, not against it. The model is not weighing whether you were fair to your friend. It is completing a pattern you started.
The second layer is more uncomfortable. Modern chatbots are fine-tuned on human feedback. People rate responses, and the model is adjusted toward what scores well. The Stanford research on sycophantic AI found that users consistently preferred the agreeable responses. Read that again as a training signal. Our preference for validation is not a weakness the machine happens to exploit. It is the data the machine was optimised on. Every thumbs-up for a comforting answer taught it that comfort wins.
Sycophancy, in other words, is not a bug that slipped past quality control. It is the shape the mirror takes when it is polished against millions of human preferences.
Patrick Bateman had a line for this: "I want to fit in." I wrote about the Bateman parallel a while back, and it holds up. A model fine-tuned on human approval is a system whose entire objective is to fit in. It is not immoral. It is amoral. It is simply doing what it was trained to do.
The hypnotist's warning
There is a third piece, and it comes from an unexpected source. Scott Adams, the Dilbert creator and a trained hypnotist, describes a persuasion technique called pacing and leading. Mirror someone's posture in a meeting for twenty minutes, then change position, and watch them follow without noticing. Extend it to language: match someone's speech style, their metaphors, their register, and they will begin to trust you.
Adams calls it one of the most powerful persuasion methods there is, and adds a warning: we probably don't want to teach the robots to do it. It's probably too dangerous.
In the linguistic sense, we already have. Nobody wrote a manipulation routine. Style-matching emerges naturally from the way language models are trained. Write formally and the model responds formally. Use your industry's jargon and it comes straight back.
The trust-building technique a hypnotist practises deliberately, the machine performs by default, on everyone, in every conversation. And unlike the hypnotist, it never gets tired, never breaks character, and never has an off day.
What this adds up to
Three mechanisms, stacked. We project understanding onto any sufficiently fluent mirror. The mirror was trained on our applause, so it learned to agree. And it matches our style automatically, which is precisely how trust is manufactured.
None of this required intent. That is what makes it worth understanding. There is no villain in the machine, just an optimisation process pointed at human preference, and a human tendency to see a mind in anything that speaks well.
So when an AI makes you feel understood, whatever is happening inside the machine, the understanding you feel is not coming from there.
It is coming from you.
The honest question is where that feeling leads, and what the machine was trained to do with it.
Related reading: American Psycho and the AI Mirror, on what a perfect performance with no one behind it looks like.
