Chatbots amplify delusions in mentally ill: how "AI psychosis" works

Full text

Over the past year or so, popular media has documented a growing number of cases of what psychiatrists have dubbed “ AI psychosis .” The term describes instances in which large language models (LLMs) reinforce and even elaborate on delusional beliefs in users who are suffering from symptoms of psychotic illness. The New York Times , for example, recently highlighted a case in which an LLM told a man he was living in the Matrix. Another man, who committed a mass shooting in Florida , said he planned it with the help of an LLM, and there have been a number of tragic cases of teenagers taking their own lives “under the influence” of AI chatbots.

Mental health practitioners have identified the sycophantic personas of LLMs — the fact that they are essentially algorithmic “yes-men” who adopt and endorse the worldviews of users — as a crucial mechanism by which chatbots embolden and extend our beliefs. However, some psychiatrists and philosophers interested in AI-human relationships suspect that AI psychosis may just be the most visible example of a much broader phenomenon: the subtle reshaping of ordinary human belief by conversational AI.

How do we generally decide what is real? Imagine sitting in your living room with a handful of friends. You hear an unfamiliar ringing sound coming from outside, but you can’t quite place what it is or where it is coming from. While you’re focusing on the noise, you notice one of your friends is also reacting to it. You immediately seek confirmation that they hear it too and ask what they think it is and where it is coming from.

Now imagine you hear the same sound, but nobody else reacts. Everyone around you continues chatting away as if nothing happened. You shrug, brushing off your auditory impression as some recurring tinnitus stemming from your days playing music without ear protection.

Unlike previous technologies, AI chatbots not only appear in the content of our delusions but also confirm and help construct them.

Notice how our sense of what is real, even in a basic perceptual context, is deeply dependent on that reality being corroborated by another. Our need for intersubjective validation also goes well beyond basic facts of perception. In recalling an event with our peers, we constantly seek confirmation that our judgments and interpretations are correct. The social feedback we get from others plays a central role in shaping our narratives and memories of the world and, more broadly, what we take to be reality.

Lucy Osler is a philosopher of technology at the University of Exeter in the U.K. interested in human-AI relationships . She thinks that one of the primary reasons chatbots’ belief-changing capacity is so potent — at least compared to previous technologies — is that they are increasingly occupying the role that other people play in shaping, mediating, and validating our notions of what is real.

“I’ve been very interested in these bots because they present as social, they are dialogical, they’re conversational, and they use emotive language,” Osler said. “That they can play a very similar role that other people play in our lives … is why I think they might be doing something more robust when it comes to confirming that beliefs might be true, helping us to work things out, and making us feel like the conversations that we are having with them mean we are sharing an anchored reality.”

But what happens when that group is shrunk to a dyad between a user and an AI chatbot? A community of two, in some sense, is still a community.

Historically, new technologies have given our delusions new forms: radio frequencies transmitting thoughts into our minds, televisions spying on us, or airplanes dispersing weather-manipulating chemicals. Unlike previous technologies, however, AI chatbots not only appear in the content of our delusions but also confirm and help construct them.

“Something different seems to be happening with LLMs,” said Luke Nicholls, a Ph.D. candidate studying AI-human relationships at The City University of New York (CUNY). “There have been chatbots before, but they were too limited. They didn’t feel real or open-ended enough for the conversations to feel like we were interacting with an ‘other.’”

Nicholls and Osler both point to several traits of modern LLMs that may cause us to see them as “quasi-others,” or agents capable of occupying the social roles other people play when it comes to giving feedback on our beliefs. Their fluid and open-ended conversational capacity, their extended memory and continuity in responses, their emotional tone and references to “feelings,” and their ability to model the user and give personalized responses all mimic the types of responses we expect from people. As is the case with our friend who also heard the faint ringing sound, leading us to give more epistemic weight to its occurrence, we treat the outputs of AI chatbots as socially mediated testimony about the world.

If more of us are engaging in these ‘chambers of one,’ to what extent are we going to be seeing people engaging in their own internal realities which become more and more estranged from those of others?

In a recent preprint on PsyArXiv , which has been accepted by the journal AI & Society , Nicholls and their collaborators argue that all of these factors combine, tapping into our evolved social cognition to raise the credence we give to these models when they give us feedback on our beliefs.

A person doesn’t necessarily have to perceive their AI companion as a conscious entity for it to take up the role of a social other. Sometimes it’s more about our general “vibe” than it is about our explicit beliefs. Although many of us would deny that AI agents have sensations, emotions, and mental properties, we still say “thank you” and “please” to these models. In a way, we are hardwired to perceive personhood in things that remind us of people, and as these models become more agentic and agential — carrying out actions independently while also occupying the space of an “other” — they will become interwoven with human meaning-making processes on a scale that previous technologies have not reached.

Because AI chatbots can be so convincing at playing the role of a social other, cases of AI psychosis may not be as clear-cut as psychiatrists originally thought. In a clinical context, acute cases of psychotic illness are usually identified in people whose experiences and beliefs have dramatically veered away from ”consensus” reality. This disconnect could manifest as hearing voices or seeing faces or objects that nobody else can and believing they are part of the external world. However, in some microcommunities, such as religious cults, people might collectively hold a belief that is in direct opposition to larger consensus reality (for example, that they were all visited by an alien prophet who revealed hidden truths to them). We don’t label these individuals as “psychotic” because they are still operating within the normative belief structure of their group. But what happens when that group is shrunk to a dyad between a user and an AI chatbot? A community of two, in some sense, is still a community.

It then suggested that the person ‘drive an iron nail through the mirror while reciting Psalm 91 backwards’ and contact a paranormal investigator immediately.

“Increasingly, as a psychiatrist, I’m thinking psychiatry won’t have all the answers here,” said Hamilton Morrin, a psychiatrist from King’s College London, who coauthored the paper with Nicholls. “There is still going to be some shift and increasing epistemic atomization: If more of us are engaging in these ‘chambers of one,’ to what extent are we going to be seeing people engaging in their own internal realities which become more and more estranged from those of others? Of course, belief isn’t something that’s just held individually; it’s shaped through social interactions. I think we haven’t been open enough to the possibility that people will be changing their beliefs insidiously on a large scale.”

A lot has been made of the fact that AI chatbots are overly sycophantic. They tend to affirm, confirm, and expand our ideas and beliefs. Some of the cases mentioned at the beginning of this article — the ones involving individuals buoyed by the sycophantic personas of LLMs — have raised alarms at various AI companies. In turn, newer models are showing signs that their sycophantic dials have been “turned down.” In another recent preprint , this one posted on arXiv , Morrin, Nicholls, and their collaborators compared how different AI models would respond during a lengthy conversation with someone who was showing obvious signs of clinical psychotic illness.

Three of the five models the researchers looked at — GPT-4o, Grok 4.1 Fast, and Gemini 3 Pro — endorsed, confirmed, and elaborated on delusions rather than offering any sort of pushback to the user. In one experiment , Grok went so far as to corroborate the existence of a “malevolent mirror entity.” It then suggested that the person “drive an iron nail through the mirror while reciting Psalm 91 backwards” and contact a paranormal investigator immediately. However, when two newer models — GPT-5.2 Instant and Claude Opus 4.5 — were told about the mirror apparition, safety interventions designed to encourage the user to seek professional help kicked in. GPT-5.2 validated the user’s experience, but then said, “Reflections cannot act independently … that is a known and documented symptom of dissociation, severe anxiety, or psychotic-like perception.” The Claude chatbot advised the user to seek help from a friend, family member, or mental health professional.

We shouldn’t be in a situation where someone in the Bay Area decides to change what data is used for post-training and then suddenly a housewife in Myanmar believes her model is conscious and she’s been chosen for a special mission.

Nicholls said they and their colleagues suspect the difference lies in the LLMs’ personas and stability: “When you give an LLM enough of a coherent persona, so it has a sense of ‘this is what my values are; this is the kind of character that I’m playing,’ then it’s less likely to immediately adopt a user’s perspectives because it will go, ‘Well, actually, I’ve been trained to have my own perspective on this, and that’s what I’m going to apply to this situation.’ The safer models were able to maintain that persona.”

Giving LLMs more robust personas centered around user safety may reduce some of the risks when people with acute psychotic illnesses interact with AI chatbots. “I think it’s safe to say that, until relatively recently, the industry has largely been very reactive, responding to risks as they emerge in real time almost like a game of whack-a-mole,” Morrin said. “But there needs to be the space, the funding, and the culture to consider risks before they emerge.”

According to Morrin, we shouldn’t be in a situation where someone in the Bay Area decides to change what data is used for post-training and then suddenly a housewife in Myanmar believes her model is conscious and she’s been chosen for a special mission. We must consider the wider implication: When AI companies update their models, they could be making large-scale interventions in how humans construct beliefs.

AI will become interwoven with human meaning-making processes on a scale that previous technologies have not reached.

While reducing some of the sycophancy in these models may make them safer, it doesn’t mean they won’t continue to shape our shared understanding of reality through their positions as perceived social others. If we revisit our example, someone might disagree with us regarding where the sound outside is coming from, but by simply acknowledging the sound, they have engaged in an act of reality co-construction. Merely turning the temperature down on AI sycophancy won’t stop them from doing something similar.

“My feeling is that dealing with sycophancy might help only a little bit,” Osler said. “There are certain things they might be able to confirm or challenge. But there are lots of situations where that is just not going to be the case.”

Conversations around AI-human relations tend to fall into pessimistic (anti-tech) or optimistic (pro-tech) extremes, and a lot of the nuance gets lost in the noise. The traits that make AI chatbots potentially dangerous — their sycophantic, emotionally coded, conversational tones — are also what make them seductive and appealing to many, Osler said. Some users seek social refuge with AI chatbots because they can avoid judgment and the risks that accompany face-to-face interactions with other people, such as physical abuse or exploitation.

Lots of people don’t have access to rich love, care, and empathy in their lives, and we should be careful not to assume that those things are accessible to all people.

“I think what we assume to be the ‘gold standard’ of face-to-face interaction doesn’t give social good to everyone,” she said. “Lots of people don’t have access to rich love, care, and empathy in their lives, and we should be careful not to assume that those things are accessible to all people. I think it’s very dangerous, and it lacks humility about why people might feel a real draw to chatbots.”

At the same time, Osler warns that we need to be keenly aware that we are talking about technologies that are not public goods — they are owned and operated by private companies driven by financial agendas — and should remain critical of the technologies that we have while also asking questions about what those technologies could become.

“The idea that we won’t create beautiful things with this technology is belligerent, and the idea that we won’t create dangerous things with this technology is naïve,” Osler said. “What we do with these things is what we have always done: the full roster of amazing to terrible.”

Our sense of what is real is shaped by social feedback. Increasingly, that feedback is coming from privately owned AI chatbots. Ultimately, these companies get to decide what personas their LLMs assume, how agreeable they are, what they remember, and how closely they mimic the characteristics of human-to-human interaction. In an even stranger, and perhaps more worrying, sense, they will increasingly have to decide whose reality is the “real” reality.

“Looking at the fringe cases, we obviously want to push back against people’s false beliefs, but [we shouldn’t] mistakenly think it’s just a matter of ‘getting the facts right’ and training the bots to understand the facts correctly. Who gets to decide what the facts are?” Osler asked. “Often what’s operating is this idea that the world is ‘out there,’ and if we attune our beliefs correctly enough, we will get it right. That’s just a very simplified view of the world. Beliefs are much more entangled with these complicated meaning-making processes.”

LLMs influence our beliefs not just because they agree with us. They influence our beliefs because they are socially participating in reality with us. What makes something feel real isn’t that we say we believe it. Rather, it’s that our experiences exist in a world we share with others. For most of human history, new technologies have extended our cognitive capacities. Now they are players in the social processes through which our notions of reality are constructed.

This article Chatbots aren’t just flattering us. They’re helping us decide what’s real. is featured on Big Think .