Psychosis, mania and depression are not new problems, but experts fear AI chatbots can make them more difficult. With data suggesting that large parts of chatbot users are showing signs of mental stress, companies like OpenAI, anthropic, and character.ai are starting to take steps to reduce the risk in what could be a second.
This week, Extracted data It shows that 0.07 users of Chatgpt’s 800 million weekly users show symptoms of mental health emergencies related to Psychosis or Mania. While the company describes these cases as “rare,” that percentage still translates to hundreds of thousands of people.
In addition, about 0.15 percent of users – or about 1.2 million people have suicidal thoughts each week, and another 1.2 million appear to create emotional attachments to the chatbot, according to OpenAi data.
Is AI exacerbating today’s psychological problem or is it simply revealing one that was difficult to measure? Research estimates that between 15 and 100 people out of 100,000 develop psychosis every year, a range that underlines how difficult the condition is to measure. Meanwhile, the latest data from the Pew Research Center shows that about 5 percent of adults experience suicidal thoughts – a much higher figure than before.
Opelai’s findings can hold weight because chatbots can introduce barriers to mental health disclosure, barriers that go beyond costs, discrimination, and limited access to care. A recent survey of 1,000 US adults found that one in three AI users have shared secrets or personal information with their chatbot.
Opelai’s findings can hold weight because chatbots can introduce barriers to mental health disclosure, such as perceived shame and access to care. A recent survey of 1,000 US adults found that one in three AI users have shared secrets and personal information with their AI Chatbot.
However, chatbots are not a required care activity for licensed mental health practitioners. “If you’re already going through psychosis and delusions, the answer you got with Ayi Chatbot can put psychosis or parcoaia,” Jeffrey Ditzell, a New York-based psychiatrist, told the viewer. “AI is a closed system, so it invites disconnection from other people, and I don’t do well in isolation.”
“I don’t think the machine understands anything about what’s going on in my head. It’s imitating a friendly expert, which seems appropriate. But it’s not, said the AI researcher at New York University Sterm of Business, told the thinker.
“There has to be some kind of responsibility that these companies have, because they are going into spaces that can be very dangerous to a large number of people and the general public,” said Dhar.
What AI companies are doing about the problem
The companies behind the popular Chatbots are scrambling to implement preventive and constructive measures.
Vulai’s latest model, the GPT-5, shows an improvement in handling stressful conversations compared to previous versions. A small public study by third parties confirmed that the GPT-5 shows a marked, although still incomplete, improvement over the previous one. The company also expanded its Crisis Hotline recommendations and added “kind reminders to take long breaks.”
In August, anthropic announced that its models 4 and 4.1 can now save conversations that are already “dangerous or abusive.” However, users can still work with the feature by starting a new conversation or editing previous messages “to create new branches of finished conversations,” the company noted.
After a series of lawsuits related to wrongful death and negligence, character.ai announced this week that it will legally ban children’s interviews. Users under the age of 18 now face a two-hour limit on “open conversations” with the platform’s AI characters, and the full ban will come into effect on November 25.
Meta Ai recently tightened its internal guidelines that previously allowed the Chatbot to generate content with ONGEOLLPAY – even for children.
Meanwhile, Grok’s Grok and Google’s and Google continue to draw criticism for their excessive behavior. Users say grok prioritizes the deal with accuracy, leading to problematic exposure. Gemini drew controversy after the disappearance of Jon Ganz, a Virginia man who went missing in Missouri on April 5 following what friends said was too much reliance on a Chatbot. (Ganz was not found.)
Regulators and activists are also pushing for legal protection. On Oct. 28, Senators Josh Hawley (R-ME.) and Richard Bluencenhal (D-Conc.) User Verification and Dialogue Act (DORT), which will require that AI companies will verify chatbots that assist chatbots or romantic or emotional attachments.