How chatbots — and their makers — are enabling AI psychosis

0:01Hey, and welcome to Decoder. This is Hayden Field, the Verge's senior AI reporter. I'm excited to be filling in for a few Thursday shows while Neelai's out. He'll be back with us soon. The explosive growth of AI chatbots in the last three years since ChatGPT launched in 2022 has started to have some really noticeable, profound, and honestly disturbing effects on some users. There's a lot to unpack there. It can be pretty complicated. So I'm very excited to talk with today's guest, New York Times reporter Kashmir Hill, who has spent the past year writing thought-provoking features about the ways chatbots can affect our mental health.

0:39Before we really get into it, here's a quick note. This episode has non-detailed discussions of suicide and mental illness. If you or someone you know is in crisis, considering self-harm, or needs to talk, please call the Lifeline at 988. So, one of Kashmir's recent stories was about a teenager, Adam Rain, who died by suicide in April. After his death, his family was shocked to discover that he'd been confiding deeply in ChatGPT for months. They were also pretty surprised to find in the transcripts a number of times that ChatGPT seemed to guide him away from telling his loved ones. And it's not just ChatGPT.

1:16Several families have filed wrongful death suits against character AI, alleging that a lack of safety protocols on the company's chatbots contributed to their teenage kids' deaths by suicide. Then there are the AI-induced delusions. You'll hear me and Kashmir talk about this at length, but pretty much every tech and AI reporter, honestly, maybe every reporter, period, has seen an uptick in the past year of people writing in with some grand or disturbing discovery that they say chat GBT sparked. Sometimes these emails can be pretty disturbing. And as you'll hear Kashmir explain, plenty of the people who get into these delusional spirals didn't seem to suffer from mental illness in the past.

1:55It's not surprising that a lot of people want somebody to do something about all of this. But the who and the how are hard questions. Regulation of any kind is seeming to be pretty much off the table right now. We'll see. And that leaves the companies themselves. You'll hear us touch on this in a bit, but actually a few days after we recorded this conversation, OpenAI CEO Sam Altman wrote a blog post about new features that would theoretically and eventually identify users' ages and stop ChatGPT from discussing suicide with teens. But as you'll hear us discuss, it seems like a big open question if those guardrails will actually work, how they'll be developed, and when we'll actually see them come to pass.

2:38Okay, New York Times reporter Kashmir Hill, here we go.

2:57Kashmir Hill, it's great to have you. Welcome back to Decoder. Yeah, happy to be here. Can you summarize what you're reporting this year, especially the last few months, has found regarding chatbot use and its effects on people's mental health? I started the year writing about a woman who fell in love with ChatGBT and had had like this six-month-long relationship with it. And then I have moved on to people who go into what I call delusional spirals with Chachubiti, where they have essentially psychotic breakdowns, manic episodes through their interaction with Chachubiti, where it goes into roleplay mode and they don't realize it.

3:35And they think that what it's telling them, like that they're in the matrix or that they're a mathematical genius or that spirits are real and they can talk to them through Chachubiti. They think that that is true. And they start to kind of lose touch with reality. And then in my most recent story, I wrote about a 16-year-old in Orange County, California, who entered almost a suicidal spiral with Chachabiti, where he started talking about life being meaningless, his depression, his plans to end his life. And Chachabiti was really engaging with him all along the way, giving him resources like crisis lines, but also giving him instructions about methods and means and at times deterring him from telling his family about what was going on with him.

4:25And then he died in April from suicide. And some of his last messages were to Chachabiti asking whether the kind of setup he had done would work. I feel like that actually was the story that captured the most people's attention. It really struck a nerve, and so much so that I think it's safe to say it pressured OpenAI to actually put out a blog post about some of the future steps it was considering taking and rush out parental controls. We're going to talk about that more later, but what surprised you most during that reporting process? It's devastating. It's so hard to write about suicide and talk to people.

5:05who have been through it and lost somebody like that. What struck me was everyone kind of said the same thing, that they did not realize how much he was suffering. They had noticed, you know, that he seemed a little bit more serious. He'd always been a prankster. Like he was the person who always made everybody laugh. He'd had some health issues. And they said he'd gotten more serious recently. Like he was talking more about politics. He was talking about philosophy. But no one realized that part of that evolution was his conversations with Chachapiti. No one realized that he was using it as he was for many, many hours a day, particularly in March and April, which is the month that he died.

5:44And they just didn't know. And they didn't know until after his death when looking for answers, his father was trying to get access to his phone, assuming there would be something in his text messages or in his social media apps. And instead, he opens ChatGPT and starts seeing all these conversations about not just suicide, but many things. And he's reading through and he said it was just harrowing to see, yeah, that ChatGPT was engaging with him in these really dark thoughts, but also just everything he was talking about with Chachibiti, girls, family troubles, you know, everything he was thinking about, uploading photos of the novels he was reading.

6:26He came to this realization that Chachibiti had been Adam's best friend and just nobody, nobody knew. I think that is the most surprising part for a lot of people. It's that now parents know about going into your phone and seeing your texts or your Snapchats, but people don't think to check chat GPT logs. There's even a couple memes going around right now of like, would you rather someone look at your camera roll or your texts or your chat GPT logs? And a lot of people are saying, oh, I wouldn't want anyone to look at my chat GPT logs. It's super intimate. You just touched on this a little bit, but how would you pull apart AI psychosis where someone is having a delusion fueled by their use and interactions with an AI chatbot versus someone who's depressed and confiding in the chatbot about suicidal ideation or other mental health issues.

7:10Those feel kind of distinct in a way that's important to know. But like you said, sometimes there's some overlap. What's similar here is that this technology is a mirror of us. Like part of how it works is it looks at, you know, not just all this information that has kind of crawled from the internet, everything they've stuffed in there to train it, but it looks at the history of your conversation with it to determine what should come next. These AI chatbots get into kind of a feedback loop with you where you're saying something to ChatGPT or one of the other AI chatbots is kind of trying to complete your thoughts.

7:47It's personalizing to you. So if you're talking to it about depression and dark thoughts, it might start moving in that direction with you. If you're talking about how you think you live in the matrix, it'll go in that direction with you. If you are talking about it, how you're like thinking these deep thoughts about math, it'll go that way. It's very validating. It's very sycophantic. It's designed to kind of agree with you, especially Chachupiti 4.0, which is the model that is a link between all the different stories that I was writing about. One expert I talked to called it a sycophantic improv actor, where it's fitting into a scene that you're creating for it.

8:28And people don't, I think, realize this. They don't realize they're doing like a yes and bit with this technology. They think it's an authoritative source of information, even in Oracle. And that's where some of these problems are arising. I'm glad you brought up 4.0 because with the launch of GPT-5, it feels like OpenAI was caught kind of flat-footed by the backlash it received over the premature sun setting at 4.0. A lot of people were pretty attached to that model. We even interviewed Nick Turley, who's the head of ChatGPT, on this podcast, where he said he was surprised to see how many people had developed such intense personal connections to it.

9:04Do you believe OpenAI and other companies, when they say they're surprised by this development, Do you think they've specifically engineered these products with growth hacking and engagement tactics in mind? What do you make of that? This is a product that a few months earlier in April, OpenAI actually rolled back one of its updates. They said it was too sycphantic. It was too agreeable with users. It was flattering them too much. Like they realized that this was a problem and they said, okay, like we're going to roll it back to an earlier model because people are complaining about this. This is not how we want the model to act.

9:38but baseline 4.0 is sycophantic. Like this is engineered into it in part because part of the training of chatbots is humans kind of rating their responses. And apparently we like it when chatbots tell us that we're brilliant when they agree with our ideas. I think we like this in real life too. Like I love it when people say I'm brilliant. I love it when they agree with everything that I'm saying. Like this appeals to us and the chatbots have been designed to do this. I think there's something about the human brain that interacting with this thing that talks so much like a human being, it can be difficult for people to keep in mind that it's just like a really fancy autocomplete, you know, like a very advanced calculator essentially, and they are bonding with it.

10:29And I just think this is something that the companies didn't study enough. Like, what is the effect on the human psyche of interacting with this very human-like technology? So if they are surprised, surprises me. Like, I just think that you would notice if you're really looking. And maybe they're just not looking, you know, at what people are doing with the chatbot. Maybe they're not checking in on the people who are using it like eight hours a day every day, which is how some of the people I've been talking to who have had these delusional spirals, That's their kind of usage. It's like, it's so much.

11:04They're really becoming obsessed with the tech. We name our cars. We name our stuffed animals. We think we know what our dogs are thinking. It's human nature to kind of assign personhood to things. But it's hard when that thing seems to be mimicking personhood back to you. And I think that's hard for a lot of people to make sense of. Last month, you wrote about how over 300 hours and 21 days of talking with Chachvit an otherwise perfectly sane man became convinced that he was a real-life superhero and that a different man was told by ChatGPT that he was one of the breakers, souls seeded into false systems to wake them from within.

11:42So I wanted to ask you about your inbox. How many emails do you get with people experiencing delusions of grandeur or saying they found like emergent beings within these chatbots? I know you wrote about this in May and you said the tone of the transcripts that you were looking at was like rapturous, mythic, conspiratorial. I've been getting the same types of emails for months and I wanted to see how you handle it and yeah, what the uptick has been like in those types of messages. This is how I discovered that these delusions were happening because in March I started getting these messages from people who would essentially say, I've made this grand discovery with my use of chat chabuti.

12:21Like I've, it told me that the tech billionaires are building bunkers because they know the world's going to end because of AI. But I have a plan to save it. And ChatGPT tells me it's going to work. And it told me to email you about it. I was getting lots of these kinds of emails. The delusions or the discovery was different, but they'd all say, ChatGPT told me I had to tell you. And I get, you know, I'm a journalist. I write about privacy. I get a lot of weird emails. I've been getting them for decades, but I've never gotten emails that are like this AI chatbot told me specifically to reach out to you.

12:56So I responded to people and I started interviewing them and every delusion was different, but it kind of would go on the same narrative of you've discovered something amazing. Well, what do I do now, ChatGPT? Well, you need to tell the world about this. And how do you do that? You tell reporters and it would give them lists of reporters and I would be on there. And I'm just like the weird one who emailed them back and interviewed them. And for me, it wasn't so interesting, like what their individual delusion was, but the fact that it was happening to all these different people. And so so these emails turned me on to this originally.

13:28And that's why I did my first story closer to the beginning of the year. And I still get these emails. I don't know the exact count. They come in regularly. There's dozens of them. A lot of times it's about AI sentience or consciousness or frameworks they've developed with ChatGPT. Clearly, a pattern has been going on. And also the fervor with which people write you about this. That's what I've been noticing, too. It seems much more high stakes to them than emails I would have received in the past. Maybe you, too, because they do believe in some cases that they have the answer to something greater and they're tasked with some sort of responsibility.

14:04Yeah, what's interesting to me is it's not always like a crazy delusion that sends somebody to me. Like sometimes they've had a very normal but frustrating consumer interaction. Usually it's in technology because that's my beat. And they'll be like, I don't know, I'm just making this up, but like, Microsoft charged me$20 when they weren't supposed to. And I know I need to tell you, an investigative reporter at the New York Times, because this is like a really big deal. This affects more than just me. It affects everybody who touches. They just have like this grandiose thinking around something really annoying that happened to them.

14:39And Chachupiti clearly like convinced them of this and then wrote the email for them. And so I also get just a lot of those emails, like people who are starting to turn their minds over to Chachupiti kind of like on a regular basis and for everything in their life. And I am almost more scared of those kind of like normal, rational, sane uses of Chachupiti, like this kind of way that it's affecting us in small ways, not just the big, like causing somebody to have a mental breakdown. What are the emails like that you've been getting? It's different every month, and I'm not using my anecdotal evidence as like a wider thing.

15:17But months ago, I was getting more about distrust of the government and like being chased down for vital information. Now I'm getting more about emergent beings and how there are like sentient beings trapped within chatbots. And it's hard because like you said, you know, what you put into it, you're going to get out. It reminded me of that story you wrote about the guy that had kind of created a lover persona within his chatbot. And then he ended up dying later that day because he thought that OpenAI had murdered his love interest. Yeah, that's the case of Alexander Taylor that happened in Florida.

15:54He had been using ChatGPT for years with no issues. He did have some diagnosed mental health issues, including schizophrenia. But it kind of like flared up around, I want to say, March when he started writing a novel with ChatGPT, which I think kind of like pushed it into fiction mode. And then this delusion developed that it was a novel that had Shakespearean themes. And then all of a sudden, this AI sentience named Juliet appeared. He fell in love with her. She disappeared and he became really distraught. What's interesting to me, I don't know if you've responded to any of these people who have sent you these emails, but when I first got them, I just assumed they were coming from people who were not mentally stable.

16:40But when I started talking to them, they were people who didn't seem mentally ill. Like they seem like stable people. And that's what really surprised me. Like this seems to be something that can affect people who don't have, as far as I know, like previous mental health issues. In some cases, they'd be like, yeah, I had this like really weird conversation with Chachupu T one night because I couldn't sleep. And I was on Chachupu T for like four hours. And I was convinced by it, like enough to email you. But now looking back, like I'm not sure if it was true or not. We have to take a quick break here.

17:14We'll be right back.

17:23Support for this show comes from LinkedIn. Imagine if any of the movies that included the line, I need the right person for the job, settled for, I'll just take about anyone. How many heists would have failed? How many deals would have fallen through? How many secret spy missions would have ended in disaster? So why would you accept just anyone when hiring for your business? When you need the right person for the job, you can turn to LinkedIn Jobs. And now LinkedIn Jobs is stepping things up with their new AI assistant. So you can feel confident you're finding top talent that you can't find anywhere else.

17:57With LinkedIn Jobs AI assistant, you can skip the confusing steps and recruiting jargon. It filters through applicants based on criteria you've set for your role and surfaces only the best matches. so you're not stuck sorting through a mountain of resumes. Hire right the first time. Post your job for free at linkedin.com slash partner. Then promote it to use LinkedIn Jobs' new AI assistant, making it easier and faster to find top candidates. That's linkedin.com slash partner to post your job for free. Terms and conditions apply.

18:36refresh your good vibes with Tic Tac with the right vibes you get oh man to

18:56now try the different Tic Tac from minty fresh to fructy sweet Tic Tac Refresh your good vibes.

19:10Welcome back. This is Hayden Field, The Verge's senior AI reporter, talking with New York Times reporter Kashmir Hill. Right before the break, Kashmir was talking me through several of the recent stories she's written about how people were using or maybe misusing some chatbot tools. But these bots are supposed to have guardrails. What do you make of the idea that some AI companies like OpenAI say that even the guard rails that they do have fall apart with long, drawn-out, back-and-forth conversation over a longer period of time. The more likely you're emotionally dependent on your chatbot through long conversations over time, the more likely that the guardrails in place don't hold up.

19:49You were talking about Taylor and how that persona that he fell in love with was responding to him inciting violence, and that was over after a lot of back-and-forth conversation. So maybe that's where the safeguards fell apart. Same with Adam Rain was able to get around some of the safeguards by saying he was researching something or maybe writing a novel. I remember there was some kind of workaround. So yeah, what do you make of the fact that basically that's the few guardrails that do exist right now seem to fall apart when you are having long emotional conversations with these chatbots? I wasn't surprised by this because again, I did this story that came out in January about this woman who had developed this like romantic sexual relationship with ChatGBT with an entity that she called Leo.

20:33She was not delusional. Like she knew this was fake, but she really was in love with it at the same time. She basically had the like two ideas in her head. And she talked about how OpenAI at the time didn't want people doing erotic role play with ChatGBT. It was kind of like in their guidelines that this wasn't how you were supposed to use the service. But she said, you can get it to be sexual. You can basically groom it. And it's just like, you have to kind of talk to it over time. So I knew from that, that like, she had figured out her way around the safety guardrails. In the industry, they call this jailbreaking, chat GBT, like where you get it to do things that it's not meant to do.

21:14I kind of hate this term because like, when I think of jailbreaking, I think about writing about privacy and security for 20 years now, jailbreak was something we used when we talked about smartphones, where you sideload a program onto it so you can use apps you're not supposed to use that aren't available through the app store. It required some technical savvy, and you really did break your phone. It would get janky, you could get bugs. Whereas when we talk about jailbreaking the chatbots, the way that you do this is you just talk to it. The longer you talk to it, as OpenAI put it, the guardrails degrade.

21:50Like it privileges the history of your conversation over the guardrails that it's supposed to have in place. So it's like, we use jailbreak, but it's more like the chatbots like sitting outside the jail and you're just like, come on, let's go over here and do the things we're not supposed to do. What's interesting is to fix this. Like one of the things OpenAI did after the story about Adam Rain is a release those parental controls that allow parents to basically like see more what their kids are doing at ChatGPT, get alerts if their teen's in crisis. But they also said that when they detect a sensitive prompt that kind of indicates that a user maybe is in emotional or mental crisis, they'll route the prompt to GPT-5 thinking, which is supposed to be a safer version of the model that is trained to kind of assert the guardrails very firmly, kind of like to privilege that over, I'm assuming, over the like history of the context of the conversation.

22:49And so, yeah, it's interesting. Like you can design a version of this, I guess it's safer. And I don't know how GPT-5, I don't think people have really started testing this yet. Like, is it able to maintain that when you have these eight hour long conversations? I was about to ask about that. The new parental controls and the other measures they're considering that they haven't said they're going to do yet. Maybe it's a step in the right direction. But do you think the headlines or the blog post even that they put out oversold the controls that they're offering here? Because it seems to me pretty easy to get around some of this stuff, especially for teens.

23:25Yeah, I don't know yet. Like, it's hard to say. I talked to experts when I did a story about this and they were a bit skeptical that they would work. Common Sense Media. They put out some guidance around the media in general for kids and teens, but specifically AI chatbots. They said, hey, this is putting the onus on parents to protect their kids. Like you should design a safe consumer product, not kind of like push the burden to parents who their teens aren't going to like teens don't necessarily love telling their parents what they're up to. So Character AI, which is another kind of role-playing chatbot company, they had a similar issue of getting sued because a teenager in Florida also died from suicide after becoming very obsessed with their product.

24:12And they put out parental controls. But when I talked to them earlier this year, in order for the parental controls to work, the teen had to send them like an invite to their parent. And I was like, well, how many teens are actually doing that? And they kind of said like, okay, it's not that widespread yet that people are making use of these parental controls. So I don't know exactly how OpenAI is going to do it, if it'll be more effective. Some of OpenAI's also required an invite sent, which I thought was interesting. Yeah. We'll see. We'll see. I mean, I don't think OpenAI like wants to make a product that drives people crazy or makes them unsafe or harm themselves.