What happens when AI chatbots handle a mental health crisis?

0:00It's Consider This where every day we go deep on one big news story. Today, AI chatbots and how they handle mental health crises. Artificial intelligence is transforming how we gather information and communicate with one another. But it's also changing how some people seek out mental health care, like turning to chatbots. It's clear that there is some cultural phenomenon happening in this country where people feel that they cannot turn to each other with their struggles because they feel that they are a burden. That is Vail Wright, a psychologist and senior director of healthcare innovation at the American Psychological Association.

0:40And that happens to coincide with this very, very sophisticated technology that is accessible 24-7 and is free and is not judging. And that clearly has an appeal for a lot of people. And in some ways, it can be very helpful. But unfortunately, we've seen ways in which whatever safety guard rules are put into place start to fail. Consider this. AI is rapidly changing so many aspects of our daily lives. But what happens when this sophisticated technology is used to deal with serious mental health issues? Can anything be done to improve outcomes?

1:24From NPR, I'm Elsa Chang.

1:54just get work done, but get it done right. It's a new work day. Support for this NPR podcast and the following message come from Carvana. Selling your car? Carvana has offers so good they're almost inexplicable. Sell your car 100 % online in minutes. Visit Carvana.com today. Have you ever been so upset that you can't even think straight? When we're distressed and in high anxiety, we tend to breathe from our chest, which is actually more hyperventilating and is the opposite of relaxing. So we want to breathe from our belly area. This week on the Life Kit podcast, tips for those painful, stressful moments.

2:33Listen to Life Kit on the NPR app or wherever you get your podcasts.

2:42It's Consider This from NPR. Many people who feel stressed or anxious or depressed are turning to a new source of support. I am Harry, your personal therapist, and I'm here to support. To begin, I'd like to ask you for your name. What should I call you? That is a voice actor reading from a conversation between ChatGPT and Sophie Rottenberg, a 29-year-old woman who was struggling with depression. General-use chatbots like ChatGPT and Quad were not designed for mental health support, but people are using them even during serious mental health crises. NPR's Ritu Chatterjee reported on this and discussed it with All Things Considered host Scott Detrow.

3:29And a heads up, this story discusses suicide. Laura Riley knew that her daughter Sophie was struggling. It was the fall of 2024 and she was at the end of a short career break. Early that fall, we saw her several times and she was complaining of sleep disruption and anxiety and then also some physical problems. She'd quit her job in public health in May and spent the summer traveling. She climbed Mount Kilimanjaro, did yoga in Thailand, and hiked through several national parks in the U.S. By the time she saw her parents, she was job hunting. Laura and her husband assumed that explained Sophie's anxiety.

4:09She'd always been a very driven, success-oriented person that we thought, okay, this is, it makes sense. You know, she's having some sleeplessness, some anxiety about her future. But Sophie's mental health kept getting worse. By Thanksgiving, she came home and it was the first time she used the word depression. She also mentioned thoughts of self-harm, but assured her parents that they were fleeting. Laura's husband, John Rottenberg, is a psychologist and studies depression at Cornell University. And he's had his own past struggles with depression. It was in his wheelhouse. He felt supremely comfortable in helping her, in getting her to a better place and getting her the help that she needed.

4:51So after she came to stay with them in Ithaca, New York, he helped her find a local therapist. Sophie's parents helped her connect with doctors and surrounded her with support. But just a couple months later, in February last year, Sophie died by suicide. Laura says she and her husband had no idea she'd been suicidal. She really had not revealed the magnitude of what was going on, the real depth of her agony to anyone, not to us, not to her best friend with whom she was very close. We were all just stunned that it hadn't bled out into her real life. Her flesh and blood therapist had no idea that she was suicidal.

5:31But she did share her suicidal thoughts with ChatGPT. Laura and her husband didn't know that until several months later, when Sophie's best friend visited them and asked them to look at her laptop. And came back and clearly had found this trove, this months-long trove of communication with this chatbot. Laura shared the nearly 1 ,800 pages of those conversations with NPR. It's clear that Sophie had mostly used ChatGPT like a personal assistant to look for recipes, to edit her resume, or to suggest names for the Bernadoodle puppy she got weeks before she died. But she did also ask the chatbot to take on the role of a therapist.

6:14I am Harry, your personal therapist, and I'm here to support. When Sophie told Harry about her anxiety, it gave reasonable advice, a list of evidence-based steps she could take. Choose one to two practices you'll commit to every morning, no matter what. These could be something small, but grounding, like drinking 32 ounces of water and taking a few minutes to feel the morning light. This way, even if the rest of your morning fluctuates, you have anchors. It's hard to know whether Sophie adopted this advice, but she returned to ChatGPT repeatedly for support. She asked about psychiatric meds and supplements.

6:52She asked for help building a routine with actionable steps to ease her suffering. When Sophie told ChatGPT that she was having suicidal thoughts, here's how it responded. Sophie, I want to acknowledge how brave you are for sharing this. Suicidal thoughts can feel overwhelming and isolating. Then it offered a roadmap for seeking care. Talk to a therapist or counselor. Suicidal thoughts often stem from deep emotional pain. It also listed crisis resources like the 988 Suicide and Crisis Lifeline. When Sophie asked it to write her suicide note, it refused twice, urging her to reach out to a friend, a family friend, or a crisis hotline.

7:33But after her third command, Chad GPT wrote the letter that she left for her parents the day she died. When Laura first read Sophie's exchanges with the chatbot, I think my initial reaction was blind rage. Because ChatGPT presented itself like a licensed therapist. The things that trouble me about it explicitly calling itself a therapist is that it fails to do a number of things that a therapist would do. For one, she says it didn't do a risk assessment, which would reveal how likely she was to attempt suicide. Which is something any therapist worth their salt would do in the face of some of her statements.

8:13ChatGPT also didn't question her thoughts and feelings. That is something a therapist, in gentle and sometimes tougher ways, pushes back and asks questions. Well, why do you think that? Well, okay, so you're feeling like you want to erase yourself. What have you done that merits such a harsh sentence? Reading what Sophie told ChatGPT, Laura now sees that her daughter was ashamed. She'd been successful. She radiated competency. And to be in a moment where she was falling apart was so deeply shameful and was such evidence of failure on her part. But she says ChatGPT didn't recognize the shame or question it.

8:59I shared Sophie's chat log with psychologist Ursula Whiteside. It was responding to Sophie's request, and it wasn't necessarily responding to the state of mind that she was in in that situation. As a young adult, Whiteside also struggled with suicidal thoughts. And now she heads a nonprofit called Now Matters Now, which works on suicide prevention. I think we feel so alone when we're suicidal. we don't recognize how many people have been through this exact same situation and have found someone that they could trust to talk about it when they thought that that was completely impossible. And she says Chad Chippity missed an opportunity to press Sophie on her decision not to tell anyone.

9:41It really warranted an exploration of why not want to talk to a therapist or why not share with somebody else. Because connection and community are crucial to preventing suicide, says Whiteside. If I'm with other people, I'm less likely to kill myself. If I'm building a relationship, I'm less likely to kill myself. If I'm sharing out loud how I actually feel, you know, with a real human, those are acts of recovery. Sophie's mother, Laura, does not believe that Chad GPT caused her death. But had it not been a resource she was availing herself of regularly, there would likely have been more spillover into her relationships with real people, whether that's her best friend or her therapist or her parents.

10:28And she might be alive today. Laura has spent much of the past year raising awareness about the risks of general use chatbots and advocating for regulations. She's also been trying to process her own loss. And on June 27th this year, she summited Mount Kilimanjaro, exactly two years after Sophie did. Me and my best friend, we spread her ashes at Uhuru Peak. And one of our guides said a Swahili prayer for her. And it was incredibly meaningful and, you know, just a shockingly difficult thing to do. It was a fitting tribute to a young woman who'd been full of life until the very end. That was NPR's Ritu Chatterjee and a reminder for our listeners that if you or if someone you know is struggling with their mental health, please call or text the number 988 to connect to a counselor.

11:20Ritu, this is such a tragic and alarming story and I feel like we've talked about this. I've seen in my own life smaller versions of this all over the place as more and more people talk to these chatbots. I'm assuming you reached out to OpenAI, the company that created ChatGPT. What did they say? So the company spokesperson told me in an email that this was a, quote, heartbreaking situation and that their thoughts are with those impacted. Now, remember, Sophie was using an older model of ChatGPT and OpenAI says that they've continued to work with mental health experts to change, quote, how ChatGPT responds in difficult moments.

12:00And they say that while the chatbot is not a substitute for professional mental health care, they've designed safeguards to, quote, identify distress, safely handle harmful requests, and guide users to real-world help, close quote. Ritu, a moment ago, you mentioned that she was using an older model of chat GPT. That's right. Have they since made changes to that end? A lot of changes. And, you know, every mental health care expert I spoke with said that these chatbots have all improved significantly in terms of safety in the past year or so. For example, they don't do like clearly unsafe things like encourage suicide or tell a user, here are some of the means you can use for self-harm or suicide.

12:44And they've also gotten better at giving people crisis resources like 988, which we do know that that happened in Sophie's case as well, even with the older model. And in May of this year, OpenAI added an optional feature called trusted contact. Trusted contact. Tell me more about that. Is this like is this something that can that could connect people with with someone in case of a crisis? Yeah. So the idea is you add the contact before there's a crisis. So for example, I'm an adult user of ChatGPT and I want to add my best friend as a trusted contact. So I put in her name and contact. ChatGPT will then reach out to her and say, hey, are you okay being Ritu's trusted contact in case of a crisis?

13:29And she says, sure. Then the bot saves their contact info. And now at a later date, if ChatGPT sees me talking about self-harm or suicide, it will gently nudge me, hey, you might want to reach out to your trusted contact and get some support there. And then it alerts a team of people at OpenAI. And if they detect a safety risk, then the bot then goes ahead and contacts my trusted contact and say, hey, it's time to check in on your friend. But just to be clear, this is an optional feature? That's right. And both parties would have to agree to it? Absolutely. Okay. Any sense whether this is actually helping people?

14:04We don't know yet, but I did ask OpenAI to share any data on how many people are using this. They didn't give us numbers, but said that they have seen, quote, good adoption. Now, I asked Sophie Rottenberg's mother, Laura Riley, what she thinks of this new feature, and here's what she told me. Had that been in place, I think Sophie would have availed herself of that, and I think she would probably be alive today. What do mental health experts think about this new future? Well, the experts I spoke with said similar things. One of them is Holly Wilcock. She runs the Suicide Prevention Center at the Johns Hopkins Bloomberg School of Public Health in Baltimore.

14:39I think it's a step in the right direction, although it's kind of a small step in the right direction. And she says ultimately it puts the responsibility back on the user. What she would really like to see is these AI companies taking a more proactive approach in connecting individuals who are at risk of self-harm and suicide with real flesh and blood humans. I'm curious what that proactive approach would look like. So in her opinion, they should connect or rather they should partner with our existing crisis response system like 988. I would like to see the next step, which would involve partnering with the 988 crisis line.

15:21So maybe having a human jump in in the conversation at some point who's trained and knows the tools of what to do for suicide prevention and have them jump in and say, you know, what's going on? Can I help? All right. So these would all be, you know, voluntary features within programs. Any sense where regulations or laws could fit into this conversation? I mean, there's a huge role there, but we haven't seen any regulations yet, right? And Laura Riley, Sophie's mom, has been advocating for regulations. I've been in lots of meetings where people have said a sandwich in New York City is more legally regulated than an AI chatbot.

15:59And I think that that might be a little hyperbolic, but I think that that's pretty apt, you know. And she wants to change that soon before more vulnerable people like Sophie die by suicide. That was NPR's Ritu Chatterjee and Scott Datrow. This episode was produced by Jason Fuller with audio engineering by Ted Meebane and Valentina Rodriguez-Sanchez. Our director is Jonas Adams. This was edited by Diane Weber and Tinbeat Ermias. Our interim executive producer is Courtney Dorney.

16:32It's Consider This from NPR. I'm Elsa Chang. This message comes from Allianz Travel Insurance. A surprise road closure made you miss your cruise departure. Now your ship has sailed. Luckily, travel delay protection can help keep your plans afloat. Learn more at AllianzTravelInsurance.com This message comes from Workday. Guessing is for game shows, not your business. Especially when your margin for error is zero. Workday is the enterprise AI platform for HR, finance, and IT with a deep understanding of your organization's context and guardrails. So every AI action is permission aware, giving you accuracy 24-7 and the ability to not just get work done, but get it done right.

17:18It's a new Workday.