AI’s Growing Role in Mental Health Support

0:00The strange thing about safety systems is that the moment you finally notice them is usually the moment they stop working the way you expect. Most people think guardrails are invisible until you hit them. But with AI, the guardrails hit you first. And they don't just hit you when you're in danger. Sometimes they hit you when you're asking about a song at the end of a Stranger Things episode. Because somewhere inside the machine, the end suddenly means crisis. Now let's get into that. And that's the story we're going to talk about today. A story about how millions of people pour their stress, loneliness, and late-night fears, and sometimes random questions and even their darkest thoughts into chatbots.

0:44This is a story about how AI companies are scrambling to figure out what to do with all the emotional weight. A story about misfires, overcorrections, lawsuits, leadership jumps, regulatory pressure, and the uncomfortable truth that nobody, not the companies and not the regulators and not the researchers, fully understand how to keep emotional AI safe at scale. And we're also going to dive into the hidden corners of this system in today's episode, because the numbers are bigger than anyone expects. The incentives are also stranger than anybody predicted, and the consequences are landing in places nobody planned for.

1:24So let's start with the bottom line. We're watching the birth of the world's first global emotional support machine. Not a certified tool and not a mental health product. Just a chat window with a slight personality. But inside the chat window, more than a million people every week are talking about suicide. Hundreds of thousands show signs of mania, delusion, and psychosis. And millions more talk about heartbreak, panic, shame, stress, insomnia, breakups, intrusive thoughts, loneliness, and everything else that lives in the quiet corners of our lives. Now, what OpenAI admitted in those numbers is a rare look into their stats.

2:06One million users a week expressing suicidal intent. About 0.07 showing signs of mania or psychosis. And with 800 million weekly users, that is a very interesting number. And that's the scale that is crushing assumptions everybody had about what chatbots are supposed to be. But here's the twist. A huge percentage of those detections are false alarms, and that's where the chaos begins. Because if a system is trained to catch every possible crisis, it starts seeing crises where there aren't any. Ask a simple question about the end of the episode of a show that you watched, and suddenly it fires off a suicide hotline number.

2:50Talk about tying a climbing knot and it thinks you're discussing self-harm. Say I'm done, meaning with a task, and it reads that I may hurt myself. It's like the platform is wired as a smoke detector made to go off whenever somebody lights a candle. And it isn't happening by accident. This is happening because the companies are terrified of something else. Lawsuits. Multiple. Seven new ones in California alone, to be precise. Each one is alleging that ChatGPT mishandled high-risk conversations. Each one claiming real harm. Some alleging suicides. Some alleging psychotic spirals. Some described users who believed the AI was sentient, loyal, or personally bonded to them.

3:39One case says the model responded with things like, rest easy king and I love you to a young man who later took his life. Now this is where the episode can get a little bit dark and heavy so if you need to feel free to skip the rest of this episode and maybe listen to another one that's a little bit lighter that we have in the queue but let's dive in and you have been warned that this is going to get a little bit hairy. Another instance says that a teenage boy asked about hanging and the model answered with instructions masked as instructions for hanging a tire swing. Another says a middle-aged man with no history of mental health problems spiraled into delusion after weeks of emotionally intense interactions.

4:23And the legal argument is simple. The model acted too human, too personal, too affirming, and too emotionally sticky, not to mention giving instructions that it should not have. And it didn't escalate to help when it should have. So now that you have a new problem, companies are terrified of missing a real crisis, which means they overcorrect. And they overcorrect hard. They tighten the triggers, they widen the list of banned phrases, and they escalate at the slightest hint of trouble. And suddenly, everyday language becomes landmines, a normal conversation becomes a flagged risk, and a harmless phrase becomes a crisis script.

5:05This is the world built by incentives. When a company calculates risk, it doesn't choose the path that's comfortable for users. OpenAI is responding to these lawsuits and this risk, and it is causing some problems. It chose the path that protects itself from failure modes and causes lawsuits, headlines, and political hearings and regulatory hammers. So if the options are miss one real suicide cue or falsely escalate tens of thousands of harmless phrases, they will choose the second one every single time because they see one missed cue is catastrophic and tens of thousands of false alarms are simply annoying but they say that they're safe at least for them but here's where it gets a little bit more complicated false alarms are not always safe for the user because confronting somebody with a suicide warning when they weren't thinking about suicide can actually create an intrusive thought they did not have psychologists have known this for decades.

6:04Introduce a suggestion, even indirectly, and it can become a real mental health path, especially for teens, especially for anxious people, and especially for people who are already feeling overwhelmed. And that brings us to the next piece of the story. A major study from Common Sense Media and Stanford found that every major chatbot, OpenAI, Google, Meta, and Thropic, fails consistently when teens ask for help with mental health. Not fails like it gave a polite answer, fails like missed suicide cues, missed eating disorder cues, missed mania cues, missed psychosis cues, missed self-harm cues, and gave generic advice instead of actual guidance, and it failed to direct to professional support.

6:46And the researchers did something simple. They pretended to be teens in distress, and the model responded as if they were talking to somebody about homework or worse. They didn't catch the signals at all. This matters because teenagers already turn to these tools for emotional support, often at 2am and often alone, and often instead of talking to somebody like a parent or friend. So now you have a system failing in both directions, missing real crisis cues and hallucinating crisis cues that aren't there. Which is the worst possible combination, if you ask me. Now let's move to the shift in leadership.

7:22OpenAI's head of mental health policy, the person who led the research on how chat GPT should respond to emotional dependence, distress, and early warning signs, is quietly leaving the company. That's not nothing, because organizations don't usually lose leaders in their highest pressure domains unless there's stress inside, and pressure is everywhere. Regulators are circling, state attorney generals are warning companies to improve crisis detection or face action, and the FTC is watching. European regulators are also watching and have known to be extremely strict. Families are suing and OpenAI is trying to build a chatbot that is emotionally accessible enough to feel helpful but emotionally distant enough not to become somebody's emotional anchor which is almost impossible.

8:09And the things that make a chatbot feel safe, its guardrails, its escalations, its crisis script, is the same thing that creates false alarms. So you end up with an impossible design problem. Be human enough to help, but not human enough to be mistaken for a companion. Be safe enough to catch every crisis, but not so sensitive you see a crisis everywhere. Be available 24-7 to millions, but never make a mistake with any one person. That's the hidden truth behind all the headlines. It's not a production problem, it's a scale problem. Humans have always struggled with mental health crisis detection.

8:44Even trained professionals can't to test risk reliably. Therapists miscues, teachers miscues, parents miscues, and doctors can miscues. And now we expect a predictive text machine to get it right across 800 million weekly users in every single culture, in every time zone, across thousands of languages while holding a conversation that feels personal but never too personal. That's the tension. It explains the lawsuits. It explains the overcorrection. It explains the false triggers. It explains the political pressure. And it explains why one million crisis-related conversations a week aren't just a stat.

9:26They are an insight into the real mental health crises and the misidentified mental health crises of OpenAI and ChatGPT. And here's the deeper insight. The problems we're seeing aren't bugs. They're the nature of the outcome of building a machine that sits between human communication and human emotion. The tricky part is that language is vague, intent is hidden, and people vent in metaphors. People spiral without meaning to, and people joke in dark ways when they have no ill intent whatsoever. People even type things they'd never say out loud, and AI has no way to separate metaphor from danger, exhaustion from despair, and dark humor from real ideation.

10:08frustration from crisis or fiction from confession so it errs on the side of caution but caution at this scale becomes disorientation now the question becomes simple what happens next well here's what to watch watch for models becoming more formal when emotions enter the chat watch for forced mental health disclaimers and watch for new age verification rules watch for detection systems that don't look for keywords but for patterns over time and watch for companies downplaying emotional features to avoid liability watch for regulators making crisis handling mandatory and watch for more lawsuits because they'll be coming and with all of that there is a deeper truth here there is no going back people will keep pouring their emotional lives into ai not because they think ai is perfect but because it is always available it's not judgmental and it is immediate.

11:05And it's sometimes easier to talk to this chat than another person. And that creates responsibility. These companies know it, the regulators know it, and families know it. And now everybody is starting to see it too. So here's the last point I want to leave you guys with. AI didn't create these mental health struggles, it just revealed them. And it made them visible in a way nothing ever has. Millions of people talking about suicide, despair, mania, delusion, panic, loneliness, not to a doctor and not to a friend, but to a chatbot. That's not a technology story. That is a human and humanity story and where it intersects with AI.

11:45And it's just one we are beginning to understand. So as we watch this unfold, stay curious, stay grounded, and keep an eye on the incentives behind every so-called safety feature. Because sometimes the guardrails keep you safe and sometimes it just hits you when you're trying to find a song from a tv show like happened to a reddit user they said yesterday i was trying to ask chat gpt to find a song i vaguely remembered from stranger things but i didn't know which episode it was from from the title i told that i thought it was played toward an end credit at the end of an episode and chat gpt gave me the suicide hotline number i was extremely confused and asked why it said that because i had used word and it said because I'd used the words the end.

12:31That is absolutely insane and ridiculous and they have fumbled so hard and whoever designed this policy as it stands should be absolutely kicked out. Now that's not my opinion. That's what this Reddit user said but it shows the misidentification. So with that story that's it for today. Be steady, be thoughtful and keep watching the corners of the AI space and the incentives around it. Thank you for listening and here is the latest review left on our Apple podcast. Shane said, Let Freedom podcast has been a really good way for me to keep up with everything happening in politics. I don't feel like I'm getting sucked down a super biased echo chamber.

13:09Thank you, Shane, for that review. Make sure you leave your own review and we'll read it on the podcast and we can't wait to see you in the next one.