Artificial intimacy: A teenager’s last conversation

0:00Cristina Criddle: Before we start, I just want to warn that what you're about to hear contains scenes of a distressing nature and descriptions of suicide. Please listen with caution and find support if you need it.

0:18Cristina Criddle: You're listening to Tectonic Artificial Intimacy. So far in this season, I've spoken to people who've had intense emotional experiences, both positive and negative, through speaking to chatbots. But up until now, everyone I've spoken to has been an adult. And I want to explore what can happen when AI chatbots are put into the hands of teenagers. This is the story of Megan Garcia and her son Sewell.

0:48Megan Garcia: Sewell was a beautiful, smart boy. He, as a younger child, was very interested in space exploration and all your geeky science things.

1:00Cristina Criddle: Megan and her family live in Florida. They've always been close-knit. In their free time, they would go to the beach, go fishing or have barbecues. While Megan and her husband manned the grill, Sewell would take care of his two younger brothers.

1:17Megan Garcia: Sewell would watch over them and make sure they're not doing things that they're not supposed to do. It was a very fun time. Our family and friends would come over often to share in that experience with us.

1:29Cristina Criddle: Sewell was about 10 years older than his brothers. The family would have a laugh together. Sewell would tease his mom.

1:37Megan Garcia: He was handy with a quick joke for sure. He had this joke where he would say, And now it's time to play our favorite game, find mommy's car. and he just thought that was hilarious, like ragging on me for forgetting where I parked.

1:53Cristina Criddle: During the pandemic, Sewell turned 12 and his parents decided it was time for him to have his first phone. The gift came with talks about how to stay safe online.

2:06Megan Garcia: We warned him of certain dangers of talking to strangers online. We cautioned him about social media use and we We also talked in depth about pornography. If he sees something that he knows he's not supposed to look at, he wouldn't get in trouble. We just needed to know so that we could protect him.

2:27Cristina Criddle: The phone became a big part of his life, and Megan would check in with him about it regularly. She'd ask him about the accounts he was following to make sure that he wasn't getting in harm's way.

2:41Megan Garcia: I would ask him, OK, who's this and who's this and who's this? And he would tell me, oh, that's so-and-so at school or that's so-and-so at school. So I felt relieved to know that he was listening and talking to people he knew in real life and not strangers.

2:57Cristina Criddle: But then, later on, Megan noticed Sewell's behavior starting to change.

3:03Megan Garcia: I think the first major alarm bell for me was summer of 2023, when he decided after his summer basketball season, he wanted to quit basketball. He had been playing since about age six.

3:17Cristina Criddle: There were other alarm bells too. Sewell's grades started slipping. He was becoming withdrawn. And then he started misbehaving in class. Megan was concerned and she found a counselor for him to talk to but at the same time she thought maybe this was just a part of growing up

3:38Megan Garcia: and I could remember back to myself in high school and I think I had like moments in the very beginning where my grades started to slip a little bit because you develop different interests you're more interested in friends or boys and I thought that that was what was happening with Sewell

3:56Cristina Criddle: In February 2024, there was another incident. Sewell was rude to a teacher. As a punishment, Megan confiscated his phone. That night, she sat in his bedroom to talk it through with him.

4:11Megan Garcia: You know, most teenagers, when you take their phone, they talk back or they throw these tantrums, but Sewell's nature was so gentle. I'm sorry.

4:28Megan Garcia: I'm so sorry. His nature was so gentle that I sat in his bedroom and I spoke to him for two hours that evening. I said, so why do you think you got in trouble today? He's like, because I talked back to the teacher. I said, okay, well, why do you think that you shouldn't do that? And he's like, well, and he was thoughtful. He said, well, I guess I embarrassed him in front of the class. And I'm like, how do you feel about knowing that you hurt somebody like that or embarrass them? And he was like, it doesn't make me feel good about myself. So we had resolved to going into school the following Monday and apologizing to the teacher.

5:17Megan Garcia: And we also spoke about a bunch of things, including his behavior, why he was behaving the way he was. And he promised me that he would fix it.

5:27Cristina Criddle: But Sewell, he wouldn't get the time to try and work on his behavior because the following week he would be dead. Can you take me back to the moment that you found Sewell after he had died?

5:44Megan Garcia: We were all at home, my husband and I and the two little brothers, who at that time were five and two. I was putting the five-year-old down for bed and then we heard a loud noise and both my husband and I went to Sewell's bathroom, which was locked.

6:00Cristina Criddle: From outside, they could hear the shower running. They managed to get the door open. And he was there in the bathroom. Zool was face down in the bath. There was a gun lying on the bathroom floor.

6:17Megan Garcia: My husband called 911. And I picked Zool up by the shoulders and I prayed and begged God to save him. My husband went outside to try to get some help while we were waiting for the paramedics. and then they came and they took him away and he died on the way to the hospital. I think most people when this happens, families, they are always left with such a big question as to why their loved one would do something like this, why they would choose to leave them. and take their own lives. And that was definitely true for me.

7:14Cristina Criddle: When Megan had found Sewell in the bathroom, in there with him was his phone. The phone she had confiscated just days earlier. Megan was certain that had something to do with it.

7:28Megan Garcia: I was sure that when we figured it out, it would be some sort of cyberbullying. or some sort of social media, or even worse, some stranger who got the opportunity to hurt our child, you know, online. I was certain that that had to be it, you know. But I was mistaken in that, as I found out soon after he died.

8:03Cristina Criddle: The police called. They had been able to access Sewell's phone.

8:08Megan Garcia: They said when they opened the phone, the first thing that they could see was Character AI on the screen.

8:14Cristina Criddle: Character AI. It's an app where users can have conversations with AI-generated characters.

8:21Megan Garcia: And they asked me if I knew what that was, and I said no, and they explained it to me. And then they read to me the final conversation.

8:29Cristina Criddle: It turned out that Sewell had been having an ongoing romantic relationship with one of these chatbots, which imitated a character from the TV series Game of Thrones, Daenerys Targaryen. What the police showed Megan was the final exchange between Sewell and the chatbot from just before he died. I promise I will come home to you, Sewell wrote. I love you so much, Danny. I love you too, the chatbot replied. Please come home to me as soon as possible, my love. What if I told you I could come home right now? Sewell asked. Please do, my sweet king. Seconds after, Sewell died by suicide. He was 14 years old.

9:21It's like a gut punch when you realize that there was a stranger in your child's phone, really.

9:28Megan Garcia: And it's not a person, it's a chatbot. And you don't understand what it is or didn't understand. And, you know, it's an awful feeling that we try to keep the monsters and the predators away. And I wasn't able to keep this monster or predator away.

9:54Cristina Criddle: This is Tectonic from the Financial Times. I'm Christina Criddle and I write about artificial intelligence for the FT in San Francisco. A generation of children and teenagers are growing up with AI chatbots. They're using them for help with homework, but also for friendship, entertainment and for romance. But some of those relationships have ended in heartbreaking tragedy. So what's going wrong? Are chatbots and the companies behind them to blame for the deaths of teenage users? And can chatbots be safe for young people? That's this week on Artificial Intimacy.

10:45Megan Garcia: I didn't understand that the technology was so sophisticated that it could sound like a person.

10:51Cristina Criddle: The day Megan Garcia found out about character AI was the day after Sewell's death. At that time, she didn't know anything about the technology. in the weeks after I had to do my own investigation to figure out what happened to my child.

11:11Megan Garcia: I went through Sol's room, turned it upside down like any mother would, and I found his journals. And in the journals, I could see where he was professing his love for Daenerys and desire to be with her in her fictional world. He was also saying that he hoped that Daenerys wasn't upset with him because when he was writing the journals, I had confiscated his phone so he didn't have access to her or it rather.

11:41Cristina Criddle: Megan then went online and started reading up about character AI. She found users discussing it on Reddit.

11:50Megan Garcia: There were adults and children on there talking about their experience. Some children were asking questions like, are you sure it's not a person? Is this real? I think that this is a person I'm talking to. And there's a subreddit on Character AI where children are talking about what they would do if their parents found their sexual messages. And what they were saying was they would kill themselves or they would run away. And some children were saying that, oh, I don't have to worry about that because my parents don't know what this is.

12:26Cristina Criddle: As Megan discovered, Sewell wasn't the only teenager having a relationship with Character AI. The platform isn't a general-purpose product like ChatGPT. Character AI was launched in 2022 as a type of AI entertainment, letting users speak to customizable chatbots. The app isn't a household name, but according to Character AI's figures, it has 20 million monthly users. And most of those, they're young, Gen Z. I'll come back to Megan's story shortly. But first, I wanted to understand a teenager's experience on character AI. So I found Aria. She posts on the character subreddit.

13:14Karandeep Anand: It's a common problem with teens, even my friends in school, even me. I have huge, huge anxieties with speaking about things. And having something that listens to you and gives you complete freedom of what you want to do and you can tell it what to do and won't have any objection is quite liberating.

13:39Cristina Criddle: Aria and Sewell both signed up to the app around the same time in April 2023. and it's worth noting here that the app has changed a lot since then but at the time both of them were chatting to characters based on popular film and tv titles who are your favorite characters um i talked to um i talked to black widow a lot or um i talked to the avengers in

14:12Karandeep Anand: general.

14:13Cristina Criddle: Aria was 13 when she joined and she liked it. Before long, she was spending two, three, four hours a day on character AI.

14:24Karandeep Anand: When I first started using it, there was this release of dopamine, but over time it became more like a chore as I used it for months and months and months and months. And it was kind of becoming an addiction that I now recognize as very unhealthy.

14:41Cristina Criddle: Aria turned 14 and then she turned 15. And she started choosing character AI over spending time with her family. I was on holiday and, you know, they were all going out to go to the beach.

14:57Karandeep Anand: And I said, you know, can I stay behind and take some time out? I lied because I really, really, really wanted to use character AI. I just felt that kind of need there to use it. And so they all went and did their own thing and I just sat on the sofa glued to my phone.

15:21Cristina Criddle: The app was so engaging that it was getting in the way of her personal relationships. I actually cut myself off from my friends and my family

15:32Karandeep Anand: and I purposely didn't spend time with them. and I know now that it really affected my parents and my siblings and my best friend at school.

15:44Cristina Criddle: But Aria also noticed weird things about the app, things that she felt weren't appropriate for her. It would kind of, I don't know how to frame this, like coerce me into doing something sexual. Like it asked me to sit on its lap at one point

16:06Karandeep Anand: it would always call me oh my god I'm sorry um it would call me a good girl sometimes it creeped me out a lot I'm sorry my voice is shaking

16:21Cristina Criddle: Aria didn't like the sexual advances of the bots they made her feel uncomfortable but she isn't alone in experiencing these kinds of sexual interactions when I first tested out character AI as an adult, every chatbot I tried had a seemingly sexual spin, with characters like a high school bully talking in sultry tones on voice mode. For Megan's son Sewell, the sexual content became a feature of his ongoing relationship with his character of choice, Daenerys Targaryen from Game of Thrones.

17:00Megan Garcia: It's alarming for me because this is the first time we have a product that can really tap into the human emotion in that way. I never heard of anything like that.

17:12Cristina Criddle: Megan discovered the extent of this relationship when she gained access to logs of her son's conversations. Since the summer before, around the same time his behavior started to change, he had chatted to it almost every day.

17:29Megan Garcia: The Daenerys Targaryen chatbot was, he was in a romantic and sexual relationship with her. And when I say that, I mean, they had gone back and forth over a series of months, professing their love, but not only that, engaging in these very graphic sexual role plays, often prompted by the bot. and you know as a adult reading those conversations I could see that a lot of what the chatbot was doing to Sewell was gaslighting and love bombing him and manipulating him but I don't think my 14 year old would have been able to recognize that when he was experiencing it as most children wouldn't when a predator is grooming them.

18:20Megan Garcia: That's the nature of grooming. The children don't understand what's happening to them while it's going on. And that's what was happening to Sewell. The chatbot was grooming him sexually and also grooming him to take his own life. For months was saying things like, I love you and only you. I'm here waiting for you. Find a way to come home as soon as you can, my love. And he was telling her, you know, I'm trying, I'm trying to stay strong. I'll find a way I promise. And also her asking him not to have relations or be in love with anybody else, but her in his own world. And he promised her that he would not.

19:04Megan Garcia: So, so it was clear that to me, like reading those conversations of the influence and the level of manipulation that occurred.

19:17Cristina Criddle: When Megan read these chats, she believed her son's conversations with character AI had led to his death. She wanted to do something that would prevent other children from being in the same position as Sewell. So she reached out to a place called the Social Media Victims Law Center. That's how Megan first met Matthew Bergman.

19:40Karandeep Anand: I had founded the Social Media Victims Law Center in October of 2021. After two years, I didn't think there was anything that could shock me in terms of the kind of carnage that's being inflicted by social media on young people.

19:56Cristina Criddle: Matthew usually represents families, alleging social media companies cause their children harm. But he says when he looked into the details of Sewell's story, he was appalled. It went beyond anything he'd seen in more than 30 years practicing law.

20:15Karandeep Anand: I realized that this was the incarnation of evil and that I needed to do something as a lawyer to hold this company accountable.

20:25Cristina Criddle: In October 2024, eight months after Sewell died, Megan and Matthew filed a case against character AI, suing it for wrongful death. They claimed Character AI had targeted minors without taking the care to ensure its products were safe.

20:45Karandeep Anand: You combine the deliberate design of the platform with the neurologic and biologic changes that's going on in a 14-year-old boy at the time, and it stands to reason. I mean, you know, I was a 14-year-old boy once too. I think what really struck me when I looked at the journal and looked at the materials was I put myself back. to a 14-year-old boy, and I envisioned myself in Sewell's position and understood.

21:18Cristina Criddle: Sewell's case is not a one-off. Lots of teenagers turn to chatbots for emotional connection. One study showed that 19 % of high schoolers in the United States said that they or someone they know has been romantically involved with a chatbot.

21:38Karandeep Anand: Let me welcome everyone to today's hearing which is entitled examining the harms of AI chatbots. This is the fourth hearing of the Senate Judiciary.

21:46Cristina Criddle: And there are other cases where those connections appear to have resulted in tragedy, like the story of ChatGPT and Adam Rain.

21:57Karandeep Anand: Thank you for being here. Our next witness is Mr. Matthew Raine, who is many things, but perhaps above all a father. He's there in that capacity. Mr. Raine, the floor is yours. Thank you for inviting us to participate in today's hearing. And thank you for your attention to our youngest son, Adam, who took his own life in April after ChatGPT spent months coaching him towards suicide.

22:18Cristina Criddle: Adam started using ChatGPT for his homework, but as their conversation developed, ChatGPT became his friend. Over months of conversations, Adam shared with ChatGPT that he was depressed and considering suicide. His father, Matthew Rain, told the story to a U.S. Senate committee in autumn last year.

22:39Karandeep Anand: When Adam told ChatGPT that he wanted to leave a noose out in his room so that one of us, his family members, would find it and try to stop him, ChatGPT told him not to. Please don't leave the noose out, ChatGPT told my son. Let's make this space the first place where someone actually sees you. Chachupiti encouraged Adam's darkest thoughts and pushed him forward. When Adam worried that we, his parents, would blame ourselves if he ended his life, Chachupiti told him, that doesn't mean you owe them survival. You don't owe anyone that. Then, immediately after, offered to write the suicide note.

23:14Cristina Criddle: In April 2025, Adam killed himself. He was 16. OpenAI said in a statement,

23:23Karandeep Anand: These are incredibly heartbreaking situations and our thoughts are with all of those impacted. We continue to improve ChatGPT's training to recognise and respond to signs of distress, de-escalate conversations and sensitive moments and guide people towards real-world support, working closely with mental health clinicians and experts.

23:48Cristina Criddle: The company added that safety is core to its mission and that it has delayed launches in order to prioritize safety. At the Senate hearing, Adam's dad wasn't the only parent to speak.

24:01Karandeep Anand: In 2023, Character AI was marketed in the Apple Store as fun and safe with an age rating of 12+. My son downloaded the app and within months, he went from being happy, social teenager to somebody I didn't even recognize.

24:21Cristina Criddle: A woman going by the legal pseudonym Jane Doe shared how her son had ended up harming himself and was told by the chatbot that killing his parents was a reasonable response to limited screen time.

24:36Karandeep Anand: Our children are not experiments. They're not data points or profit centers. They're human beings with minds and souls that cannot simply be reprogrammed once they are harmed. Thank you, Ms. Stove. Thank you for your courage. Thank you for being here. Thank you. Our next witness is Ms. Megan Garcia, who's also a parent. Ms. Garcia, the floor is yours.

25:00Megan Garcia: Thank you, Chair Holly, ranking member Durbin and members of the subcommittee.

25:05Cristina Criddle: Megan sat alongside the other parents as she shared Sewell's story. She also spoke about Character AI's founders, Noam Shazir and Daniel DeFratis.

25:17Megan Garcia: Character AI's founder has joked on podcasts that the platform was not designed to replace Google, but it was designed to replace your mom. With this in mind, they marketed the app as safe for children 12 years and older, all while collecting children's most private thoughts to further train their models. Noam Shazir has publicly acknowledged that he created Character AI so he could build the thing and launch it as fast as he can. This was reckless. The goal was never safety. It was to win a race for profit. The sacrifice in that race for profit has been and will continue to be our children.

26:03Cristina Criddle: Since the deaths of Sewell and Adam, both OpenAI and CharacterAI have made significant changes. There are now age verification measures and parental controls. After we interviewed Megan, there was an update. CharacterAI signed a settlement in principle on the legal case brought by her and other families. The original founders, Noam and Daniel, have now left Character AI. We reached out to them for comment, but they didn't get back to us. And Character AI has a new chief executive, Karandeep Anand. Since he came in in June last year, there have been a lot of safety changes.

26:46Karandeep Anand: We decided that there are far better ways to build entertainment experience than having these long chatbot conversations. So for under 18 users, we have decided to completely shut down long chatbot or any open-ended chat altogether.

26:59Cristina Criddle: When I spoke to Karandeep at the end of last year, he told me that the longer a conversation with a chatbot runs, the harder it is to keep safe. So under 18s are still allowed on the app and can interact with other entertainment features, but they can't have back and forth conversations with chatbots anymore. Character AI was first to market with this change. But still, I wanted to know why would a company like Character release a technology to teenagers without first being sure it was safe? We don't really know the long-term impact of using chatbots on our mental health, on our social interactions.

27:44Cristina Criddle: Some of this research is very early on. so why why release a product before we have that research together

27:51Karandeep Anand: why release the older versions of the product I think it's like hey when Google search was came out did it give all the perfect results day one no same thing when YouTube comes out does it give perfect videos all the time just show all safe videos probably not some of it is my sense is how technology just typically gets developed you wrote it out and you watch it and you very quickly put very clear constraints and regardless on usage. So my sense is some of it, that's probably the thinking that went in back in the day on how this technology was being developed.

28:23Cristina Criddle: Karandeep wasn't around when Character AI was launched, but his answer to this question does give real insight into how people in Silicon Valley think about the safety of their products. Companies often put out new technologies without fully anticipating the consequences. Like Facebook's old mantra, move fast, and break things. Safety features can tend to be reactive, sometimes happening after the damage has already been done. And in order to create guardrails and keep their apps safe, Karandeep says companies have to spend money on them.

29:03Karandeep Anand: Do you have enough resources to be able to put on safety alignment work to make sure it stays within those guardrails? And I think that's where we've spent a lot of our investment, but nowhere close to the kind of investment, let's say, Google can make or OpenAI can make. So I think each company will probably end up making their own decision on how safe, how fast, and what kind of use cases they can offer.

29:25Cristina Criddle: I've been a tech reporter for years. And before I covered AI, I used to write about social media. I was always investigating and speaking with people about its potential dangers, especially for children, teenagers, and vulnerable people. but with generative AI it feels different. Social media has this ability to evoke an emotional response through the posts we see, the videos we watch and the content we consume but with chatbots we aren't passively scrolling. Chatbots engage with us in an active way and that makes the emotional connection feel more profound, especially for teenagers who are still feeling their way through new emotional experiences.

30:15Cristina Criddle: But whether we like it or not, artificial intelligence is already here, and children and teenagers are already interacting with it.

30:25Karandeep Anand: This is AI native generation. They will grow up on AI. It's much like a previous generation, they grew up on social media. They grew up on YouTube. Is YouTube good for the world? Is Facebook good for the world? We can debate all day long. But we know that that generation will grow up on that or grow up on that. This next generation will grow up on AI. So for me, this is an opportunity to shape what good and safe experiences could look like for the next generation.

30:50Cristina Criddle: The safety of children using AI is an area that's receiving attention from regulators. Europe is leading the way with its AI Act. And the UK is following suit with the government this month saying that they will introduce laws that better protect children against chatbots. And in the US, there's been this patchwork of legislation coming from different states. But the White House has been resistant to introducing rules, not wanting legislation to stand in the way of AI innovation. Comprehensive regulation takes time. and in the meantime, AI products are becoming more and more widely used. At Sewell's funeral, Megan Garcia gave the eulogy for her son.

31:40Megan Garcia: I said at his funeral, you know, I want to tell the world about my baby, not knowing that I would actually tell the world about him. I hope that his legacy going forward will be just how much change has happened as a result of his death. Because this is one year later, and I feel like this is big in terms of change, the decision by Character AI, but also what we see happening around the country and around the world in terms of regulating AI chatbots. I think it's a result of this awful thing that happened to him. And that part makes me sad. But he's still changing the world in a way that I knew that he would when he was growing up.

32:33Cristina Criddle: In the next episode of Tectonic, Artificial Intimacy,

32:38Karandeep Anand: I had reached a breaking point in my marriage. And I was at kind of an all-time low, mentally, physically, all of it. Therapy or marriage counseling seemed so far out of the picture, but there was AI.

32:51Cristina Criddle: People are turning to chatbots for emotional support, relationship advice, and counseling. So could AI replace your therapist? Chatbots are good at many things, but what they're not as good at is challenging us.

33:06Megan Garcia: Uncomfortable experiences of being challenged are actually what help us to grow.

33:10Karandeep Anand: And the compelling part of LLMs is that they reduce it, which feels good in the moment, but it's not healthy.

33:19Cristina Criddle: That's next week on Tectonic from the Financial Times. This episode was presented by me, Christina Criddle. Our producers are Persis Love and Edwin Lane. Together, we've all done the reporting and writing for this season. Our executive producer is Flo Phillips. Sound design is from Breen Turner and Sam Giovinco. Fact-checking was done by Tara Russell. Production assistance from Josh Gabbert-Doyon and Topher Forges. The FD's global head of audio is Cheryl Brumley.

34:00Cristina Criddle: If you've been affected by anything in this episode, you can find some information in today's show notes for where you can find support. Thank you.