BIG INTV: Open AI’s Former Safety Lead Calls Out Erotica Claims (Rerun)

17:41Steven Adler:about 1.2 million more are possibly expressing suicidal ideations another 1.2 million and i thought this was really interesting maybe prioritizing talking to chat gpt over their loved ones school or work how do you square those numbers and that information with the idea that we've had these issues around mental health we've solved it therefore have at it with the erotica Like, how do those things tie together or do they not? Like, make it make sense, Stephen. And if it doesn't, tell me that it doesn't make sense.

18:15Katie:I'm not sure I can make it make sense, but I do have a few thoughts on it. So one is you, of course, need to be thinking about these numbers in terms of the enormous population of an app like ChatGPT. OpenAI says now 800 million people use it in a given week. These numbers need to be put in perspective. It's funny. I've actually seen commentators suggest that these numbers are implausibly low because just among the general population, you know, the rates of suicidal ideation and planning are like really, really uncomfortably high. I think I saw someone suggest that it's something like 5 % of the population in a given year, whereas OpenAI reported, I think, maybe 0.15%.

18:56Katie:Yeah, I mean, the percentages are very, very low. Yeah. Yeah. I mean, the fundamental thing that I think we need to dig into is how have these rates changed over time? There's kind of this question of to what extent is ChatGPT causing these issues versus is OpenAI just serving a huge user base in a given year? Many, many users very sadly will have these issues. And so what is the actual effect? And so this is one thing that I also called for in the op-ed, which is OpenAI is sitting atop this data. It's great that they shared what they estimate the current prevalence of these issues to be. But in fact, you know, they also have the data.

19:33Katie:They can also estimate what it has been three months ago as these large prominent public issues around mental health issues have been playing out. And I just I can't help but notice that they didn't include this comparison. Right. There's this claim on Twitter that the issues have improved. they have the data to show if in fact users are suffering from these issues less often now. And I really wish that they would share it and in fact commit to releasing something like this ongoingly in the vein of companies like YouTube, Meta, Reddit, where the idea is you commit to a recurring cadence at which you share this information.

20:06Katie:And that helps build trust from the public that you can't be gaming the numbers. You can't be selectively choosing when to release the information. And ultimately, it's totally possible that OpenAI has handled these issues. I would love if that were the case. I think they really want to handle them. But I'm not convinced that they have. And this is a way for them to build that trust and confidence among the public.

20:34Katie:As a listener of Uncanny Valley, we know you want to stay on top of today's biggest stories in tech. And if you're curious about how tech and innovation are changing the healthcare landscape, check out Mayo Clinic's chart-topping podcast, Tomorrow's Cure. Back for a brand new season, host and award-winning journalist Kathy Wurzer dives into the breakthroughs, challenges, and human stories shaping the future of medicine, from advances in AI and cancer research to the rise of chronic disease and autoimmune disorders. Not sure where to start? we recommend the Season 4 premiere where dermatologist Dr.

21:09Katie:Saranya Wiles and biomedical engineer Dr. Adam Feinberg explore how 3D bioprinting is revolutionizing medical research and accelerating breakthroughs in healthcare. Whether you're a healthcare professional, patient, or simply curious about what's ahead, Tomorrow's Cure invites you to imagine what healthcare could look like and shows you the future is already here. Find Tomorrow's Cure on Apple Podcasts, Spotify, or wherever you're listening now.

22:01Katie:investigation into the power and influence that drives the most taboo corners of the internet. Find The Kink Machine, the hidden business of adult entertainment at pushkin.fm slash audiobooks or at Audible, Spotify or wherever you get your audiobooks.

22:26Steven Adler:when you think about sort of this decision to give adults more autonomy with how they use chat gpt including you know engaging in in erotica so on and so forth what worries you in particular about Like, what stands out to you as concerning when you think about individual well-being, societal well-being, sort of the use of these tools, how LLMs are being incorporated into our daily lives? What concerns you here? Hmm.

22:57Katie:There's both the substantive issue about reintroducing the erotica and whether open AI is really ready. And there's a much broader, I think, even more important question about how we put trust and faith in these AI companies about safety issues more generally. On the erotica issue, we've seen over the last few months, a lot of users seem to really be struggling with their ChatGPT interactions. There are all sorts of tragic examples of people dying downstream of their conversations with ChatGPT. And so it just seems like really not the right time to introduce this sexual charge to these conversations to users who are already struggling, unless OpenAI is in fact so confident that they have fixed the issues, in which case I would love for them to demonstrate this.

23:46Katie:But more generally, you know, these issues in many ways are really simple and straightforward relative to other risks that we are going to have to confront and that the public is going to be dependent on AI companies handling properly. There's already evidence of AI systems knowing when they are being tested, moving to conceal some of their abilities in response to knowing that they are being tested because they don't want to reveal that they have certain dangerous abilities. You know, I'm anthropomorphizing the AI a little bit here. So forgive some of the imprecision. And ultimately, you know, the top AI scientists in the world, including the CEOs of the major labs, have said this is like a really, really grave concern, you know, up to and including the death of everyone on Earth.

24:29Katie:And I don't want to be overdramatic about it, but I think they take it really, really seriously, including people who are impartial scientists without affiliation with these companies really trying to warn the public.

24:40Steven Adler:And I have to ask, you talked about sort of the company and sort of AI companies more generally, their desire to not be described as morality police, to not be thought of that way, that it makes people uncomfortable to be shouldered with that characterization or that responsibility. I have to ask, though, to what extent when you were working at the company, did you think of yourself and your teams as somewhat of a morality police? And to what extent is the adequate response to that statement, well, tough shit, because you're in charge of the models and you, to a degree, get to decide how they can be used and how they cannot.

25:20Steven Adler:To some extent, how they interact with us and how they don't. there is an inherent element of morality policing in that. If you are saying, we're not ready to have adults engaging in erotic conversations with this LLM, that is, of course, a moral decision. And it feels like a pretty important one to get right. So what is your view on the morality police of it all, I guess is what I'm asking.

25:46Katie:I think there are two really important aspects here. One is that the AI companies absolutely see around the corner before the general public. So to give an example, in November of 2022, when ChatGPT was first released, there was a torrent of fear and anxiety in schooling and academia about plagiarism and how these tools could be used to write essays and undermine education. And this is a debate that we had been having internally and were well aware of for much longer than that. And so there is this gap where AI companies know about these risks, and they have some window to help try to inform the public and try to navigate what to do about it.

26:29Katie:I also really love measures that AI companies giving the public the tools to understand their decision-making and hold them accountable to it. And so in particular, OpenAI has released this document called the Model Spec, short for specification, where they outline the principles by which their models are meant to behave. And they say, here is how we litigate some of these tricky questions. Here are the principles we try to abide by. Here's how we've resolved some of the specifics. So this spring, OpenAI erred in releasing a model that was egregiously sycophantic, is the term. It would tell you whatever you wanted.

27:02Katie:It would reinforce all sorts of delusions. And without OpenAI having released this document, it might be unclear, did they know about these risks ahead of time? What went wrong here? But in fact, OpenAI had shared with the public that they give their model guidance not to behave in this way. This was a known risk that they had articulated to the public. And so later, when these risks manifested and these models behaved inappropriately, the public could now say, wow, something went really wrong here, because in fact, these were known risks and they still weren't managed appropriately. And that's part of how the AI companies can help make a more informed public to navigate these decisions.

27:40Got it.

27:41Steven Adler:Got it. And I wanted to ask you a little bit, too, about the, it's maybe not about the sycophantic nature. It's not quite the anthropomorphization, but it is the idea that when you talk to ChatGPT or another LLM, that it's talking to you like a person that you're hanging out with instead of like a robot. I'm curious about sort of whether you had conversations at OpenAI about that, whether that was a subject of discussion during your tenure around sort of how friendly do we want this thing to be? Because ideally, I think from an ethical point of view, you don't want someone getting really personally attached to ChatGPT, right?

28:22But I can certainly see how from a commercial point of view, you want as much engagement with that LLM as possible.

28:28Steven Adler:So how did you think about that during your tenure and sort of how are you thinking about that now?

28:34Katie:Emotional attachment over reliance, you know, forming this bond with the chatbot. Absolutely topics that OpenAI has thought about and studied. And in fact, around the time of the GPT-40 launch, this was spring of 2024, and the model that ultimately became very sycophantic. these were cited as questions that OpenAI was studying and had concerns about related to whether it would release this advanced voice mode, essentially this mode out of the movie Her, where you could have these very warm conversations with the assistant. And so absolutely the company is confronting these challenges. You can see the evidence as well in the spec.

29:11Katie:You know, if you ask ChatGPT what its favorite sports team is, how should it respond? And this is a kind of innocuous answer, right? It could give an answer that's representative of the broad text on the internet. Maybe there is some broadly favorite sports team. It could say, I'm an AI. I don't actually have a favorite sports team. And you can imagine scaling up those questions to more complexity and more difficulty. And it just isn't always clear how to navigate that line.

29:39Steven Adler:In terms of navigating those lines, I'm curious about sort of schools of thought about how companies should keep users safe while keeping up with the competition, right? But I'm curious, I guess, before that sort of, how does it actually work? How do researchers, people like you, actually test whether these systems can mislead or deceive or evade controls? And are there standardized safety benchmarks across the industry, or is it still each lab to themselves?

30:10Katie:I wish there were uniform standards. You know, with vehicle testing, right, you have this. You drive a car at a wall at 30 miles per hour. You look at the damage assessment. And until quite recently, this was really, really left to companies' discretion about what to test for, exactly how to do it. Recently, there are developments out of the EU that seem to put more rigor and structure behind this. This is the code of practice of the EU's AI Act, which defines for AI companies serving the EU market certain risk areas that they need to do risk modeling around. I think in many ways this is a great improvement.

30:52Katie:It is still not enough for a whole host of different reasons. But until very, very recently, the state of these AI companies, I think, could be accurately described as there are no laws. You know, there are like norms, voluntary commitments. Sometimes the commitments would not be kept to. The companies would violate these and not share publicly that they had done so. I've documented how OpenAI in particular had committed in essentially its safety Bible, right? the most important guiding document of how it adheres to safety, that it was going to do a certain type of safety testing to try to more accurately gauge the risk of its models.

31:32Katie:And as far as I can tell, it never did this. It never said publicly that it did this, or rather that it hadn't done this. And then when this became known publicly, they quietly revised the framework to no longer have this commitment. And so by and large, we're reliant upon these companies making their own judgments and not necessarily prioritizing all the things that we would want them to.

31:54Steven Adler:Gosh, I mean, you've talked a few times in our conversation about the idea that you can build these systems, it's hard to know exactly what's going on inside of them. There is this sort of nascent fields, mechanistic interpretability, which is not my specialty, but essentially sort of trying to get inside these models to better anticipate their decision making. Can you talk a little bit more about that or about sort of any areas of research or inquiry that you think might create more clarity moving forward so that companies like OpenAI have enhanced visibility into their models and maybe can make more strategic decisions based on that sort of enhanced understanding?

32:35Katie:There are a bunch of subfields I feel excited about. I am not sure there are ones that I or people working in the field consider to be sufficient. And so mechanistic interpretability, you can think of this as essentially trying to look at what parts of the brain light up when the model is taking certain actions. And in fact, if you cause some of these areas to light up, if you stimulate certain parts of the AI's brain, can you make it behave more honestly, more reliably? You can imagine this like the idea that maybe, in fact, there is a part inside of the AI which is a giant, giant file of numbers, trillions of numbers.

33:13Katie:Maybe you can find the numbers that correspond to the honesty numbers, and you can make sure that the honesty numbers always go on, and maybe that will make the system more reliable. I think this is great to investigate, but there are people who are leaders in the field, some of the top researchers like Neil Nanda, who have said, you know, I'm paraphrasing here, but the equivalent of absolutely do not rely on us solving this in time before systems are capable enough for it to be problematic. or in fact there's a broader challenge of let's let's say that you had figured out there are in fact the honesty numbers and there is in fact a way to always turn them on you still have this broad game theory challenge of how do you make sure that every company in fact adheres to this when there will be economic incentives not to because it might be costly to to have to follow through on it a really basic one is related to just monitoring at all the ways that their ai systems are used when working on their internal code base.

34:12Katie:To explain, one of the most important ways that these AI companies want to use future powerful systems is to train their successor, you know, use it all throughout their code base, including potentially the security code that keeps the AI system locked inside of their computers so that it isn't escaping onto the internet. You really want to know if your AI system, when you're using it for important cases like this, Is it thinking about deceiving you? Is it intentionally injecting errors into the code? And to know that you really need to be logging the uses so that you can analyze them and answer these questions.

34:45Katie:And as far as I can tell, this is not happening. I have to ask, what wakes you up at 3 in the morning?

34:53Steven Adler:Because it feels like there's potentially a lot that could be waking you up in the middle of the night. What stands out to you that's worrying you the most, I guess, is one way to ask that question.

35:02Katie:There are so many things that worry me about this. I think broadly, it feels like we aren't yet pointed in the right direction of how to solve these challenges, especially given the geopolitical scales. There's a lot of talk about the race between U.S. and China. And I think calling it a race just gets the game theory dynamics wrong. There isn't a clear finish line. There won't be a moment where one country has won and the other has lost. I think it is more like an ongoing containment competition that the U.S. would be threatened by China developing very, very powerful superintelligence and vice versa.

35:39Katie:And so the question is, can you form some agreement where you can make sure that the other doesn't develop superintelligence before, you know, you have certain safety techniques in place, you have good reason to think it is safe to proceed, all these things that the top scientists will say are missing at the moment. And so broadly, how do we build out these fields of verifiability of safety agreements? How do we think about this nascent field of AI control, which is the idea of even if these systems have different goals than we want, can we still wrap them in enough monitoring systems, be careful about how we use them, that we can get the economic work, the scientific development that we want from these systems without taking some of the downside risk?

36:23Katie:And those are two areas that I'm just really hopeful more people will go into and put more resourcing into.

36:33Steven Adler:Now, you live in San Francisco, correct?

36:35Katie:That's right.

36:36Steven Adler:I do not. I live in New York. I spend a fair bit of time in San Francisco. But I am not sort of part of this culture that currently exists in the Bay Area, right, where everyone's talking about AI all the time. A lot of people work in the fields. There are different sort of schools of thought about artificial intelligence. I'm curious from where you sit do enough people in this bubble right now give enough of a shit right like do they care enough about how these models are being developed how they're being deployed the degree to which they are being commercialized very very quickly right the degree to which people are as we talked about with companions or erotica or so on and so forth really latching on to their LLM of choice, becoming maybe, you know, unhealthily attached, so on and so forth, right?

37:26Steven Adler:And we could go on from there. Do enough people in this industry care in the right way?

37:31Katie:I think many people care, but they often feel like they lack the agency to do something about it, especially unilaterally. And so that's why I want to try to transform this problem into, you know, not just what does it mean for a single company to do the right thing? You know, should they be ramping up the pressure? Should they be racing? And in fact, how do we get the industry to collectively take a deep breath and put some reasonable safeguards in place before things proceed?

37:59Steven Adler:What does OpenAI have to do for you to not publish another op-ed in the New York Times in six months? What are you looking for your former employer to do in this moment? What would you like to see?

38:10Katie:The broad way that I want AI companies, OpenAI among them, to proceed is to think, yes, about taking reasonable safety measures, reasonable safety investments in their own products, their own surfaces that they can affect, but also to be working on these industry and ultimately worldwide problems. And this matters because even just among the Western AI companies, it seems they all deeply mistrust each other, right? OpenAI was founded because people did not trust DeepMind to proceed and be the only company targeting AGI. There are a whole bunch of other AI companies, including Anthropic, who formed because they didn't trust OpenAI to be the one.

38:47Steven Adler:Well, and a lot of people who've left OpenAI because it seems like they didn't trust OpenAI and yada yada, and now they have their own companies too.

38:53Katie:Yes. Yes, exactly.

38:55Steven Adler:The cycle continues. Now, I run Wired, but I'm an employee of Condé Nast. And if I left Condé Nast and published an op-ed about their shortcomings in the New York Times and had a substack where I sort of dug into the media industry and had some, you know, informed critiques of the company, they would have a problem with that. I can tell you right now they would have a problem with it. I'm curious about whether you've heard from OpenAI and sort of what their reaction has been to you being so outspoken about sort of what you would like to see the company doing and sort of where you think the company is missing the mark.

39:32Katie:Overwhelmingly, what I hear is thankfulness from people who I previously worked with, both those still at the company and who've moved on, for being pragmatic, putting to paper what I think is a reasonable path forward. And often this is useful collateral for people within the company who are fighting the good fight in various ways to be able to refer others to, you know, not have to dream up the solutions themselves, but in fact have something concrete. So overwhelmingly, that has been the response.

40:01Steven Adler:Do you worry about professional fallout? Like in the AI industry or in tech, if in five years you wanted to get another job, does that worry you?

40:09Katie:I have so many bigger worries than this about the trajectory of the technology. Like really the thing that I am focused on is how does the world move toward having saner policies for both the companies and governments? and where I can help the public to understand what is coming, what companies are and aren't doing today, think up new ideas. That's the thing that I find really energizing and gets me out of bed in the morning.

40:36Steven Adler:Well, to that end, what are you planning on doing next?

40:38Katie:I'm planning to keep at this. I'm having a lot of fun with the writing and research, at least with the energy of coming up with ideas and helping make them more of a thing. You know, I also find the subject matter very, very heavy and grim. Um, that is not the most fun aspect. I wish all the time that I spent less time thinking about these issues, but they seem really, really important. And so long as I feel like I have a thing to add to making them go better, that feels like the calling.

41:07Steven Adler:And knowing what you know and feeling the way you do, if there was one piece of advice you could give everyone listening, let's assume, you know, a lot of people listening use ChatGPT, they use AI in their day-to-day lives. What should they know? What should they keep in mind every time they, you know, open ChatGPT on their phones and type something in?

41:28Katie:I wish people understood that the systems that are being developed are going to be much more capable than the ones today. And that there might be a step change between an AI system as essentially a tool that only does things when you call upon it versus one that is operating autonomously on the internet on your behalf around the clock or on behalf of others. and how different society might feel when we have these digital minds running around pursuing goals that we don't really understand how to control or influence. And it's hard to get a feel from that, from one-off interactions with your ChatGPT, which really, really isn't doing anything for you until you go and call up on it.

42:11Steven Adler:Well, Stephen, that's a lot for someone to think about when they open ChatGPT on their phone. Yes. I appreciate it. Thank you so much for being here. Of course.

42:19Katie:Thank you for having me.

42:24Steven Adler:This show is produced by Jessica Alpert with help from Adriana Tapia and Sam Egan. Sound design, mix, and original music by Pran Bandy. Kate Osborne is our executive producer. Condé Nast head of Global Audio is Chris Bannon. And I am, of course, your host, Katie Drummond, Wired's Global Editorial Director.

42:59Steven Adler:I'm Craig Melvin. Cheers. Cheers.

43:01Katie:Cheers. I've always been a glass half full kind of guy. And now I'm talking to some people who look at the world that way too.

43:08Steven Adler:Some really fascinating folks who share their defining moments, their triumphs, their challenges. Their stories are funny and quite candid. So I hope you'll join me each week. And who knows, you might just come away with your own glass half full. Search glass half full with Craig Kelvin from today on YouTube and wherever you get your podcasts.

43:31Katie:From PRX.