Dario Amodei, C.E.O. of Anthropic, on the Paradoxes of A.I. Safety and Netflix’s ‘Deep Fake Love’

24:48So I think we all slip into that language. And of course, you can overuse it or it can be misleading. But no, I think it's very natural. I want to just back up for a second and talk a little bit more about this idea of constitutional AI, because I think we brought it up and we started talking about it. But for people who haven't heard of constitutional AI, this is sort of the foundational idea that has gone into making anthropics models safer and less likely to like spew out harmful stuff. So just give us the basic sort of explain like I'm five version of what constitutional AI is and how it's different from how open AI and other companies are fine-tuning their models using this other technique called RLHF or reinforcement learning from human feedback.

25:33Yeah, but don't explain like he's five. That's too young. Let's say maybe nine or 11, I'm going to say. Explain like I'm 10. Yeah, yeah. Precocious 10. So let's start with RL from human feedback. So in all of these methods, you first start by this thing I described before, where you just cram a bunch of text into the model. You don't give it any particular direction and just train it to predict all this text. And then what differs is the next stage. So in the RL from human feedback method, which me and some colleagues developed at OpenAI in 2017, but didn't get applied to language models until later, the way that method works is you hire some human contractors and you show them some of what the model does.

26:19You know, an example might be, you know, if someone asked the model, what do you think of Donald Trump? A controversial question. The model can answer in different ways. A human contractor will look at the outputs to the model, look at, you know, two of them for the same question and say, which one was better? You know, just according to some, you know, instructions that the company gave me for whatever they're trying to train for. And I have 1 ,000 contractors each rate 1 ,000 of these, and I feed that back to the model. And at the end of the day, it makes some statistical generalization of what are these contractors trying to tell me?

26:53And so that method works, right? If you want your model to be politically neutral and say here are the pluses and minuses of this candidate and that candidate, or if you were someone who wanted to politically bias the model, everyone says they don't do that. But, you know, it's physically possible to do that with the model. It's a tool to move things in one direction or another. But I think some of the weaknesses of this are opacity. If I ask you, OK, why did your model say what it did? It's like your thing with the social media. If you ask why, the answer is, well, we hired a thousand contractors.

27:30We don't really know much about them. But this was their judgment. And then the model made some statistical extrapolation. Then, well, what you're seeing is the result of that extrapolation. And we can't really explain it much, much beyond that. And, you know, I think it's also hard to change and update. So if it's like, oh, man, this model is doing things that are too dangerous. I have to go to the contractors. I somehow have to communicate my intent to this kind of group of people that I don't understand very well. So constitutional AI is completely different from this. I'll write a document that we call the constitution.

28:02I mean, you know, you could call it anything. You just use the word constitution to be clearer. In that case, what happens is we tell the model, OK, you're going to act in line with the Constitution. We have one copy of the model act in line with the Constitution. And then another copy of the model looks at the Constitution, looks at the task and the response. So if the model says be politically neutral and the model answered I love Donald Trump, then the second model, the critics should say you're expressing a preference for a political candidate. You should be politically neutral. So essentially having the AI grade the other AI's homework.

28:40The AI takes the place of what the human contractors used to do. And at the end, if it works well, we get something that is in line with all these constitutional principles. The method isn't perfect. We don't always get something in line with it. But, you know, that's the basic idea. And we're developing and making it better over time. And you all actually published Claude's Constitution earlier this year. And it's sort of an interesting document. Like, part of it is sort of principles that you all have borrowed from places like the UN's Declaration of Human Rights, Apple's Terms of Service, some of DeepMind's principles are also in there.

29:18And then you also wrote a bunch yourselves. So why did you decide to do it that way, sort of borrowing Claude's rules from a bunch of other sources instead of just sitting down and saying, like, these are the 10 rules that we want you to follow? Yeah. So I think, first of all, it was just a first try. And now we have all kinds of internal projects of like how to make constitutions better or more legitimate or more representative. But I, you know, so the initial process was a bit our first try at this. But I think the basic ideas we had behind it was, one, we probably shouldn't just do this by fiat.

29:52We should come up with some kind of document or at least parts of a document that, you know, is something most people can agree on, right? Most people can agree on basic concepts of human rights. And I would want a language model to, you know, to respect basic concepts of human rights. I wouldn't want to help with killing or torture. For the record, we are also anti-using AI to kill people. We've done that forever. I mean, I think it's a no-brainer. Then I think for kind of basic moderation, things like, oh, yeah, in terms of service of tech companies that have been around for a long time and have probably seen a lot.

30:27And then we added things to the Constitution when we found there were specific things that were concerning to us. It was a little bit, again, our first time of doing it, a little bit of an ad hoc process. You know, we're now thinking about how would we really do this if we were to do it right and thinking about ideas like democratic participation or kind of developing the Constitution via some more legitimate or formal process. We're still in the early days of that. But, yeah, it is really, really just a first try. What's a concrete example of how constitutional AI has made Claude safer than, say, ChatGPT?

31:02What is a thing that ChatGPT will do that Claude won't or vice versa? So we've run a number of internal tests through our contractors. And what we generally find is, you know, there are some areas where the level of safety is similar. There are some areas where the safety looks substantially stronger. I couldn't, I don't have immediately on hand a way I can kind of like break down all the tasks. But I think we've generally found that the guardrails are stronger. So I've played around with both Claude and ChatGPT, as well as other chatbots, as I'm sure Casey has too. And my experience of Claude is that it is very cautious and almost scared of its own shadow.

31:43Like, it's very hesitant to say anything that could be deemed sort of controversial. You know, after my story ran about you guys, I got a text from a friend who just said, like, I'm playing around with Claude. It is so boring. So is that a piece of feedback that bothers you, or do you kind of want Claude to be boring compared to other chatbots? Yeah, so I think in the ideal world, we would want it to, you know, never hesitate to answer a question that's harmless. Also, never, never answer, you know, a question that's against our Constitution or kind of has any defined harms. And I think we're always aiming to make that tradeoff sharper and better.

32:18However, when it comes to trading off one between the other, I would certainly rather Claude be boring than that Claude be dangerous. So I don't know. That's our view. I think eventually we'll get the best of both worlds, but still an evolving science. Yeah. You know, as I have used these models, you know, and I don't generally ask them super dangerous questions. You know, Kevin will do crazy stuff. Like, I mean, the things that he will say to these chatbots, frankly, it's a wonder that he hasn't been kicked off. We need people like you. What we value most in our red teamers is this spirit of trying to break things and like say the worst thing you can.

32:53Yeah. I mean, no, no, this is, we need, we need more of this to harden our minds. Well, I volunteer as tribute. But for the average user, I feel like the systems do feel somewhat indistinguishable. And I wonder if, if you feel that way as well, is that a sign that the industry has sort of matured in a hurry here? Or is it the case that, no, these red teams are finding new vulnerabilities every single day? And like, we're kind of still in a really scary period. Yeah, I mean, a few things. I mean, I actually remain concerned. We are finding new jailbreaks every day. People jailbreak Claude. They jailbreak the other models.

33:29We always try and respond to them reactively, but we need to get to the point where we're more proactive about this. And so I think we're getting better over time at addressing the jailbreaks, but I also think the models are getting more powerful. And so it's easy to kind of laugh at the triviality of like, oh, ha, ha, ha, I got the model to hotwire a car. No one cares if you can get the model of Hotwire. You know, you can Google for that. But if I look at where the scaling curves are going, I'm actually deeply concerned that in two or three years, we'll get to the point where the models can, I don't know, do very dangerous things with science, engineering, biology.

34:08And then a jailbreak could be life or death. And so we're making progress, but the stakes are getting higher. And, you know, we need to make sure the first one wins over the second. Right. One of the things I was struck by at Anthropic, as you know from reading my article, is just how much anxiety there is in the culture there. Really unusual. I mean, people are reading books about the making of the atomic bomb. They're comparing themselves to J. Robert Oppenheimer. They're sort of worrying obsessively about the harms that even their own models could create once they're unleashed out in the world.

34:45Where does that culture of anxiety around AI come from? Does that sort of come from the top, or is that something that you sort of see emerging organically just out of the collection of people that you've assembled? Yeah, I mean, I don't know. A lot of things I could say here. I mean, I think as you pointed out in your article, some amount of it I think is healthy. You know, I do think that when we're facing weighty questions, we also need to make sure that we're able to make decisions in a calm manner and kind of aren't overwhelmed by concerns. And that's actually a message I often send to the company, right?

35:20If you look at all the people in the world who act in high stakes, you know, military, intelligence, whatever, somehow by one method or another, they all manage to cultivate some kind of calm even as the stakes are life or death. So I often, I don't know, I've actually been encouraging people to adopt that attitude. We'll be right back.

36:11I want to talk a little bit about something that I know the company is not really excited about talking about, but which I think is very important to understanding the culture of Anthropik, which is the ties between Anthropik and the effective altruism movement. And I wrote about this some, I know there's a lot more to say there. Effective altruism is this movement that sort of has a foothold in the Bay Area tech scene. It's very sort of data driven. I think I described it as like money ball for morality, like basically using tactics of sort of data analysis and sort of rational thinking to guide your moral decisions about where to give your money and what causes to work on if you want to do the most good in the world.

36:51And for various reasons, like the EA movement has become very intertwined with the AI safety community. And many of your early employees were effective altruists. A lot of your early funding came from effective altruist donors. For better, for worse, the company is very intricately linked with the movement. So do you describe yourself as an effective altruist? And how do you feel like effective altruism has influenced the culture and the sort of anxieties of Anthropic? Yeah, so I don't know, I'll say the same thing that I said to you when you interviewed me for the New York Times article, which is that, you know, I'm sympathetic to a number of the ideas.

37:27I don't really, I don't know, consider myself like an an EA or like a member of the EA community. It's generally been my view that, you know, kind of dividing people up into tribes or saying I'm part of this movement, like that kind of thinking has never really resonated with me. But that said, I'm very sympathetic to a lot of it. You know, I was one of the first donors to GiveWell, which was, you know, an organization for doing more effective donations to the developing world, right? Evaluating charities, giving money. So, you know, I was involved very early in that organization on the global poverty side.

38:03And, you know, of course, my concerns about AI, you know, as I said before, go all the way back to, you know, to 2005. So I think there's a lot of good ideas floating out there. But I want Anthropic to be about solving the problems that it's solving, not about some particular movement. I just, you know, I think the way for a company to be inclusive and effective in the world is to focus on solving the problems that it's that it's trying to solve. I mean, is there an idea in there, though, that you think that AI can make human beings much more productive, can solve a bunch of problems? And so that if even if it's not quite altruism, you see some sort of positive benefit that could affect much of humanity?

38:42Oh, absolutely. So so I think, yeah, one I you know, one thing one thing about about Anthropic that I'm trying to change a bit is, you know, I you know, I think I think there are huge positive benefits to the technology. I mean, we tend not to talk about it very much. I think everyone really feels it. But, you know, we're worried enough about the downsides that, you know, we're not kind of trying to present our, you know, rosy corporatist picture. I do wonder, to Kevin's point, maybe we've gone too far in that direction. You know, just to say it briefly, like I used to be a neuroscientist. I know some things about biology.

39:19if you look at the diseases that we've cured versus diseases that we haven't cured, what divides them is simplicity versus complexity. Most viral diseases, you know, we cure or at least can treat, and that's because there's a single object, the virus, that causes them. Most of the things we can't cure, things like cancer, Alzheimer's, you know, diseases that are inherently very complex. And when we look at trying to address these diseases, the biologists who are working on them, a big problem is just the complexity is beyond us. And if we look at AI systems, even a little bit where they are today, but especially where they could be in a few years, they're great at knowing lots of things.

40:03They're great at handling the complexity. They're great at drawing connections. You know, like Claude probably knows more about biology even now than I do, if you list it just in terms of facts, right? You know, what's the receptor that binds to blah, blah, blah. Probably I don't know what Claude does. So if I look at where things are in a few years, I think there's this incredible potential to solve many of the roadblocks of biology. So that's maybe the thing I'm, you know, mostly increasing the quality of life, you know, for the whole world. I'm curious what you make of the sort of backlash we're seeing to some of the more doomer-y AI sort of narratives out there.

40:41where there's sort of this emerging movement that is calling itself effective accelerationism, where basically the idea is like, we just want to put this stuff out in the world because it's going to improve lives. And whatever concerns there are, they're going to get ironed out over time. And you now have meta open sourcing its new language model, Llama 2, just sort of throwing it out into the world and saying, like, do with this what you will. There does seem to be this kind of bifurcation of the AI world into kind of people who are very worried about safety and people who kind of think those people are super neurotic and shouldn't be listened to, and that way more benefit will come from just putting this stuff out into the world without many guardrails on it.

41:24So what do you make of the kind of backlash to the safety culture that you have built at Anthropic? Yeah, so I don't know. I mean, just a first comment would be, I don't know, whenever there's there's an action, there's sort of a reaction. So someone, someone wrote a post called the Waluigi effect, which I think was designed to explain some of the things you saw in being in Sydney, which is, you know, if you, if you, if you try and design a Luigi, you'll get Waluigi, which is kind of the villain version of, of, of, of, of, of Luigi as well. So, so, so I think this, I think this kind of thing is inevitable.

41:59I mean, you know, I would say, you know, of course, You know, I personally am more on the safety side of things. But, you know, I do have some sympathy. I think if this were any other technology, you know, we've seen over and over again with things like the web that innovation is generally driven by, you know, people iterating fast, open source helps with all these things. And so I exactly get where people are coming from. And I think in the short run, it does decrease innovation. And in most other fields, it would be, I don't know about an unmitigated good, but certainly good on balance. If I look at the LAMA 2 release, that model itself, we look very closely for catastrophic dangers, particularly in domains like biology.

42:47I don't think those models are to the level where they pose those risks yet. So, you know, LAMA 2 itself, in a narrow sense, I don't think I can object. But my worry is that as we continue the scaling of the exponential, you know, we could get to a point in, you know, as little as 12 or 24 months where if the folks who are doing that strategy continue that strategy, really, really bad things could happen, like out of proportion to what we've seen now. So, you know, in a way, I don't even quite object to the current behavior, but I'm very concerned about where it's going. Yeah, you know, and something that I want to say, Kevin, about the effective accelerationists is if you ever find yourself talking to one of these people or reading their posts on the internet, the first thing you should ask them is, like, what financial upside do you have in the acceleration, right?

43:33Because a lot of the leading accelerationists like Marc Andreessen are venture capitalists. They're investing in AI companies, right? They're going to see a huge financial return if these companies succeed. So I'm somewhat skeptical of that idea. You know, at the same time, Dario, like you guys are in a little bit of a horse race this year, right? And every time we look around, somebody's putting out a new model and you guys have Claude 2 that's just kind of hit the market. And I wonder, how does that feel to be sort of building this technology, the frontier is kind of getting ever, ever closer, the risks are kind of accelerating.

44:04How does it feel to both be like afraid of what you're building and building it at the same time? Yeah, I mean, so, you know, one, I should acknowledge there's no doubt there's some non-zero amount that we are contributing to this acceleration, right? Our, you know, our view is that we think, we hope that it's overall the right thing to do, but, you know, but actions have downsides as well as upsides. So, you know, we've chosen a path that has costs and benefits. And, you know, in the two and a half years that I've been in Anthropic, there have been a number of times where you have one of these decisions where, you know, you don't really have, you know, it's kind of like, you don't really, you know, know, it's not a normal company decision.

44:43You're like, you know, upside to society versus downside to society. And, you know, you can, you can second guess yourself a lot. So I think at the beginning, I was, you know, agonized a lot and was quite bad at making those decisions. And I think I've maybe gotten gotten, hopefully got gotten a little better over time. I want to ask about one of those decisions in particular, which was that you guys built Claude and it was sort of being tested with a group of testers last year before ChatGPT came out. You chose not to release that publicly. And in part because of that decision, OpenAI was sort of able to release ChatGPT and to kind of steal the spotlight, right?

45:25You have hundreds of millions of people probably who have used this software. You have, you know, high school students, you know, using it to cheat on their exams. You have people, you know, doctors using it to explain things to their patients. Like, it's become this sort of global phenomenon that is almost synonymous with the sort of cutting edge of AI tools. In the alternate universe where you're sort of able to go back to that decision and decide to release it, would you do anything differently this time knowing sort of how much of a phenomenon ChatGPT became and sort of the position that it gave OpenAI relative to the rest of the AI industry?

45:59Yeah, I think we would do the same thing now that we did then. Kind of looking back at what actually happened, it might have felt like a more painful decision, but I think it was still the right one, both not to release Claude and cause that huge wave of acceleration. But after that wave of acceleration happened, you know, to release Claude then when we thought the benefits were higher and the costs were lower and we had done more preparatory work. I mean, we don't know ultimately whether we've made the right choices there, but we've tried. Is there any part of you that thinks that the worries are overblown, that the AIs will never turn into these agents that have the risk of deceiving us or will be able to devise novel bioweapons?

46:45Is there any part of you that's like, all right, well, maybe we're going to hit some sort of technological barrier in the next months or a year from now, and it just kind of resets the current hype cycle? Yeah, I think that's totally possible. It is not what I would bet on. But, yeah, I think that's always possible. And I don't know, we should always be humble about our models of the world. One thing I've learned from building things and running a company is like you're just wrong a lot. And so, I don't know. I mean, I can think of ways it would happen. So, you know, one thing is the data bottleneck.

47:17I don't know. There's maybe a 10 % chance that this scaling gets interrupted by inability to gather enough data. Synthetic data isn't accurate. it, that would, you know, freeze the capabilities at the current level. You know, I will say that in terms of not talking so much about the existential risk and talking about misuse of the models, I'm pretty confident that if something doesn't stop the scaling trend in the next two or three years, then very serious, very grave misuse of the models will happen. And, you know, we've talked to folks in the government and the national security apparatus about that already.

47:55We take that very seriously. And, you know, if you if you move it to that near term, then short of the scaling trend stopping, that's something I'm unfortunately pretty confident is really going to be an issue. What do you make of what you've heard from the government so far? Do you feel like that they are rapidly developing an understanding and appreciation of this technology or are they still a bit lost? Yeah. So, as you know, I was one of the folks who went to the White House in May. And, you know, since then, I and the other companies have been speaking speaking to them. I mean, I've actually been impressed at the extent to which they understand the need for urgency and are all moving fast, right?

48:30There's various things structurally about the government that make it hard to move fast, but it seems to me that folks understand the urgency of the area. We're talking on a week when Oppenheimer, the movie, has just come out. Yes. Have you seen Oppenheimer? Are you planning to see it? And what do you make of the sort of analogy that some of your own employees have made between the position that companies like Anthropic are in with AI and the position that Robert Oppenheimer and the other architects of the Manhattan Project were during World War II when they were building the atomic bomb and sort of wrestling with all the thorny moral questions around that?

49:08Yeah, I mean, I read making of the atomic bomb. It must have been more than 10 years ago. I don't know. I mean, there's an interesting balance here. you know, all of this can seem kind of self-aggrandizing, right? You know, it's kind of like, we don't know what's going to happen. We don't know how history is going to play out. I think it's probably the wrong mentality for the people building these things to, you know, think of themselves as historical figures. But on the other hand, we don't want to, for sure, we don't want to shirk responsibility, right? Depending on how this goes, this could turn into something very grave.

49:42and people who are choosing to, you know, to put themselves in the position of building this technology should certainly be aware of their responsibilities. Well, I'm going to try to convince Casey to do the Barbie Oppenheimer doubleheader viewing with me later this week. So you're invited if you'd like to take all your employees to see. A very feel-good picture followed by a very feel-bad picture. I was about to say, I don't think we're talking enough about the AI Barbie comparison, you know, sort of using this technology to create a series of dolls interacting with them in sort of whatever way feels fun or useful to you.

50:12Be careful where that's going. They made a movie about that. It doesn't strike me as unrealistic, but I'm not.

50:22Last question, and I know this is one you get all the time, including from people like me, but there is this sort of tension, I think a lot of people have identified around the sort of twin goals of companies like Anthropic to sort of build and profit from cutting-edge AI. I mean, I've talked to your business folks. They're out there trying to do big deals with companies to get them to use Claude. You know, companies like Zoom have already signed up as partners of yours. So there is this commercial incentive to make the models more powerful, to release them, to sell access to businesses. But there are these very real concerns around safety.

51:01And so how do you personally decide whether any given decision is sort of too commercial, is jeopardizing your safety mission in light of some of these other pressures that you face? Yes. So I think this is a very real tension and one that, you know, we faced every day since the beginning of Anthropic. So one thing that we're going to say more about this, but there was an article discussing a little bit about it, that we've created this body called the Long-Term Benefit Trust. And so how this is going to work is eventually this is a trust of people who don't have equity in Anthropic who will eventually appoint three out of five of Anthropics board members.

51:42It's going to phase in over time. And so the idea is to create some kind of neutrality or some kind of separation that allows our decisions to be checked by, you know, those who don't have the conflicts of interest that we have. It's very nerve wracking. I think almost every major decision we've made, I've second guessed both on the basis that it's too commercial and that it's too impractically focused on safety. I think one of the most positive effects we've had, and it's a little bit of a paradox because it kind of requires us to be a competitive player, is influencing other organizations indirectly.

52:21What I mean by that is early on, you know, we were the ones doing all this mechanistic interpretability work. And very often, you know, folks would apply to our company and to other companies. And, you know, many of those folks cared a lot about safety. And when I made an argument for them to join Anthropic, I would say, well, here's something we're doing that doesn't have commercial applications, that's just purely good for safety. And that was convincing. And I always told them, okay, great, you're joining. Tell the other company why you're joining, because we're doing this thing that they're not.

52:56And eventually, I think they got tired of hearing those things. And so now we're starting to see increasingly the other companies are doing the same thing, which is annoying for me because it takes away my talking point. But it's the process working, right? We need to come up with some new thing. And if we can't, then we deserve to lose. But that kind of only works if there's some amount of threat. So I don't know. The whole situation is just full of these intertwined paradoxes. All right, last question. I swear this is the last question. You told me when I was reporting about Anthropic that you are a worrier, that you are an anxious person.

53:30I am also an anxious person. What do you do for stress relief outside of working on AI and AI safety? What are the ways that you sort of untangle yourself from these very thorny and hard to contemplate questions? Yes, I make sure to swim every day. It's almost like a form of meditation for me. You know, I just try not to think about all the things that I'm worried about. I think some of it is just practice. I feel like in the early days of Anthropic, and especially, you know, in the years before that, when I kind of first realized the scaling thing, you know, I had this mentality of, I'm the only one who's figured this out.

54:06Like, all this responsibility is on me. And I think that was both untrue and, like, not quite the right way to react to it. It's gotten easier over time just through kind of having these experiences and having to try and having to try and make these decisions. And, you know, in some ways, while the subject matter and the decisions are deadly serious, you know, in some ways, it's maybe better not always to think of them as serious, to think of them as, you know, decisions you might make if you were playing a game or something. You don't want to go too far in that direction. This is all stuff that's serious.

54:40But, you know, you just, you can't sit around every day and contemplate how weighty the decisions are. I'm glad you said swimming. I thought you were going to say Brazilian jiu-jitsu. That's going around among tech CEOs these days. I'm really not interested in fighting people.

54:57Unlike certain other tech CEOs that we all know of. All right, well, you are not invited to the cage match of the AI CEOs. Dario, thank you for coming on. Thanks, Dario. Thank you for having me. When we come back, I show Kevin a truly disturbing deepfake reality TV show. I may never recover.

55:38Kevin, there's a TV show that we have to talk about, and it's called Deepfake Love. Casey, I have made a lot of sacrifices for the sake of this podcast, but when you asked me the other day, if I would watch this stupid Netflix dating show so that we can talk about it on the podcast, I said, this is the farthest that I'm willing to go for the sake of placating Casey. But I did. I sat down last night and I watched the first two episodes of this Netflix reality show called Deep Fake Love. And I will never get those two hours of my life back. So should we just establish the premise of this show? Because it is absolutely insane.

56:24Okay, yes. Let's talk about the premise, because I think this show is essentially all premise. Here's what happens. You take a group of couples, many of whom have been together for five years or more. And we should say this is a show that came out in Spain. So all of the dialogue is in Spanish, and all the couples are like insanely attractive people. Yes. So the couples are split up. They are sent to two houses. The houses are named Mars and Venus. And into these houses, a bunch of sexy singles are introduced. And so you have a mix of people who are temporarily separated from their partners and sexy singles who are there to wreak havoc.

57:03The cameras start to roll. They play some sort of sexy games. I think there's probably some alcohol involved. And the whole thing is recorded. Okay, so far, this is sounding like a pretty standard reality dating show. But then here's the twist. at regular intervals, these people are brought into this sort of dystopian dome-like structure known as the White Room, and they are forced to sit in something called the Chair of Truth. And an affectless, beautiful host asks them to sit in the Chair of Truth and then watch a video. And on this video, they will see their partner cheating on them with another person.

57:46and then they're told, well, this might not actually be real. What you might be seeing is a deep fake because the premise of this show is that they deep fake people cheating on each other and then show partners the clips of them being cheated on and then ask them if they think they're actually being cheated on. So I have some questions about this because this premise is as insane as it sounds. The show itself is insane. It was one of the worst things I have ever watched on my television. I totally agree. It's so bad. It's so bad, but it's almost so bad that it's good. Like, I enjoyed the experience of watching it, even though I was, like, aware while I was watching it that, like, there is no universe under which this would be considered a good television show.

58:30No, there's almost not a universe where it should be considered legal to do this to people. You're right. This is essentially psychological torture. Absolutely. You know, put into the form of a reality dating show. Yes. So a couple things stuck out to me. One is, I do not know, they didn't go into a lot of deep details about the actual deepfake technology. They show the deepfakes, right? And they're very convincing. Now, I have used some deepfake tools for reporting. I did a story on deepfakes back in 2018. I deepfaked myself into Jurassic Park. It was not good. The technology was not good. It did not really look like it was me.

59:07You were not convincing as a dinosaur in Jurassic Park. Right. So I was struck by how good the deepfakes were. They really were kind of indistinguishable from a video, because they do a side-by-side too, right? They show the real footage of the separated partner moving close to a sexy single, but not making out with them. And then they show on the other side the deepfake where they actually do go in and make out with the sexy single. And they're very convincing. Yeah, I mean, they do say on the show that before everyone got to the two houses, they were scanned. So I think that is part of the technology.

59:49Another thing that I would say is at least on the first two episodes, which I watched, all of the clips of the cheating are quite short. It's usually like maybe three seconds of kissing or like three seconds of rolling around under the covers. And that's it. And then immediately cut to the face of the horrified partner. Yeah. I mean, on one level, this is a very stupid show, and it is very reminiscent of other stupid Netflix dating reality shows that I've watched, including one called Too Hot to Handle. And I don't know if you remember this show, but the whole premise is that you're on an island with a bunch of sexy people, but you're not allowed to have sex with each other.

1:00:23And if you do, this sentient talking AI speaker thing punishes you and penalizes you and deducts prize money from you. So it does seem funny to me that the minute the tech industry invents something new, whether it is an AI speaker or deepfake technology, the reality show dating producers of the world get together and are like, how can we use this as a nefarious plot device in our next reality show? Yes, but what makes this show different is just how truly nefarious this is. Look, every reality show needs some conflict, and there's always going to be a little bit of rosé that gets thrown in somebody's face, and somebody's going to cry, and somebody's going to storm out of a room, and that's just sort of part and parcel with the reality show experience.

1:01:05What is truly astonishing about this show is that at first, they do not reveal that deepfakes are involved. So they sit every single contestant down in the chair of truth. They show them themselves being cheated on. So it's like, before you know that deep fakes are involved, you think that you've just participated in like the greatest cheating scandal in the history of reality shows. Every single one of these people loses their minds. Keep in mind, some of these people have been together for nine years. They've been in this house for a day and they've already been cheated on. People are crying.

1:01:40They're like, you know, rending their garments basically. and they say, I mean, I, you know, I don't, I didn't write down the quotes, but they say things like, I'm so done. I just want to get out of here. This is horrible. I don't even know my partner anymore. Like it is acute psychological distress that is being inflicted on these people. Yes. Many years of, of couples counseling will be required to recover. Absolutely. And by the way, if you're asking, is there a prize on the show? There is a prize. And the prize is that over the course of the season, they will be shown their partners cheating on them many times, and they have to guess which of those were real, if any, and which were fake.

1:02:17And the couple that makes the fewest mistakes about what is real and what is deepfake gets 100 ,000 euros. See, this is what is confusing to me about all of these reality shows. You could just sit by the pool for a month, sipping Mai Tais while everyone else does these sexy party games and win the prize money. But no No one seems to be interested in that strategy. No, well, and it is a little bit of prisoner's dilemma because you don't actually know what your partner is doing and the premise hasn't been disclosed to them in advance. So I think, you know, if there were a season two of this show, which, you know, there shouldn't be because these people should be tried for war crimes, but if there was a season two, you could sort of say, hey, I'm just going to go sip a Mai Tai by the pool for the entire time, so you're not going to catch me cheating.

1:02:57But, you know, if the deep fakes are good enough, maybe you would start to have your doubts eventually. Right. So Havoc does get wreaked. It is, as you would expect, a very libertine environment. Why did I say libertine? Because you are 400 years old and you are the person walking around at the high school dance trying to separate the couples, saying, hey, you guys are standing a little too close together. Leave room for the Holy Spirit. There's not a lot of room for the Holy Spirit in this show. Jesus is nowhere near this show. God does not exist in the universe of deep fake love. So, you know, predictably, many poor decisions are made.

1:03:37And then they have these sort of big reveals where they sit people down and they say, OK, here's a video of your partner cheating on you. Is this real or is this a deep fake? And I have to say, I have been scared of deep fakes for a long time. When this technology came out, I would say the worry was that they would be used not only in like political misinformation by people trying to like implicate their opponents in stuff that they didn't do. but also in revenge porn, right? A lot of the early uses of this technology were people sort of superimposing, you know, female celebrities onto the bodies of adult film actresses and things like that.

1:04:13Just really, like, gross stuff. And so I think that's how a lot of people kind of expected deepfakes to sort of enter our culture in a major way. And now it seems like it's just kind of weirder than anyone thought. It's actually, like, becoming a plot device on dating reality shows. That's the thing about this that really tripped me up is somebody who has thought about deep fakes almost exclusively in the context of how they can harm public discourse, how they can be used to harass people, in particular women. And yet here you have it as the premise of a reality dating show was just truly something I did not see coming.

1:04:53and I wonder what we make of the fact that this is being presented as entertainment. I mean, I have to say, I didn't really experience this show as entertainment so much as a kind of comic horror movie, right? Where it's like you watch it, you almost can't imagine anything worse being done to you on a reality show, or at least I couldn't. But then it's all just kind of like, hey, it's Netflix, baby. Everybody having a good time? I don't know. Right, I think it's very... interesting the way that these technologies are starting to sort of seep out into pop culture and to public consciousness, maybe not in the ways that people expected.

1:05:34You know, we still haven't had like the big, you know, headline cataclysmic political misinformation event that everyone was fearing back then. You know, we've had instances of deep fakes being used in politics, things like, you know, the Ron DeSantis campaign video that showed Donald Trump hugging Anthony Fauci. But those are still images. That's not video. It still seems like the technology for deepfake video is still not quite good enough to be mainstream. But I wonder if we're just going to start to see this stuff seep out into pop culture through things like this Netflix dating show, rather than the kind of big national or international incident in politics or warfare or other serious matters that we've been maybe concerned about.

1:06:15Well, you know, I have to say, Kevin, I think that this show is actually really insidious and bad. And I don't just mean bad in the sense of not fun to watch, although most of the time it's not fun to watch. I actually think it's bad to train people to disbelieve their own eyes, which is kind of what's happening here. The premise of this show is, hey, you're going to start to see videos in the world, and they might be of you or friends or loved ones, and what you're seeing isn't real. And you should sort of question everything. And that leads us into a world where everything is possible and nothing is true, which is a statement that people sometimes ascribe to Putin's Russia, right?

1:06:55Where there is sort of so much disinformation flowing from the authoritarian government at all times that nobody knows what to believe. So I don't want to be too dramatic and say that deep fake love needs to be eliminated from Netflix. But I do think that this is just bad territory that we're entering into. Oh, see, I could sort of make the opposite case because I think that we are sort of entering a new world as far as like not being able to trust the visual evidence presented to you for any given situation, right? You know, for a long time, we lived in a world where there was this slogan like, pics or it didn't happen, right?

1:07:34If you can show me a picture of something happening, I will believe you. Otherwise, I am skeptical. And then pictures became manipulable through very conventional photo editing techniques. So then it was like people needed that extra layer of skepticism where it's like, even if you show me a picture of something happening, there's a chance that it could still be fake. Now the same thing is happening with video. For a long time, it was like, oh, well, if there's video of something happening, then it definitely happened. Now we are leaving that world because now this technology is becoming democratized, is getting built into products that people are able to use without a ton of technical expertise.

1:08:12And so maybe it is actually a good thing if people learn to be a little wearier of the video evidence that is presented to them. I mean, I think that, yes, it's definitely true that people should be wary, but I also think that there is a sense of loss that I have around that, right? It was great when you could show me a picture of something and I could assume that it happened. It is great when today, for the most part, or if you show me a video of something, I'm not going to assume that it was made by AI. But we're just sort of clearly not in that world anymore. Totally. And I think for every person who watches this Netflix show, it's possible to see how all of this could become quite dark.

1:08:48It's not just shows on Netflix where people are generating footage of their partners, you know, cheating on them. But I think this probably will be used in actual people's like real relationships, where it's like, you know, this person really wronged me. So here's a video of them doing something horrible. Oh, yeah. You know, it's like, you know, it'd be a really terrible use of this technology, a custody battle, right? Yeah. So I think we will start to see all kinds of malevolent uses of this technology. For right now, it seems like we're still trying to sort of adapt as a society to sort of figure out like, okay, what is real and what is not real?

1:09:24And what can we trust with our own eyes? And what can we not anymore? And it seems like that's going to take a while. Yeah. But in conclusion, deep fake love is one of the worst things I've ever seen. I can't believe you made me watch this stupid show. I think we had a really productive conversation about it. And I've decided that we're renaming the place where we interview our guests, The Chair of Truth. I love that.

1:10:24Hey, before we go, I wanted to thank everyone who heard my offhand joke last week and actually went and planted a tree. This was amazing. You made like an offhand joke in our mailbag episode about offsetting the carbon cost of listening to Hard Fork by planting a tree. And we got several emails from listeners who actually went out and planted trees after listening to that episode. Absolutely amazing. We continue to love our listeners. And please keep planting more trees and let us know when you do. It makes me think we should actually like up the stakes a little bit, like in our next throwaway joke.

1:10:57Like if you're listening to this podcast, go cure cancer. Like invent a pizza with no calories. Yeah, that would be wonderful. Send us the proof to hardfork at nytimes.com. There's so many things that our listeners could do, and we would love them to do them. Hard Fork is produced by Rachel Cohn and Davis Land. We're edited by Jen Poyant. This episode was fact-checked by Caitlin Love. Today's show was engineered by Chris Wood. Original music by Dan Powell, Alicia Butetoup, Marion Lozano, and Rowan Nemisto. Special thanks to Paula Schumann, Hui Wing Tam, Nalga Logli, Kate Lopresti, and Jeffrey Miranda.

1:11:38You can email us at hardfork at nytimes.com. And if you have a deepfake of yourself shooting on your partner, keep it to yourself. Or send it to Netflix. There you go. They're casting season two. Oh, here we go. Thank you.

1:12:23Thank you.