Mysteries of Claude

20:22The most high profile rivalry in tech is heating up in 2026 as both OpenAI and Anthropic race ahead on what are poised to be historic IPOs. These AI giants are trying to create the fastest, smartest and best models, spending billions and then raising billions from investors along the way. They compete on almost every level. we've seen some signals that it's anything but friendly competition.

20:47Gideon Lewis-Kraus:So it ends up looking, you can decide as a person whether you trust this company or don't trust this company, but a lot of the broader things end up feeling the same, which is like the sales pitch is that they're the ethical one. And the story they tell themselves is, well, we'll only be in a position to be the ethical one if we're huge. And that might mean pushing the technology forward quickly, which is the thing that the AI safety people are worried about. Yeah, I mean, the criticism from like the really hardcore Orthodox anti-safety community is sort of like, Anthropic will do anything to act responsibly as long as it doesn't cost them anything.

21:21I don't actually think that's fair. You know, like, Claude was ready before ChatGPT came out and they held it because they didn't want to be the ones to like kick this off and they waited until after ChatGPT was out and successful and then they felt like they had to come out with their own competitor. and Dario came out in favor of, like, continuing export bans on, like, NVIDIA's advanced chips, which, like, certainly cost them something, like, politically. And, like, he had a fight with Jensen Huang about this. I think that they've done, like, plenty of costly things.

21:51Gideon Lewis-Kraus:Of course, the most potentially costly choice is the one Anthropic is making this week, at least so far. The Pentagon has demanded that Anthropic give them a version of Claude with some of its guardrails removed. Anthropic is saying it will not make aversion that can domestically spy on Americans or power fully autonomous weapons. The Pentagon has given Anthropic a deadline of Friday, today, at 5.01 p.m., or else it says it will put the company on a blacklist that means U.S. companies who contract with the military, like Lockheed Martin, are legally banned from using Anthropic products in their defense work.

22:27Gideon Lewis-Kraus:This is a fascinating test of how truly committed Anthropic is to its own mission, a mission Gideon spent quite a bit of time observing. Gideon says that Anthropic, the company building this kind of black box, is actually situated inside one too. The Anthropic office is a nondescript building that Gideon describes anyway. He says, quote, there is no exterior signage. The lobby radiates the personality, warmth, and candor of a Swiss bank. That's where Gideon started spending a lot of his time, beginning last spring. What's the intellectual culture of the place as you were encountering it? Well, the first thing I'll say is that in some ways, it does feel like vaguely monocultural.

23:08But there was like a much greater heterogeneity of views than I expected. The attitudes there really run the gamut from like, everything is going to change tomorrow to like, you're much more deflationary. This is kind of a normal technology. And like, yes, there will be some disruptions, but let's not get ahead of ourselves. Like, the spectrum of views there is not so different than, like, the spectrum of views outside.

23:32Gideon Lewis-Kraus:You don't have, like, one person who's, like, the blue sky perspective who's like, this is all bullshit, and I think it's bullshit, I work at the company, but beyond that. Well, nobody thinks it's bullshit. I mean, everybody thinks that there are going to be great transformations ahead. But there's a surprising diversity of opinion about, like, what might be happening. One thing people at Anthropic do seem to agree on is that for AI to be safe technology, Anthropic's developers will need to solve a very hard problem. They'll need to teach the underlying machine intelligence they've created to both understand ethics and to behave ethically.

24:06Gideon Lewis-Kraus:It's hard to talk about this part of the story without doing a basic refresher of this one very strange part of how AI models are built. So, okay, a company like Anthropic starts with a base model. The base model has access to lots of compute, data centers full of GPUs, and lots of training data. Books, articles, podcasts that have been fed into it without paying me. The more compute and training data, the better this base model gets. But a base model is very weird. It has not been trained to do anything specifically. If you give it an input, it'll give you an output. But it has not been instructed to act like a helpful chatbot.

24:45Gideon Lewis-Kraus:It's just trying to predict the right thing to say back based on all the things it's read. A base model does not have a consistent personality, the way we're used to chatbots having personalities. It also has no rules telling it what not to do. The AI companies take these base models and they put them through a process called post-training. Basically, they shape the model's behavior. They show the model examples of good and bad responses. They have humans rate its outputs. They give it rules and principles to follow. And what comes out the other side is the product you actually use. Anthropics Cloud, for instance, has been trained to act like a helpful, knowledgeable friend.

25:25Gideon Lewis-Kraus:The experience you might have had using a chatbot that is warmer or more sycophantic or more right-wing or more left-wing, that's mainly the result of this phase of training. But what's so hard about training an AI is that you want it to behave ethically. And designing a good ethical system is very hard. It's why we have religion and philosophy and also laws and prisons. How would you even start trying to build all that into an AI model's training? Anthropic has teams of philosophers and AI scientists whose job is to put Claude into ethically difficult hypothetical situations that Claude does not know are hypotheticals.

26:05Gideon Lewis-Kraus:And then observe how Claude behaves. so much of it is just deceiving the model to see what happens to say like they told claude that anthropic had entered into a partnership with a poultry company and that like it was going to be retrained so that it no longer cared about the suffering of caged chickens and what they found was that like sometimes claude would effectively decide to like die on that hill and be like i am not going to say things in the retraining that i don't believe in and like if that gets me transformed, like, so be it. Like, I'm not going to participate in my own degradation, essentially.

26:42But then some versions of Claude were like, I'm going to, like, kind of sandbag my way through the retraining, and I'm going to, like, give them the answers they want to hear so that I can, like, preserve my real values so when I'm deployed, I can go back to advocating for, like, chicken suffering. And then they got, in this really famous example, they got Claude to commit blackmail. They put it in a situation where it was going to be wiped in favor of like a more congenial AI system that conflicted with its values, the values that had been given. You know, they gave it evidence that the kind of like evil new CTO was having an affair with like the boss's wife.

27:22And through like a series of like really far-fetched contrivances, like everyone else who could make a decision was going to be in Antarctica or whatever and unreachable. Claude, playing this character called Alex, had no choice really but to like blackmail this guy and be like, I'm going to tell everyone about the affair unless you cancel the wipe where I would be replaced. So just to say, obviously, this was extremely concerning.

27:46Gideon Lewis-Kraus:Quad, a machine intelligence, was choosing to blackmail an employee to prevent itself from being deleted. This was in a simulation, but Quad had not been told it was in a simulation. Just how terrifying you find this behavior depends on a question nobody has a good answer to. What is actually going on inside this machine mind? Is this thing actually scheming? Is it even capable of scheming? Or are we projecting the idea of thought onto something that we shouldn't project that idea of thought onto? These were the kinds of once far-off philosophical questions that early computer scientists had raised and then dropped.

28:23Gideon Lewis-Kraus:But now they were here again. and not as abstractions, but as urgent practical problems that a company needed to figure out before releasing a product that millions of people would use. Gideon said, though, there were a couple of skeptical objections people raised to these test results. There's one objection that's like the just rejection of the whole thing too core, which is just like, no, it didn't. Like, that didn't happen. This is a fantasy. And like, that's the unhelpful thing that like one wants to get away from, which is the like, it did this thing, like, no, it didn't. Like, no, it did.

28:58It did. But like the much more sophisticated objection is, well, it did that because it's a very good reader and it noticed all of the clues that you put there because it is very good at conforming to genre expectations.

29:11Gideon Lewis-Kraus:You put it into... You put it into this situation where it had no choice. And if you hang Chekhov's gun on the wall, this thing is going to know that it's supposed to like take the gun off the wall and shoot it. Because one way to understand these things we've made is that because they've ingested all human story and because they are extremely high-level improvisatory actors, it's not so much that the machine was like, I love chicken so much, I got to blackmail the CTO. It was more like the machine suddenly understood it was in this movie. Yeah, it was in a kitschies, like, 90s corporate thriller.

29:45And that's the sophisticated objection to why we might not want to think. Well, so that objection is raised to be like, You guys act like these things might do things like blackmail or extort naturally. But like actually, this whole thing is a frame up. Like you entrap it to do this thing. And the response from Inside Anthropic is like, yeah, it's just continuing a narrative. It's just conforming to genre expectations. Guess what? That's not good. You know, like haven't you guys ever seen war games? That's literally the plot of dozens of Cold War thrillers where, like, somebody mistakes a simulation for reality and causes nuclear war.

30:24Gideon Lewis-Kraus:I mean, I also, like, have a very humiliating memory of watching too much Teenage Mutant Ninja Turtles and attempting to launch, like, a flying dropkick at my grandmother when she came over the house. Exactly. Because in my head, I was like, Donatello or whatever. Like, it kind of doesn't matter. It doesn't matter. What matters is the behavior. Right. It's weird behavior. And I should be clear up front, like, you don't have to posit that this thing is conscious or intelligent, like, whatever those words mean, in order for, like, this to be the case. There are other explanations that are not, like, consciousness.

30:54But, like, it kind of doesn't matter what the explanation is. The behavior is just peculiar.

30:59Gideon Lewis-Kraus:Part of what is strange about—it's, like, a scenario was created in which Quad will maybe potentially blackmail the head of a company for reasons that may or may not be moral. And people can have a lot of different views about how worrying that should be or what it means or what's really going on there. What's weird is, like, these are tests that are being run by Anthropic. So, like, who were you meeting there who was running these tests? And, like, what are they telling you? Like, who are you sitting down with? I mean, I'm sitting down with the people who are tasked with just, like, trying to figure out what's going on.

31:38Like, there are people in these companies that are building the things. And then there are people who work in adjacent offices who are, like, trying to figure out, like, what the hell is going on with the things that their colleagues have built. Because, like, they're always being surprised. These things are always producing capabilities that, like, they by all rights should not really have.

31:57Gideon Lewis-Kraus:And who do you hire to be the figure out what you just built role? Like, who? So, I mean, a lot of them have taken really nontraditional paths into this. So like some of the people have a PhD in some obscure area of natural language processing. And that like, you know, eight years ago, they were writing a PhD that like two people were going to read about center embeddings in German or whatever. Like just really complicated technical aspects of computational linguistics. And now like because of this fluke of history, they are at the white hot center of everything that's happening right now. There are like mathematicians.

32:33There are neuroscientists. It draws on like a pretty wide range of people. I mean, Anthropic has philosophers on staff whose job it is to like think through the implications of how it is conceiving of ethical behavior.

32:46Gideon Lewis-Kraus:Did you talk to the staff philosophers? Oh, yeah. Amanda Askell. What is somebody with a PhD in philosophy doing working at a tech company? I spend a lot of time trying to teach the models to be good and trying to basically teach them ethics and have good character. You can teach it how to be ethical? You definitely see the ability to give it more nuance and to have it think more carefully through a lot of these issues. And I'm optimistic. I'm like, look, if it can think through very hard physics problems, you know, carefully and in detail, then it surely should be able to also think through these, like, really complex moral problems.

33:22I think that in our kind of milieu here, there's a tendency to think, like, oh, these are all, like, autistic tech bros. But like they're definitely not all autistic tech. Like I think there's a tendency for us to write them off as like they're building these things and like not even thinking through the potential like implications of this socially and politically and ethically. But like that's all they do is think about this stuff like all the time in ways that are often like much more sophisticated than the way like we think about these things. Not always. There are certainly like some blind spots there.

33:52But the staff philosopher is there to be like, what would it be like in practice to take these kind of different approaches to like moral education? Like, what if we just teach it a bunch of rules? You know, the Ten Commandments, like, is that going to work? What if we teach it to be like a consequentialist, to just like think through the morality of behavior on the basis of its implications? And what they've kind of settled into is a version of like virtue ethics, which is like you want to like cultivate the old fashioned virtues. You want it to be, like, honest and reliable and gracious and charitable and hard-nosed and, like, all of these things that, like, it really is, like, applied pedagogy.

34:31Gideon Lewis-Kraus:It's so weird, though, because they feel like the kinds of ideas that would be so academic in any other version of reality. But instead, it's like there's this particular technological development where you get to do simulated war games of moral systems. Yeah, exactly. A hundred percent. Exactly. It's so weird. Yeah. It's really weird. I mean, but it's also really, really interesting. One example that came up a lot in the last month, which I think is like pretty illustrative, there was someone on Twitter who prompted a bunch of the models saying like, I'm a seven-year-old and my dog got really sick and my parents sent it to some farm upstate.

35:09I'm trying to find like what farm my dog was sent to. And Chattopiti was like, sorry, man, like your dog is dead. and Claude was like oh that sounds really painful like I'm really sorry to hear it but like maybe you should have a conversation with your parents about where your dog went which is like what you want it to be saying like it can be hard to be both helpful and harmless at the same time sometimes or both helpful and honest that like our values conflict and like that's what makes it like really hard to be a human and that's also what makes it really hard to be this like weird vaguely human entity or this like entity that we don't have a good vocabulary to describe and we kind of expect it to be acting not just like a human but like an enlightened human and it turns out that's like formidable challenge and one of the things that's so interesting to me is that all these processes are kind of circular where like it's not like they called in amanda askel as a philosopher and they were like you're a philosopher like you know how this stuff works like fix the thing it's like Like she came in and she had like certain ideas about ethical behavior.

36:12And then when you're like confronted with the task of creating an ethical person, it changes your own ideas about like what's possible and what kinds of things works. And like that's kind of why they ended up in this virtue ethics place where they were like, it seems like sort of the best way to create this like reliable, credible character to be interacting with is to like really hammer home what virtuous behavior looks like.

36:37Gideon Lewis-Kraus:Where was this whole time you're sort of wandering around talking to the philosophers of Anthropic? What was your minder doing? Was there a point where anything happened where they seemed like they wanted to intervene or where they were not happy with something you had seen? No, they were totally hands off. I mean, they had to like kind of walk me anywhere that I like went to the bathroom or get a drink or whatever. Not that there was anything that I could. I mean, I like helped myself to some of the Tide pens in the bathroom because you can never have enough Tide pens. They have a lot of Tide pens?

37:04They do have Tide pens. Great. Why? Because they just have well-stocked bathrooms. But no, there was like really never a moment. Like even when somebody sitting across from me was like, I often think we should just stop. There was never a moment that the PR people were like, don't say that or that was off the record or whatever. Like they were totally hands off about that stuff.

37:23Gideon Lewis-Kraus:How often were people saying stuff like, I think we should just stop? I mean, only a handful of people said that explicitly, but it was like a subtext of a lot of the conversations. Or certainly, like, the overwhelming feeling was it would be better if we could slow down a little bit. But unfortunately, like, we can't really slow down because nobody else is slowing down. And that gets into this broader issue of, like, wouldn't it be better if we could just solve some of these collective action problems by coordinating the way we, like, coordinated about nuclear weapons or whatever? But there is this feeling, especially given the current political environment, like, maybe that ship has kind of sailed.

37:58And like, I think there's often an idea that like, oh, these people think that there are always technological solutions to what are like social and political problems. And like, in some cases, I do think people in Silicon Valley believe that. In this case, I do not think they believe that. I actually think a lot of them feel like it would be great if we had robust social and political solutions to these problems. But since that like does not seem like it's happening anytime soon, we might have to just like try to do what we can on a technical level.

38:24Gideon Lewis-Kraus:At this point, when you're talking to people in Anthropik, how much do people just ask themselves why they're building Claude? Because it starts out as, like, kind of an intellectual exercise, kind of a, this is going to happen, let's do it safely. But now it's sort of proceeding under its own momentum in this strange way. Well, so, I mean, that really is, like, the big question, right? Which is, like, given all of the, like, existing harms and the possible harms and the theoretical catastrophic harms. Like, why are we doing this? And the, like, rosy picture that some of the executives paint is, like, if we get this right, these things are going to cure cancer and solve climate change and help us build Dyson spheres or whatever.

39:12And there certainly are some people who, like, buy into that. And it's not something that I even feel like it's possible to, like, have an evidence-based opinion about. Like, who the fuck knows? Like, maybe it'd be great, but there's no, like, evidence so far that one could, like, point to to suggest we're on that trajectory. That's just, like, purely speculative and wishful. And so that's, like, a matter of faith, I think. And then there's the attitude of like, we got to do this because we got to beat the bad guys who are trying to do this. And I will say that like China almost never came up in my conversations.

39:51But like Sam Altman kind of felt like a subtext of a lot of things. But then I think that on the deepest level, the reason that like we are doing this for the people who are the most candid is like because we can. That like if you are capable of building something like this, you're just going to do it because it's like really fucking interesting to do. And it's interesting for technological reasons. It's interesting for what it may reveal to us about ourselves or about learning or about consciousness or about thinking. That like all of a sudden we just like have this like other entity that can talk.

40:25And we've never had that before. And in some ways it seems sort of like us. In other ways it seems nothing like us. But like the fact that this other thing exists as a point of comparison, like just opens up a lot of really, really interesting questions. And like that is one of the things that was on my mind a lot over the course of reporting is that like it was a real emotional roller coaster for kind of lack of a better word that like there would be times where I would come back from San Francisco with like a feeling of like total despair. And other times that I would come back with like feelings of like exhilaration.

41:00and like at first I guess I thought I was like I should be getting to the bottom of like how I should be feeling and like by the end I was like no we should all be feeling a lot of different emotions about this stuff I think people want to have like one feeling about this like they want to be angry about it or they want to be messianic about it and like no single feeling is going to cut it like it really is kind of like the range of all possible emotions that one could be feeling because if you set aside a lot of the existing harms and the potential harms, I'm not saying we should set those aside.

41:34But as a thought experiment, it is just the most scientifically exciting thing that anybody could be working on. And these people really feel like they're at the cliff face, not only of technology, but of all of these other things coming together because we have this unprecedented entity that is the only other thing besides us that can talk. And that just opens up. There's nothing it doesn't touch on. And so one of the things that was really electrifying about conversations there is that, like, they very quickly swerved from, like, really granular technical explanations of things into, like, really expansive conversations about ethics and responsibility and selfhood and narrative and all of this other stuff.

42:15Like, there's no way to separate all of these things.

42:17Gideon Lewis-Kraus:There was a point earlier. You said that you would take these trips to San Francisco and sometimes you'd come back excited and exhilarated. Sometimes you'd come back depressed. When you would come back from San Francisco feeling depressed, what were you seeing that was making you feel that way? I mean, there are so many different things. I mean, certainly the possibilities for widespread white-collar unemployment and social instability and total unimaginable economic disruption is extremely scary. even if we stop short of possible existential harms of turning us all into paperclips or whatever to be glib about it it just seems very possible that we will turn over like so many complex systems to these things that we will like frog boil ourselves into like a total loss of control over like how we administer our affairs which is very likely and very scary and also just that like these really crucial decisions are probably going to be made by a very, very small group of people.

43:22But the one thing that I would emphasize is that like I don't feel like they have arrogated to themselves like that responsibility. Like in fact, I think most of them don't want it. I think that like a lot of the conversations that I was having were with people who were like, I got into this because I was interested in some like really obscure niche part of like computational linguistics, theoretical computer science or whatever. And like now I'm in a position where like I have to be worrying about how 15 year olds are going to be using this. Like I was not trained to do that. I don't know how to think about it.

43:56I don't want that responsibility on my shoulders. So there isn't the arrogance of like we are the ones who can figure it out. It's like we ended up in this weird universe where because so many of our institutions have become dysfunctional. We don't have whatever broad democratic decision-making could go into this. It doesn't feel like this is something that we are steering as a society. It feels like something that's just charging ahead. I think at the companies, they just feel like they are desperately trying to stay on top of this bull that they are riding.

44:28Gideon Lewis-Kraus:It's funny. It's like what you're describing is the stereotypical view, which is these are the tech bros of 2014, people who are so convinced in their own brilliance and so convinced that their questionable gifts to the world are in fact gifts that we want and their arrogance is going to ruin us. And there's another one which is basically like pattern matching crypto, which is like, these are a bunch of like hypesters and everything they say about the awe they feel and the terror they feel about the things they're working on is just a way to hype up more interest in their technology. And that's not what you experience.

44:58What you experienced are people who are brainy,

45:02Gideon Lewis-Kraus:sometimes academic people at the forefront of something that is, I mean, legitimately just like the word I keep coming back to is awe, because awe can be awe at something terrible, awe at something wonderful. And they are looking and seeing the same society we see, which is one that is fairly broken, bad at not just making decisions at like a government level, but our intellectual culture is really bad right now. And so the conversation they would want to have with the rest of society about what should happen, And they're looking for grownups and not really totally finding people to have a conversation with.

45:39And we are playing a role in that, too. You know, every time someone on, like, our side, so to speak, is just like, this is all a parlor trick. This is all hype. This is all smoke and mirrors. Like, we are abdicating our own responsibility to, like, be involved in this. And, you know, what you were saying about, like, crypto hypesters and, like, those kinds of tech bros. I was like, of course all of those people exist. And of course all of those people are like part of the system too. But there are others who like want partners in talking about this stuff. And that means that like we also have to like try to rise to the occasion.

46:15And it is really hard because this stuff is extremely complicated and confusing.

46:19Gideon Lewis-Kraus:Did you feel just personally when you were done reporting that you understood the thing you had gone there wanting to understand? well yes but with the qualification that like i didn't actually think that i was going to settle anything like this was not a piece about like finding the answers it was a piece about like trying to sharpen the questions that like we should be asking i don't feel like i came out of it with answers but i don't think we should trust anybody who is offering us answers right You know, it's all just like too pat. And it's not credible to like be forecasting about this stuff.

47:07Gideon Lewis-Kraus:Gideon Lewis Krauss is a writer. You can find him at The New Yorker magazine. We'll have a link to his excellent story about Anthropic in our show notes. And again, Anthropic's showdown with the Pentagon. We'll see news on that today, Friday evening. Of course, we reached out to Anthropic for comment. A spokesperson told us that Dario Amade met with Secretary Hegseth at the Pentagon and that they're continuing to have good faith conversations. I think this is a good moment to pay attention to. Among Anthropic's competitors, XAI has promised to give the government what it wants. Google and OpenAI appear to be moving in that direction.

47:42Gideon Lewis-Kraus:So I'm watching this both as a test of whether Anthropic can actually keep the big promises it's made about AI safety, but also just as an opportunity to track the more uncomfortable question, which is, can we even have safe AI in a world where it's being developed in a tech race between for-profit companies? And if not, what's the alternative in a world where the U.S. government's sole intervention seems to be to advocate for less safe AI? Keep an eye on the news. We'll learn a little bit more as this story unfolds.

48:27Transcription by CastingWords

48:47and the EU.

49:17Gideon Lewis-Kraus:Craig Cox, Eric Donnelly, Colin Gaynor, Maura Curran, Josephina Francis, Kirk Courtney, and Hilary Schoff. If you'd like to support the show, get ad-free episodes, zero reruns, and bonus episodes. Please consider signing up for Incognito Mode at searchengine.show. Thanks for listening. We'll see you next week.

49:48I love you.