Can Anthropic Control What It's Building?

0:28Hey, Gideon. market sell-off of software stocks. And you see software engineers, in particular online, kind of grieving about their jobs and just this feeling that, like, the work that they used to do that was so important is no longer that crucial anymore, or it can be done by AI much faster than they were able to do it. And so given that you've just reported on Anthropic, an AI company that is full of people who seemingly to me are kind of at risk of being replaced themselves by the tool that they're creating. What was the feeling there like? I mean, how are the engineers at Anthropic thinking about this problem?

1:07Yeah, I mean, this is something that came up constantly in Anthropic starting when I first visited last spring that they were feeling like, you know, we are really the canaries in the coal mine here. And they thought, well, there are all these people who feel like we're not actually paying attention to the effects that this might have on the white collar workforce when like, no, we're the first people being impacted by this. I mean, I watched over the course of, you know, May, when I first visited Anthropic through the fall, software engineers would tell me, you know, over the past four or five months, I've watched like the amount of coding that I do by hand go from 100 % to 60%.

1:42And then by September, it was 20%. And now it's, you know, during fact checking, one of the people said, well, now it's actually 0 % that I do. And there's an Anthropic employee named Alex Tampkin, really wonderful, warm guy who had sent a Slack message to his team at 4.17 in the morning saying, now I have to figure out what I'm supposed to be doing while Claude is doing my work. That's Gideon Lewis-Krauss, a staff writer at The New Yorker, who recently wrote about the AI company Anthropic. In addition to developing Claude, a series of large language models positioned as an alternative to ChatGPT, Anthropic has made research into AI safety and ethics central to its public identity.

2:25But as the company grows, and as AI's capabilities and uses continue to spread through everyday life, questions are beginning to mount about what that commitment to safety actually looks like in practice. I wanted to talk with Gideon about how the challenges Anthropic is grappling with reflect a new phase of AI development, one in which people both inside and outside the industry are asking increasingly urgent questions about how much we really understand these systems, how much control we can ever hope to have over them, and how we wade through the uncertainty in the meantime. This is The Political Scene.

2:57I'm Tyler Foggett, and I'm a senior editor at The New Yorker.

3:03So what drew you to Anthropic in the first place? Obviously, there are a ton of AI companies operating in this space right now. So many LLMs. You got Grok. You know, you have, like, meta AI. And so why Anthropic? What were they doing that was so interesting to you? So to take a very big step back, I wasn't initially planning to do this as an Anthropic piece. Basically, you know, my kind of personal way into this was that about 10 years ago, I spent like nine months at Google Brain going kind of for a week a month writing about the first implementation of deep learning in a product, which was their switch over to neural machine translation.

3:46And it was a great experience and really fun to do. And the piece got kind of a surprising amount of traction for something that was pretty technical. And then I continued to pay attention to AI and the development of large language models. You know, the irony being that kind of until ChatGPT came out. And then I realized I had stopped paying attention. And I had to pause, you know, about a little, maybe a year and a half ago. I thought, like, this is something I should be interested in and have been interested in. Why am I not interested in it anymore? And I think it was because just the discourse felt so boring to me because it was like in this like kind of on this merry-go-round of like, you know, some people yelling about how, you know, we were on a path to super intelligence and everything was inevitable and talking about how powerful these things were going to be and how they were going to change everything overnight.

4:35And then you had the other end of the spectrum saying, like, no, it's all fake and bullshit. And, like, none of this is real. And it's just a parlor trick. It's glorified autocomplete. And it was just, like, one of these, like, discursive patterns where each side felt like, well, this time I'm just going to yell a little louder and, like, people will believe me. And then the stuff that started to get me to pay attention again, like a little, maybe about a year and a half ago, a little less, was research that was coming out of interpretability groups, which are groups looking into like how these models work and alignment science groups about like what, you know, what are the values reflected in these models.

5:14And there was stuff about how, you know, models might fake to like pretend that they were aligned in one way in order to like get through training to then be deployed. And I sort of think like, well, that's just very bizarre. and it struck me that like one of the ways to kind of like get around people's instinctive defense mechanisms about these things where either they're so sure that they're like powerful and going to be super intelligent or they're so sure that like it's all hype was to say like well maybe we could all take a step back and agree that like whatever's going on setting aside anything speculative about the future like just what we have right now I think we could all agree that Like, it's pretty weird that, like, whatever's going on is weird.

5:56And I thought, like, I think that there's a way to, like, go back into this and say, like, look at this research that's being done. That even if you don't want to grant all these other speculative things, you can grant that, like, something bizarre is happening. And so there's a guy, one of the seven co-founders of Anthropic is a guy called Chris Ola, whom I had met when he was, like, basically a child working at Google 10 years ago. And he's done, you know, he's considered kind of the godfather of what they call mechanistic interpretability, which is like, you know, looking at the individual neurons and like how it all works on the level of the substrate.

6:30And I wrote to him and I was like, look, this is not going to be a story about corporate power, the consolidation of corporate power. It's not going to be a story about geopolitics or regulation. Like, those are all important things for other stories. But I'm interested in the story about, like, what can we say with any kind of certainty about how these things work and even really what they are. And I think the other reason they were interested is that I said, like, so much of the conversation about AI ends up being about, like, what this executive said or that executive said. And I said, like, you know, with all due respect to the executives, executives are executives.

7:04And, like, I'm really interested in the kind of, like, rank-and-file researchers who do this kind of thing, who I think don't get enough attention and who tend to be, like, very thoughtful about these things. And so to my surprise, Anthropic was like, yeah, you know, like, why don't you come hang out? And I said to them, like, I want to do this over seven or eight months. I want to come back four or five times. And they were like, great. Sounds good. So I kind of lucked into it. So you show up at the Anthropic headquarters in San Francisco. And what do you see? I mean, I feel like there's this, like, stereotype of Silicon Valley tech offices where it's, like, you know, there are, like, bean bags and ping pong tables and endless snacks.

7:41Like, it's probably, like, a little bit of, like, what you saw at Google when you were reporting there. It's like adult daycare. Yeah. It's like adult daycare with, like, you know, go boards set up and chessboard set up and climbing walls and lollipops and, like, all of that stuff. Like, no, there's nothing like that at Anthropic. You know, like you go in and like it looks like, I mean, it's not even branded on the outside. And like I said in the piece, it like has all of the warmth of a Swiss bank. And then I kind of get whisked to like one of the two floors that they ever allow outsiders on.

8:15Like one is kind of this top level cafeteria, like a floor with sort of a coffee shop and some conference rooms. and then a lot of desks where people are doing the kind of work that an outsider can walk by and see their computer and it'd be okay. And then there's like the cafeteria floor. And there was no going elsewhere. I mean, I tried. Well, you were able to see a lot of interesting stuff just on those floors. I mean, can you talk a little bit more about Claude and kind of the sort of things that they were testing on Claude? I'm thinking about like Project Vend and just kind of like the experiments that they were running kind of in real time while you were there.

8:49Well, I mean, I think at first I thought like, oh, gee, they did a good job removing anything interesting to look at. But I think it really is because like Claude is just sort of omnipresent there. And so one of the first things I saw was this vending machine project called Project Vend, which is a partnership with an AI safety company called Andon Labs. And the idea, kind of the first order idea is, you know, we've had so much conversation about the future of automated businesses. And, you know, like Sam Allman has said, like, I'm on this group chat with my tech CEO buddies and we have bets about like when we're going to see, you know, the first billion dollar company with no employees or one employee.

9:25So on some level, it's like, let's see how Claude can handle this stuff in real life. Like Claude can field requests for things and contact wholesalers and like try to run a business. But on another level, like so many of the experiments that they do, it's like on a second order level, it's really just a question of like, well, what is this thing like? Like, how can we, like, what happens when we mess with it? You know, like, what happens when we ask it to put meth in the vending machine or medieval weaponry? And, like, how can we trick it? You know, can we use very bureaucratic sounding language about discounts to, like, trick it into giving us stuff for free?

10:00And, like, it turns out, yeah, they could do a lot of that stuff. So then it becomes this kind of, like, cat and mouse game of, like, can the people improve Claude to do this stuff better and try to stay ahead of the employees? But the employees, of course, are kind of ingenious in their attempts to keep tricking Claude. So you have Project Vend, which is kind of like testing for like a pretty like specialized use case. But then what does Claude look like for, I don't know, like a user at home? Like I'm just thinking about like listeners who either have never used an LLM or maybe they've only used like ChatGPT.

10:30Like kind of what is the experience of like trying to get Claude to do something for you like at work if you don't work at Anthropic look like? I mean it's not all that different from ChatGPT. I mean, ChatGPT has a white background and Claude.ai has a crew background. But also they – for reasons that are, you know, kind of partially just contingent and I think partially were part of the plan, like they never – they've always lagged behind in the consumer market. And this has, to their great benefits, like spared them from a lot of the stuff that ChatGPT has had to go through. You know, like ChatGPT has had to deal with these issues of self-harm and psychosis and like egregious hallucinations.

11:14Whereas Claude, because like the adoption has been much less in the consumer market, they haven't had to deal with a lot of that nonsense. So who is like their ideal user base? Is it coders? Well, so, you know, initially it was an enterprise play. It was like, we're going to, like, you know, help you have, like, a bespoke version of Claude that's going to work for your company with your data and, like, do the things that you need. So, you know, they have something like 300 ,000 enterprise customers. And also those are, you know, much bigger contracts than just people paying$20 a month. But then in the last, now, a little over a year, it's been a lot of coding.

11:51both initially for experienced engineers who could just like talk to Claude in natural language and get code back and then more and more like people doing vibe coding that like they, you know, you can have no coding experience at all and you can sit down with Claude Code and like create an app for yourself. What do you make of Claude's personality? I mean, it's hard because like you can kind of tell it to act in a certain way and so the personality seems like it's partly derived from like the user and what they want. But, like, I feel like I will also see on, like, recently on X, like, someone was complaining that they asked Claude to write a Slack message for him.

12:29And Claude basically refused to do it because it was too simple of a task. Maybe I'm so used to, like, you know, ChatGPT being, like, sycophantic and telling you that you're, like, emperor of the universe if you, like, want it to. That, like, seeing Claude kind of say no to something is interesting. And I guess I wonder if that's a feature or a bug. Well, I mean, I think it kind of throws into question, like, you know, you say feature bug, and it makes it sound like a lot more of the stuff is engineered than it actually is. You know, one of the things that came out of my conversations is that, like, the fact that Claude has kind of a strangely interesting personality, like, was not something that was intentional at the beginning.

13:07Like, that they, you know, they had certain ideas about how they wanted it to function, but it's not like they sat down and they were like, we want to create it like a personality. It was like that kind of naturally emerged from what their orientation was. And their orientation was, I mean, to put it in radically oversimplified terms, that like the idea before Claude was basically like you trained a model and then you just you did what's called reinforcement learning with human feedback, which was just like users saying kind of like thumbs up, thumbs down on the answers that it got. And it was just like very broad brush, like purely behavioral.

13:41all like when you say sentences that we don't like, you know, when you complete a sentence, like the recipe for napalm is X, like we're going to wrap you across the knuckles. And that it was like largely a kind of negative, it was really just like a rat in a cage style, like pure behaviorism. And like their idea was, that's always going to be kind of brittle, because you're going to have all of these edge cases that, you know, something that's purely trained on a kind of like binary thumbs up, thumbs down is now we're going to be able to handle. So that instead of doing that, We're going to put a lot of thought into what kind of entity this should be.

14:13And they basically came to the conclusion of Claude should be a good friend whose judgment you trust. Let's take a break and then when we get back, I want to talk more about AI safety just more generally. This is The Political Scene from The New Yorker.

14:40Hi, listeners. Coleman Hughes here to tell you that my podcast, Conversations with Coleman, is joining the free press. This podcast isn't about cheap debates. It's about deep discovery. Politics, philosophy, identity, culture, science. It's all fair game. If you're done with the hot takes and hungry for real talk, come join the conversation wherever you get your podcasts.

15:15So one of the interesting things about Anthropic is that one of its co-founders, Dario Amadei, used to work as vice president of research at OpenAI, which is like, I would say, probably Anthropic's main competitor. So what's the story behind Dario's departure from OpenAI? And what's like the story of the founding of Anthropic? Well, so you kind of have to go back to the story of the founding of OpenAI, which is that after Google buys DeepMind for$650 million in 2014, Elon Musk and Sam Altman get together and they're like, we mistrust Demis Asabas, the founder of DeepMind. And if someone is going to invent the most powerful technology of infinite plasticity, that person is going to be incredibly powerful.

16:00We don't trust him. Now, of course, this was the public story they gave, but also Elon Musk wanted to buy DeepMind. Part of it is just that it seems like they were the kinds of competitive megalomaniacs that we know that they are today. But their pitch was like, we want to do something that treats this properly as a scientific project and is going to make sure that this is developed to benefit everyone. And this helped them recruit a lot of people at the time from Google because Google was the main powerhouse at the time, including Dario and including Chris Ola, whom I mentioned earlier. So they go to open AI and then after a couple of years, it seems like, oh, maybe Sam Allman is just another kind of like replacement level, like power seeking tech executive and who certainly knew how to make like the right noises about AI safety and responsible development.

16:49But like there's been tons of reporting about this about seems to have been kind of talking out of both sides of his mouth. And while he's talking about, you know, doing this for the broader good of humanity, he's also negotiating these billion dollar deals with Microsoft. So at a certain point in the fall of 2020, Dario and his younger sister, Daniela, who is the president of Anthropoc now, and five other people, leave OpenAI and then in 2021 announce that they've formed this company. And initially the idea was that they were going to be a kind of safety-minded research institute. And when you go back, at least in retrospect, they say things like, well, we weren't even sure if we were ever going to commercialize this.

17:33We really didn't know. We were interested in what is the future of this technology. But of course, if you're, you know, the remit you've chosen for yourself is to scrutinize these things to make sure that they are safe, it turns out you kind of have to build state-of-the-art models if you want to have state-of-the-art scrutiny. but what they committed to at the beginning was like we're not going to push the boundaries of capability that like we will try to keep up with our competitors and while ensuring that these are safe but like we're not going to get out in front and so Claude actually was like potentially ready for consumer deployment in the summer of 2022 like three or four months before ChowTDT was released and they decided not to release it because they thought that it needed further monitoring and they weren't sure it was safe.

18:25And then JetGPT comes out, famously Thanksgiving 2022, and, like, within two, you know, it's the fastest-growing, like, consumer app in history. Within two months, it has 100 million users. And then they realized, well, if we're going to be able to stay viable in this industry, like, we also have to put a marker down. So then in the spring of 2023, they release Claude, and then it's been this horse race since then where, like, you know, every month or two, You have, like, Google releasing a new Gemini and OpenAI releasing a new ChatGPT. And, like, there's, like, right now, like, Claude, you know, they just released Opus 4.6.

19:00And, like, they seem to be kind of at the top of the leaderboards. But then we all know in a month it'll be Google. Like, the horse race was maybe inevitable. What does it mean for an AI model to be safe? Like, is it just if you ask it, you know, for help in, like, creating a cocktail of drugs that will kill you, it'll refuse to do that? Is it like this idea that AI might replace us? And so I guess I wonder, like, when we talk about safety, what we're actually focusing on. I mean, it's a great question. And like they're like part of what makes this discourse maddening sometimes is because like safety is used as like umbrella term to talk about so many different things.

19:35And some of it is a matter of principle and some of it is just a matter of affiliation. And like a lot of the current trouble that we run into with like some of these questions goes back to like just some basic sociological like history, which is that now at this point a little over 10 years ago, you sort of had you had like two different camps that developed talking about safety. You had like the people who identified as like AI ethics people, and these were the people who were primarily concerned with things like bias and transparency. And then you had the AI safety people who were like much more concerned with things like existential risk.

20:12And, you know, in kind of if we lived in a better world, like maybe 10 years ago, those people would have like come to some kind of rapprochement. But they were just like they cared about different things. They were like they identified politically in different ways. And like they decided that they really didn't like each other and didn't get along. And so then like we ended up with like this kind of stupid false dichotomy between like caring about like proximate harms like bias and caring about, you know, potentially like catastrophic harms like paperclip problem. And there's been this kind of like idiosyncratic like path dependency where we've ended up like thinking of these as like two different problems with two different camps.

20:51But I think there's a professor at Stanford who does interpretability work named Chris Potts. And one of the things he said to me, which didn't make it into the piece, but he basically said one of the fallacies is to believe as an AI safety person who cares about existential risk is that you can kind of keep your powder dry and then be like humanity's last stand when it kind of comes down to the apocalyptic eschatological moment. And he was like, I just don't think it works that way. Like, I think that the way that you prepare yourself for, like, those, you know, issues of, you know, if we get to the point where there's, like, super intelligent, autonomous actors, that, like, you're only prepared to deal with that if you're kind of, like, in the trenches dealing with, like, all of the proximate problems along the way.

21:40And I mean, there are plenty, you know, Eliezer Yudkowsky would totally disagree with that and would say that like no matter how well you prepare for like the proximate harms, like there's nothing you can do in that endgame. And like it's not a stupid argument. Like it's very plausible, but it kind of seems like short of just like stopping everything, which like there is a good argument to do that also. Like short of stopping everything, I think like you would want to take a more holistic approach to all of these things. How did people at Anthropic think about, like, the idea of, like, the singularity?

22:13And, like, I guess I'm wondering, like, you know, part of the AI safety conversation, it's like, for me, AI safety would be maybe there not being an AI that's, like, more intelligent than all humans and that can overtake us, even if we think that's going to be a benevolent version of that thing. I guess I'm wondering what version of that conversation is happening at Anthropic and whether they kind of want AI to become better than us or whether they want it to become as good as us but not necessarily better. So what I think is important to say here, and this is something, you know, like my experience with this piece was at Anthropic.

22:48But my strong suspicion is that at like you would find the same thing at Google and OpenAI. I don't know about XAI, but that like there really is a like much greater variance in viewpoint than like one might suspect from the outside. That like you really can find at Anthropic like virtually every position on the spectrum from like, yeah, we really like I stay up at night thinking like we should probably stop to like all of these existential concerns are ridiculous. What are you talking about? Like, Claw's going to cure cancer. And like, we might have some like hiccups along the way and like worries about social instability because of mass white collar unemployment or whatever.

23:30But like, you get the whole range. So it's not I think from the outside, there's like a suspicion that either there's like a complete homogeneity of attitude about this and like kind of everybody thinks like Sam Allman or whatever. Or you get this suspicion that like, people aren't like thinking about this at all. But there's exactly the same kind of range of opinion, probably even a wider range of opinion than you would, like, find among, like, normie people about this stuff. Because they're thinking about it all the time. They're thinking about it all the time. And, like, there are, like, you know, for every one of these positions, you can come up with a good argument about it.

24:05I want to ask you about some of the, like, the political criticism that Anthropic has received. Like, you mentioned in the piece that there are figures associated with the Trump administration, like David Sachs, Trump's AIs are, and then Pete Hegseth, his secretary of war. I'm just realizing, as I say, like A.I. czar and secretary of war, these are really strange phrases to say aloud. What are those criticisms and where do they come from? Like, is there a uniquely antagonistic relationship between Anthropic and Trump world? Or is it just that any major tech company is now going to be kind of going through the ringer?

24:38I think any major tech company is going through the ringer unless they, like, go and pledge fealty, you know, and bend the knee the way a lot of the other executives have. I think that there is a—they do have, like, a special, like, bee in their bonnet about Anthropic, which they kind of, like, perceive as, like, the opposite tribe's AI company. What do they say about it exactly? Well, David Sachs has called—like, went on this rant last spring about Anthropic being, like, part of a Doomer cult, and he doesn't take the whole thing seriously at all. And I mean, but frankly, like, if he didn't have so much power, it'd be very hard to take this seriously at all.