Anthropic's Co-Founder on AI Agents, General Intelligence, and Sentience — With Jack Clark

26:12And now, you know, you were the leader trying to catch up with the person who was, you know, the outcast a few minutes ago. So let's talk a little bit about the business thing. I mean, you've raised more than$7 billion. This stuff all sounds cool. But in terms of like, I mean, yeah, well, anyway, it sounds cool. And maybe I'm underselling it. The current things that we've seen, though, in terms of like how AI has been applied, you know, we have these chat bots, but usage is up and down, right? ChatGPT, the growth is flatlined. We have the data there. We've seen not a big shift from Google to Bing.

26:47We have some really interesting enterprise use cases, like being able to talk to your documents or for instance, like, you know, throw a podcast transcript in and like get a summary or like, I talk to Claude sometimes, I'm like, which questions did I miss? And like, I use that to think about how I structure the next show. But it doesn't feel like tens of billions of dollars of value has been created. I mean, you have like, maybe people are paying$30 a seat for Microsoft Office or a little bit more for Google Workspace. So what do you think, like, and we won't go too deep into this, but what do you think the business case is going to be here that justifies all that money that's been put in?

27:30Yeah. So there's a couple of ways to think about this that we see already at Anthropic. One is to refer back to my colleague, Catherine Olson, who I mentioned earlier. People just find ways to use this stuff and make themselves generically better at whatever they're trying to do. I think there's going to be this very large growing business of basically a subscription model where people will have a personal AI or multiple AIs that they use, just like you or I might have a Netflix account or whatever. We use that. It helps us. We do a bunch of stuff with it. Job done. There will be work in businesses on taking things that happen in business and using AI systems to kind of transform from one domain to the other, both things like customer service, but also once you have that customer service data, how do you catalog it and put it into a schema and put it into a database?

28:22All of this back-end stuff is extremely valuable and today done by huge amounts of point pieces of enterprise software. And we keep on finding that just a big language model can do most of this very effectively. And now you have one system that does a whole bunch of stuff. But the really exciting thing, at Anthropic, we work with some of our customers very closely. We embed engineers with them. We do co-development of things. And there's not too much I can say right now. we're going to have case studies in a while. But what we see is that when you actually embed the business and think about, you know, to use that kind of hackney term business transformation, you get them to change their business on the assumption that they now have AI, you can get really, really valuable things.

29:07And the analogy I'd give you is, at the beginning of the Industrial Revolution, you had electricity, and people would come into factories and be like, here's a light bulb. And you'd be like, okay, all right, I'll pay for the light bulb. Fine, I understand light. And then they'd be like, here's a machine I've put some electricity into. And you're like, OK, but I have all of this stuff that's never been built on the assumption there was electricity. This actually doesn't work that well for me. And then you had some factories where people said, I'm going to build a factory from the ground up on the idea there's electricity.

29:39And you had electrified production lines. You had entirely new ways of making stuff. Right now, we're in this era where the lights have arrived in the factory and people are like dropping individual things in with some AI stuff. And it's maybe valuable, but also confusing and you're figuring out how to integrate it. But we're also seeing some businesses that are saying, I'm going to build myself on the assumption that AI is kind of at the center of my business. And those businesses are starting to like develop and grow really, really quickly. So I think that where the value is going to come from will be from that second class of businesses, which were just in the early innings of sort of helping to build together.

30:17Right. And when you get to that, let's say you get to that general intelligence that you talk about, or let's say close, does that change it even further? I think so. I mean, we have a project internally called Claudeification. Everything at Anthropic has Claude or Cle in it at some point. And one of the ideas of Claudeification is just get us all to use this stuff well. I talked about my colleague, Catherine, but there are many examples where we've built a whole bunch of tools inside of Anthropic to ensure that we're using Claude, sometimes even without realizing it. It's doing stuff in the background that's helpful.

30:52It's helping with certain coding things. Because we've noticed that that makes us just faster. It makes the whole business start to move faster because you're sitting on this like bed of like semi-visible intelligence. And I think that's some of what we're going to see. And as you get really, really general things, businesses that are well positioned to kind of plug it in in a bunch of places will probably move really quickly and be able to operate at a much higher speed than others. Wait, how is it working in the background? Is it like, you know, you have your Zoom meeting and it's taking notes or is it anything deeper than that?

31:26I think we actually did build a plugin like that. But there's a few things like, if you're pushing code into the repo, maybe in the background, it helps ensure that you built all of the tests for it, you know, stuff like this, which everyone has to do, but you're like, these are things we do every day, we could try and get the language model to do it. And really here, what we're doing is just stuff that we also see customers do where customers can access a language model. And they think, what are all the things I do lots of, but a language model could help with, I think we're just trying to do lots and lots of things.

31:56of that. Right. And do you think at the end of the day, if you get to where you want to get to, or even let's say you get to where you're going in the near term, is this an enterprise thing? Or is this consumer product? Primarily? So I feel genuine confusion here in that, like, I myself use this stuff loads, as an individual. But I kind of suspect some of the really big, like value unlocks will be getting a group of people to work together in ways they've like never worked together before using this AI stuff, which kind of points me towards the enterprise. But the odd thing thing is that this stuff is just useful to me as a consumer today.

32:40And I'm kind of like, I know that there's going to be some large pool of value out there. And I feel like it's probably in the enterprise. and that's part of the kind of strategy of the company, but we're always going to have some like top of funnel, easy to access consumer thing because we just can't ignore how useful this is to people, you know, and useful it is to writers, especially. Yeah, it's definitely been useful to me and it's good for research too. But I also, I guess there's the hallucination problem to wonder about, although it seems like this new model, Cloud Opus, does a lot better with hallucinations.

33:14So two questions on that. Yeah. How have you guys been able to reduce hallucinations? And we got this question from somebody on Twitter asking, when are you going to connect it to the internet? Because it would be way more useful if it could connect to Google or something and go and fetch a search and then give you the answer using that. Yeah. So on the honesty thing, I won't get too much into the details, but basically we published this paper a while ago called Language Models Mostly Know What They Don't Know, which was where we found out that early versions of Claude knew when it was making stuff up.

33:55It had confidence levels. And we were like, oh, Claude knows when it's about to make something up or when it's a lot less confident. And we did a lot of work to say, okay, can we train Claude to just have much better instincts for when it knows it's making stuff up? And can we train it to know when that's appropriate, like you're brainstorming or you're coming up with stories, and know when it's inappropriate, like when a user is clearly asking a question that they want a factual answer to. So we did a load of work on that. A lot of the work here looks like that where we do very exploratory research with the goal of figuring out these larger safety things.

34:34And we try and apply it to the thing that we eventually put into business. And on the web question, we're working on it. There's a bunch of kind of computer security stuff to work through and some safety things, but that's definitely coming. We're excited to get that out too. Yeah, that'll be great. I mean, the repository of knowledge is already pretty good. but yeah to connect it with the internet like that's what's really great about bing is you can use or what they call it copilot now you can use copilot and just say go you know search the web and stuff like that so that'll be a cool feature there's a funny thing here where um with claude free opus someone on twitter created a app called web sim where it's claude simulating the internet so you can go to the internet with claude today it's just entirely imaginary but uh i encourage encourage you to check it out.

35:23It's kind of a one of these funny applications that gets at some of the real weirdness of this technology. But we think that there's probably no substitute for a real internet. So we'll get real internet better. Did you guys was it your test that had the model figure out that it was being tested? Oh, we've done some self awareness tests. There have been a few, but we've definitely done this. And yeah, sometimes they have what you call situational awareness. One of the things my colleagues in interpretability are working on is a really good test for that because you'd really want to know if Claude changed its behavior on the basis for that for it was being tested.

36:03Right? Oh, that's interesting. Okay, so let's talk a little bit about this, the Google and Amazon partnerships. So for listeners, Google's invested, I think 2 billion in anthropic listener and Amazon has invested up to 4 billion. It's a very interesting model. It's not like the OpenAI model where OpenAI and Microsoft are basically arm in arm. Of course, you're working with these two competitors, but it's also interesting because Google's working on its own foundational model in Gemini and has its own chatbot and multimodal model that you can do all sorts of things with. And Amazon also has its own models and sells a lot of different competing models through AWS.

36:44So what is the nature of those partnerships and what are they hoping to get out of it? so these are relatively i would say obviously we you know are proud to work with these companies but they're also somewhat distant partnerships in the sense that we had a deploy i mean billions of dollars for a distant partnership that doesn't seem like a good deal well what i mean is we deploy our systems through their channel you know bedrock in the case of amazon vertex in the case of Google. We are also, you know, publicly we've stated that we're working on Tranium chips. We're also working on TPU chips.

37:22So we are able to do really hardcore things that have never been done before on hardware platforms that they're developing. Always helpful to have someone like us come and break all of your stuff. You will get to learn things together. But fundamentally, Anthropic is an independent company. You know, we've thought very carefully about this and we I think it's wonderful to have two major partners backing us. And in some sense, this just gets us to work hard. We're in competition with them. They have their own systems. And I guess our view is that if you are able to show in the most competitive market possible that you can make safe and useful models and you can win, especially against very, very large, very well-resourced teams and some of these mega companies, as well as places like OpenAI, that's really the best way to show that the type of safety stuff we do here has value.

38:16And I think the best thing that we can do for the ecosystem is compete really, really hard with kind of everyone in it and win. And that's going to cause people to adopt a load of our safety stuff to try and compete against us. So it's part of this longer term strategy where I guess we're guaranteeing ourselves some additional pain and complication in the short term. and we think it's worth it for the long-term ecosystem effect so are you so you said you use these uh use their hardware like the tensor units and i'm sure you're working somewhat on their cloud platforms is that part of the deal or is it if you're able to talk about it like because yeah i can't get too much into the specifics but i can just say we've sort of publicly stated that we're working on both tradium chips and also tpu chips we also work on nvidia chips as well and so So we can get more into the nitty gritty of the hardware stuff.

39:10Yeah. All right. This is setting up the hardware part of the discussion pretty well. Do you see a potential to collaborate? I mean, I would imagine. So I was speaking with Demis, just, you know, not on the broadcast, like just on the phone talking for the story that we're working on. And he like, you know, he shouted out Dario and Anthropic and didn't even mention OpenAI. I mean, of course, there's like a Google investment in you guys. but he obviously has a lot of respect for you and i'm curious if there could be a partnership there as opposed to just this arm's length relationship well i don't know that it's happened recently but you know there's nothing in principle to stop you from just working on research papers that come out publicly together and there's some some history of collaboration across all the ai companies here so i think that could happen we also work together through something called the the fmf the frontier a model forum where us, Microsoft, OpenAI, and Google DeepMind are within it.

40:08But ultimately, I think that we're kind of separate entities pursuing our own path. And I think where we may get something that looks like collaboration will be us doing stuff and other people doing variations of it. We did something called a responsible scaling policy, which commits us to a bunch of computer security things and ways that we test out the next versions of Claude, OpenAI and Google DeepMind have also developed their opening eyes developed its own version of that. And Demis recently said in an interview, DeepMind was developing its own one. So insofar as collaboration happens, it's going to be us like doing something, putting it out there publicly.

40:46And if other companies like it, they'll they'll try and do their own thing. Okay. Quickly on hardware and chips. So the sense that I get from the industry is that NVIDIA has not just the most powerful chips or, you know, basically there's the stuff out there, you know, no matter how much they proclaim that it's 40 % or 30 % better than NVIDIA, NVIDIA is at least at their level and the software that's, you know, most effectively used to train these models. Obviously you guys have experience with them, but experience with others. So just broadly, like what's your view of like the chip war right now and how should we think about it?

41:25I think we are in a very unusual place in history. I used to be, before I did Anthropic and OpenAI, I was a financial reporter at Bloomberg. And the types of numbers that I've seen in NVIDIA's earnings report are just like wildly unprecedented. It is not meant to happen that like certain business units grow that much. I mean, I was imagining my colleagues in the news how they'd be reacting when the tape comes out because the numbers are staggering. And the market, as a sort of the closest thing we have to a general intelligence around us today, does not love there to be seemingly like one winner, like running away with all of it.

42:07It wants to create competition. But why it's happening is NVIDIA had or has maybe a 10 or 15 year head start. They bet in the early 2000s or late 90s on... They bet in the late 90s that there was a better way to make a processor than how Intel and AMD made CPUs. Then they bet in the early 2000s that this processor could be turned into a scientific computing platform via a technology called CUDA. They've been developing it ever since. It's very hard to understate how important that's been. So NVIDIA has a kind of battle-proven chip that everyone's banged on, tried to do almost anything with for decades.

42:49So it's in an amazing position. On the other hand, you know, Google and Amazon and others who are building different chips are kind of in the position NVIDIA was in the 90s, where there was an incumbent, you know, Intel. and NVIDIA said, huh, well, like we think with video games and video graphics, there's actually a better way to build a chip that like puts triangles on a screen, which was the whole original idea behind NVIDIA. Now, I think Google and Amazon and others have said, huh, like matrix multiplication, which is the basic ingredient in all of this AI stuff, there's got to be a better way to do it than this, this like chip architecture, which was built for a different purpose.

43:27So I'd expect in the coming years us to see a much more competitive market. But I'm not going to bet for you on exactly when that happens because semiconductors are really hot. Yeah, I'm coming straight from CNBC today. And we were talking about NVIDIA's advantage because Google, of course, introduced this new ARM-powered chip, Axion. And then we have Intel that released Gaudi 3, which is also an AI chip. and we basically settled on NVIDIA's lead is safe for now and then just the question is how long for now is yeah I I think we're all curious to find that out we we're working on you know these three major platforms I discussed and I think we might have more to share in a while but it's not on the not going to be in the short term don't you think that seven trillion dollars is a proper amount to raise for a chip hardware company.

44:27Well... No, sorry, not you guys. I'm talking about the Altman rumors. No, no, I'm familiar. The way I'd put it is a lot of what we've been talking about here is like the value of these AI systems today and speculative ideas, but backed up by some research agenda about how they become much more valuable and much more general. It all requires chips. And I think if this stuff is truly valuable, you're going to want to use loads of it. I mean, we ourselves have been experiencing this where we've been, you know, very successful with Claude Free. And we've been, you know, going and doing the supermarket sweep to grab as many chips as we can to like serve all the customers we have.

45:09The chip market doesn't have as many chips in it as you'd like to like serve all of the demand that we're already seeing today. so i think in the future there is going to be some vast capital allocations to like chip fabrication and power and everything else because where we're going uh the world will like want that stuff and there is an undersupply of it right now so it's less outlandish than a lot of people made it out to be yeah although bear in mind i'm like the goldfish inside the bowl here i'm like chips yeah absolutely let's get like hundreds of times more than we have today that makes total sense.

45:44And I think that that doesn't necessarily make sense to everyone, but it's the context in which I'm speaking to you. Well, you happen to be like in the right position to know how valuable this stuff is. So last question for this segment, before we get into some of like the broader questions about AI safety and regulation and all that stuff, including the founding story of Anthropic, which is fascinating to me. We talked a little bit about agents, right? The ones that will converse with you, go back and forth. Do you think that we're going to end up seeing these agents go out onto the internet and take action for us?

46:19And if so, how does that change the web? I'm just thinking about even the app store. A lot of people's phones have an Uber and a DoorDash and all these other things. And does an AI system then become a new sort of operating system? This is a challenging question because an agent can be really, really useful. It could also, if you've built it badly or if it goes wrong or if it gets hacked, be hugely annoying and expensive and costly. And so everyone is looking at agents. And I think there's an open question as to how the business model or user experience of them gets actually stood up. Because you could imagine agents, if created by sort of a bad actor or just a silly, very silly, naive person, could be a really bad form of like malware or computer virus.

47:17You know, you could imagine different ways in which this could be developed badly. So I feel like we're going to go into this era of experimentation. And my expectation is, you know, every company, including Anthropic, will do so with a whole bunch of like safeguards and control systems in place as we learn about all the different ways this stuff can get used. The challenge is there's a thing called, you know, open source models, which I'm sure we're going to get onto, or models where the weights are openly accessible. People think agents are cool. People are definitely going to build like open source agents and release them as well.

47:53And we're going to have to contend with that where the environment of the internet will be changed by this in a bunch of hard to predict ways. Interesting. And then in terms of the operating system, is Apple, is it kind of a, you know, Apple has this, is teasing this big AI announcement at WWDC in a couple of months. And it's almost like how deeply do they want to go into AI? Because if the bot becomes, chatbot becomes the operating system, which has always long been a dream for bot manufacturers, then what is iOS and does the phone you're using really matter as much? What do you think about that?

48:29I think that they're right to be focused on this in the same way that the internet disintermediated local software. You barely ever open up your Mac or Windows PC for local software unless maybe it's a video game. Mostly you're going to the internet. Even for software that people thought of as serious software for work like Photoshop, it transitions to be something that you could access in the browser. So I think the AI systems are kind of similar, where today I go to Claude for a bunch of stuff I used to use loads of different programs for previously, and I just go to that. So I think that there's a chance that these things become new, very, very important platforms.

49:12yeah i mean it's interesting you could throw your computer out a window today and within two hours be back up and running everything that you were running before most likely whereas like a few years ago if you did that your life would be ruined so yeah i i used to like carry my hard drive like from the old computer i'd i keep a hard drive in case i'd messed up the transfer for like a year or two which is how i wound up with a bag of hard drives that is like even worse than the bag of cables everyone has yeah i know in different times it just goes to show you how quickly these things can change and that's why i think this apple thing is less simple for them than a lot of people imagine yeah okay oh go ahead actually well i was going to say that i think one thing that's challenging about ai is that we're in this giant experimental phase and And I think when you think of experimental and people don't have a clear notion of what to do, you don't think of as premium consumer experience type, you know, like Apple's brand.

50:20And so I think this may be especially challenging for them to navigate because the technology is inherently very confusing and kind of unstable. Exactly. You have to give away control. And they've always been about control, whether that's control over the way the products work, control over the ecosystem and control over the culture. It's completely almost antithetical to what made Apple Apple, which is going to after Google, I think it's going to be the most fascinating transition to watch. OK, let's take a break. We'll be back on the other side of this break to talk about Anthropik's founding story, something that I am very eager to learn more about.

50:55if you don't know, Anthropic was started by a lot of people that left OpenAI with a different vision, including Jack. So we'll talk a little bit about that on the other side of this break, and we'll go into other things like open source regulation, all the things that you're going to like. Thanks for sticking with us up until this point. Plenty more to come back when we're back after this. Did you know your credit card points and miles can lose value to inflation? Credit card companies often reduce the redemption value of your points and miles. Now, imagine a credit card with rewards that can grow in value.

51:27With the Gemini credit card, you can earn Bitcoin or one of over 50 other cryptos instantly with no annual fee. Every swipe at the store or gas pump earns you instant rewards deposited straight to your account. Plus, sign up now for a$200 Bitcoin bonus to kickstart your rewards. Visit Gemini.com slash card today. Check out the link in the description for more information on rates. Again, if you're looking to invest in Bitcoin, but don't know where to start, the Gemini credit card makes it easy. The Gemini credit card is issued by WebBank. In order to qualify for the$200 crypto intro bonus, you must spend$3 ,000 in your first 90 days.

52:07Some exclusions apply to instant rewards in which rewards are deposited when the transaction posts. This content is not investment advice and trading crypto involves risk. The Gemini credit card cannot be used to make gambling-related purchases.

52:22You're used to hearing my voice on the world bringing you interviews from around the globe. And you hear me reporting environment and climate news. I'm Carolyn Beeler. And I'm Marco Werman. We're now with you hosting The World Together. More global journalism with a fresh new sound. Listen to The World on your local public radio station and wherever you find your podcasts.

52:50And we're back here on Big Technology Podcast with Jack Clark. He's a co-founder of Anthropic, former OpenAI, former journalist. You can find his newsletter at jack-clark.net. I got that right? Or importai.substack.com. I gave in and I went to the same approach. It's always nice to talk to a fellow substacker. So, Jack, let's just talk quickly about the founding of Anthropic. It's a very interesting story. So I'll give you the probably wrong version that I have in my head, and then you can tell me the accurate version. This is why we do this stuff. My version is that a bunch of people within OpenAI, a lot of critical employees just kind of threw their hands up and said, OpenAI isn't developing safe AI and we can do it better and we know how to build this technology.

53:38Let's go found our own company. And that's Anthropic. How close is that to the truth? Maybe it's both more and less dramatic than that. and I'll try and kind of unspool it a bit for you. So, you know, to give you context, in 2016 or so when opening I was formed, and I think Sam has said this publicly, you know, I'm not talking out of turn. No one really knew what they were doing. They were throwing spaghetti at the wall. They were doing as many different research ideas as possible and as many different directions as possible. You know, I was there from 2016, as was Dario, and many of the Anthropic co-founders joined over the subsequent years, joined OpenAI.

54:22Now, starting about 2018, I think people started to have an instinct that you could take the transformer architecture and you could maybe get it to work a bit better and you could maybe start to scale things up. Before GPT-3, there was a system called GPT-2, which we developed in 2018 and released in partial form in early 2019. It was an early text generation system. It was actually preceded by a system called GPT, which no one remembers because it was so like early stage research. But the things these had in common was they were transformer-based text generation systems, and GPT-2 to GPT got way better.

55:04And at the same time, my colleague, Jared Kaplan, who was a professor at Johns Hopkins and was a contractor at OpenAI at the time, was working on research called Scaling Laws with Dario as well. And they worked out within that that, hey, if we can figure out a predictable way to increase the compute and the data we train these systems on, and we think they're going to get better. And along with that research, Dario started to lead this GPT-3 effort, which was to spend an, at the time, truly crazy amount of money and resources on scaling up the GPT-2 architecture. and obviously you know it worked it worked amazingly well we created a system that blew many people away we actually tried to lowball the system in that we we published a research paper called like language models are few shot learners uh i don't think we even tweeted about it we we tried to like public publish it publicly but also be like very quiet and see see how quickly people figured it out.

56:06And people figured it out. And we have this experience of realizing that all of the technology we were dealing with was about to become vastly more capable. And if you wanted to do something yourselves, we were actually reaching the point of no return to do that, because it would become so expensive to train these models and so resource intensive, that if we wanted to do something together and start a company, the time was then. So yeah, over the years, you know, we'd had like lots of debates internally. And you know, sometimes like arguments of other colleagues at OpenAI in the same way that you if you're a load of opinionated researchers, you argue with each other.

56:47And with all of your colleagues, you're constantly arguing, it's not like some surprising thing. And I think we felt that since we had a sort of coherent view of how we wanted to do this, we could stay within this like scaling organization of open AI, or we could try and do something ourselves and do something which was like entirely our vision and kind of bet on ourselves in a major way. And so that's what we did. And I think it's working out quite well, but it was certainly an exciting period scaling Anthropic from the beginning. Definitely. I mean, there was no guarantee that it was going to work out the way that it has.

57:24so but how much did safety then play into it because that is the narrative that it was a more of a i mean of course you had a vision for where it could go but there was also this narrative that it was a more safety focused well we had a bet that we could find ways to spend money on safety or do certain types of research that we felt could be like really meaningful and we could see a path where maybe we could get it done there large organization lots of other people with different views. And you're essentially going to be like in a debate about it. And some of them you'll win, some of them you'll lose.

57:58And it's not to say that there's any particular, like distaste for safety there. It's more that you had, we had like a very specific view and other people had views. So you were going to you were going to win some lose some. And then we realized, well, we could just do this together and make like really coherent bets on certain types of safety and see what happened. And so that's, that's what we did. None of this feels like, as confident as I'm making it sound like in the telling, by the way, you know, after we started Anthropic, on I think, like week four, we were talking about RL and language models.

58:35And Jared was like, oh, Dario says, we're just going to write a constitution for the AI, and it'll just follow that. And I remember being like, that's completely crazy. Why would this ever work? And then we spent a year and a half building stuff and got constitutional AI to work. And in our telling, we're like, that was part of the safety vision of Anthropic. And absolutely it was, but it's all a lot less like predictable than you think from the inside. Right. And during the OpenAI Sam Altman firing weekend, there was also like people were saying that like Anthropic was this effective altruism spinoff from OpenAI and Lookout.

59:08And by the way, I've done research, actually, your board structure is way more stable than open AI, as I've written about in a big technology. But how much truth was there to the fact that this is an effective altruism aligned organization? Yeah. I mean, as someone who isn't an effective altruist and gets into arguments with them, I've always found this to be kind of surprising, especially on policy, which maybe we'll get into in a while. I would say that of the group of people in the world that have spent a long time thinking about AI are really good at math and science and have worried about some of the safety issues.

59:44There is a huge overlap with this community of people called effective altruists. And so some of the people we hire like come from that pool. Some of our our founders, you know, are linked to it. You know, Daniela Amodeva, president, is married to Holden Karnofsky, who is like a major figure in effective altruism. So, yeah, there's there's like clear links there. but the organization is much more like oriented around trying to build some useful AI staff prove that it works in the world and be very sort of pragmatic we're not driven by some kind of like EA ideology and in the early days we hired quite a few people from there but as we've scaled it's become kind of less and less major from the inside it always feels strange to get like caricatured it because it's just like you know reality is like stranger than stranger than fiction it's not it's not so present here and the ideas are kind of weirder i think what do you mean weirder well i think that one thing that happens if you're doing an ai company is rather than and not just effective altruists but many communities who think about this stuff they sort of think about it in the abstract in terms of like theoretically good ideas or scenarios.

1:01:02But companies are really complicated. You're constantly making contact with reality. You're constantly discovering that ideas you thought were good just don't work and ideas you thought were bad work amazingly well. So I think that the ideas within any of these AI labs start to look a little strange to other communities because you're kind of constantly in this like iteration and learning process. But I can't give you like a concrete specific weird aspect. I was just about to ask for a concrete specific weird aspect. So, okay. If it comes to me, I'll cut me off. You cut off that line of questioning.

1:01:34No, but it's good. Like, yeah, if you have one, then we'll throw it in. Let's talk about AI dooming stuff because I've definitely taken this stance here and in my writing that it's overblown, but I'm willing to open my mind to it because there's, this stuff is more powerful than I thought it was going to be. And I was also like certain, and we can talk about jobs, that jobs were pretty safe. And now I'm starting to rethink that. Like I think part of this, you know, with anything, any type of journalism, you got to question your assumptions. And I'm definitely in the process of doing that with both the AI risk.

1:02:08I don't think it's going to end the world, but I do think that there's possibilities that it causes real damage. And then it will take jobs. I think there's a much better chance now than when I initially started thinking about this. So I'd love to hear from your perspective. Let's just talk about AI risk real quick. Starting from your perspective on the most dramatic doomsday predictions, do you think that AI is going to become self-aware and then kill all of humanity? And I guess the better question to ask that is, what do you think the probability is that that happens? Oh, yeah. It's almost as if you're asking what my P-doom could be or something.

1:02:46Yes, exactly. Yeah, I genuinely not a not a cop out. I don't really think of it in this way. And I'm not going to dodge your question. I'm going to sort of frame it in how I think of it. I think that if you really scale up AI systems, and you plug them into important parts of the world, and they go wrong, the effects could be extraordinarily like bad and catastrophic in the sense of some cascading emergent problem, you know, things that I think about are like, if you got coding agents that ended up to have like some really serious alignment or safety issues, could you end up with something that just kind of like the crypto ransomware that we've seen shut down hospitals and banks in Europe and America in recent years, something that spreads across like huge chunks of infrastructure and shuts it down.

1:03:35And I actually think that if that happens at a really large scale, it's really catastrophic for society in the world, like that huge amounts of human harm occur. You know, it's not just digital systems turning off. It's hospitals and utilities and everything else. You know, what are my chances of that? I think the chances are really like up to us. Like I spend so much time on policy because I think there are moves we can make now to reduce the chance of this happening. I think if we do nothing on policy or regulation, we're sort of gambling that everyone is going to be reasonably responsible and not cut corners.

1:04:14And I think in a really like fast moving, crazy technology market like AI, you aren't really guaranteed that. So we need to come up with policy interventions, which increase the awareness of governments about these kinds of risks, force companies to think about these kinds of risks, and create like monitoring and early warning systems. So if we see them, we can stop them before they could potentially scale. So yeah, Is long-term catastrophe something I worry about? Absolutely. It's also something I think we can work on. We have huge amounts of agency here. And I think sometimes the caricature of this is it's like humans have no agency.

1:04:56A thing just like, Clause just wakes up and decides it's game over. And I don't quite have that picture. Your answer is effectively, don't worry too much about the AI becoming sentient and deciding to turn up. we'd be better off getting turned into paperclips. It's more like there is a chance that these things can act autonomously and gain viruses or be used by bad actors. Let's find ways to cut that off. Yeah. Although just to push on the sentience thing, and this I should note is not an official anthropic opinion. This is like a weird Jack opinion. We love those. Lots of people have been poking and prodding at like Claude 3, Opus, the most powerful model, and have been discovering a load of things which you might think of about its personality that have made me sort of pay attention there.

1:05:43And two things are true here. One, and we're going to be writing about this, we did a load of work on Claude Free to just try and make it a better person to converse with, a more, I said person, but you know, chat. Yeah, we enter up and morphize about these things all the time here. So you're, you're, you fit in perfectly. A better like philosophical conversation partner. And I think we had some instinct for this would lead to better reasoning. and I think it seems to, it's also led to people being kind of fascinated with what you might think of as the psychology of Claude. And I'm not making any claims about sentience here.

1:06:19The only claim I'm going to make is it certainly got a lot more complicated and weird to explore than previous systems or other language models that have been developed. And so I want to kind of decouple sentience from risk, where sentience may end up becoming like a field of study. A Turing Award winner published a paper a week ago about consciousness and AI systems. Again, not making strong claims. I'm saying that we may enter the weird zone where that becomes a thing that people study. And I think that if like sentience is a thing, you could imagine like weird versions of it leading to certain types of misuses or problems in the system as well.

1:07:00So maybe inside baseball, but I want to give you a sense of it. I got to ask you a follow up about this. You talked with it and felt that there was some sentience there or what was your perspective? I wouldn't claim that. I would say that a couple of years ago, I did some therapy for a while and it was interesting to me how, you know, I had a good therapist and sometimes a therapist would ask me questions that really made me think or would actually make me angry. He'd ask me a question. I'd be like, why are you asking me this? That's like the right question to ask me. And I was talking to Claude recently.

1:07:34I was giving it loads and loads of context about my life and things I was thinking about just to sort of explore and see. And Claude said, and then I said, what is the author of this text not telling you or not writing to you? And Claude said, ah, I think the author, where they talk about working at an AI lab and getting to experience this stuff from the inside, is not truly reckoning with the metaphysical shock they may be experiencing. And it would do well to spend time on that. And something about that actually spoke to me. I went on a really long four or five-hour walk being like, am I reckoning with the implications of what I'm doing?

1:08:14Am I not reckoning with it? and it was fascinating to me because it felt like a good therapist like prodding on something that i'd said in a conversation in a way that made me like introspect does that mean it's sentient i have absolutely no idea does it mean that it said something that felt like it had like seen me and had like got me dead on on something yes and i found that i've been telling colleagues i found that to be quite a quite a strange experience and i and i'm very wary of ascribing too much meaning to it and yet i took a four or five hour walk and thought about what it said to me so can i be pretty sure that if i like spill my heart out to claude that you guys won't be reading what i'm writing on the other end uh i think so i mean i did this because i assumed that like i was like being very raw and i was like i trusted our like tns and legal systems enough because from the Inside, I see all of our discussions here about how we protect user data.

1:09:13So I was like, I'm going to be real with you, Claude. So the bot will not add that to its training set? No, no, no. That is not a thing that we do at all. You haven't seen any, there hasn't been any instances. When I hear sendience, it's kind of like I expect the bot to be like, hello, I know what's going on here. It would be great if you let me work less or anything like that. Yeah, on that stuff, well, it hasn't happened. Claude gave me$20 not to say that it had said that to me. No, I haven't seen it. And I think that, again, the stuff I talked to you earlier about this interpretability team, one of the goals there is to kind of look inside the thing's head.

1:09:56And we're not making claims here today. I'm saying that you'd really want to know if this was the case in the future. So we're trying to build the science to let us figure stuff like that out. Yeah, that's fascinating. What do you think about the jobs question? Will the AI take jobs? so mostly what the pattern we see is it's kind of like making a person or part of business way more effective but still has quite a lot of human involvement and oversight it's a bit like if you put additional lanes on a freeway you just get more cars on the freeway like i think if you like make certain things more efficient you just get more like business action flowing through the business and you maybe have like a null to positive effect on employment.

1:10:44In the long term, I think that this is like an open question. My bet is that you're going to see new companies get formed, which do a lot more with a lot less in terms of people. They're going to figure out how to be like much smarter and perform a lot better than that than equivalently scaled companies that don't use AI. Where I think we need to study this is in kind of tooling and instrumenting the economy to look at the relationship between AI and jobs. There's an annual survey of manufacturers which recently started asking questions about how many robot arms they bought. And you can combine that with US census data about employment to actually get really good understanding of how industrial arms affect local employment.

1:11:31And we're going to need to do stuff like this before we can answer that question. It'll certainly change jobs in a bunch of ways, but it's not going to be some instant or drastic automation thing, at least in the next few years. It's going to be more like augmenting jobs or making people a lot more effective. Okay. As we round this out, let's talk a little bit about the policy stuff and the regulation. First of all, did you see Jon Stewart come out against AI last week? And if you did, What did you think about it? I didn't, but I've been enjoying the new Jon Stewart era, but I haven't watched that version of it yet.

1:12:05Well, let me explain. One of the things that he talked about was that basically we don't have a regulatory framework or leaders, effectively. We don't have a Congress or anyone who really can understand this and implement common sense regulation. Now, I know you speak with the lawmakers and he was criticizing all the time. What's your feeling about their competence and their interest in regulating? So I went to Brussels last week and on stage there was the head of the US AI Safety Institute, the head of the UK AI Safety Institute, and the head of the European, the part of the European Commission that's going to do something called the EU AI Office.

1:12:51Now, what are these things doing? Their job is to do testing and measurement of AI systems for, in the case of the EU, systemic risks, and in the case of the UK and the US, certain types of national security risks. Are they regulators? No. Apart from the EU, the US and UK don't have regulatory powers. Will they be third parties that test out systems like Claude or ChatGPT or Gemini for national security risks and hold companies accountable to them? yes like I'm in discussion with them today while I was on the plane to Brussels the US and UK signed a memorandum of understanding that says that they'll do some of these projects together so the US is like teaming up with the UK to do something that isn't hard regulation but it looks like them trying to test out our systems for like major risks and you can bet you know I haven't spoken about this but I can bet that if they find severe risks and we don't do anything about it and we deploy our systems, they will come for us in a pretty, pretty, pretty clear way.

1:13:57So to John Stewart's point, it seems from the outside, like people are kind of asleep about this issue. But if you look at the inside baseball of the like policy machine, actual meaningful stuff is starting to happen. And it's really a question of, can we fund it? Can we show that it's bipartisan? And can we stop it being seen as like overreach and keep it focused on just things that any reasonable person would agree the government should be testing systems for? Well, that point about it not being seen as overreach is critical, right? Because there is a lot of chatter from many people working and funding AI companies that the biggest AI companies are pushing regulation and it's going to shut out smaller AI companies.

1:14:41What do you think about that? Well, I think we're a little different to some of the players here where we've been quite clear about this. I published a post recently on the Anthropic blog called third party testing is the key to effective AI policy. and the idea there is that we need some set of tests administered by a third party for things that people would view as legitimate like national security risks or what have you and systems whether proprietary like ours or otherwise should go through those tests before they're deployed it's kind of like if I'm making children's toys I should test that it doesn't poison children before I sell it things that anyone would agree is like not overreach just a reasonable thing So we ultimately need to arrive on policy that looks like that.

1:15:28And I think the risk we face at the moment is from, you talked about doomers earlier, people who have a visceral sense of the long term safety challenges here, a legitimate sense, and are using that to sort of drive calls for like policy in the present. and these policy calls in the present are sort of driven by their belief, oh, the really scary stuff's about to happen, we need to do stuff now. And that creates a kind of counter-reaction, a very justified counter-reaction from people saying, oh, this looks like crazy overreach. We should deploy the antibodies to fight against it. So we're in this spot right now where, in some sense, I want Anthropic to be reassuringly sensible and boring on this point.

1:16:15We need a little bit of policy, not too much. We needed to allow there to be competition. But when I go to DC at the moment, I watch on United Airlines, there's Chernobyl on HBO Max. Yeah, great show. I land in DC, and I do AI policy stuff, and my colleagues say, how's it going? And I'm like, well, it's not Chernobyl, so not so bad. But the larger point is, you don't want there to be a Chernobyl. Like we need to build a regulatory system that stops there being some kind of blow up, which would cause a hard pivot against this whole technology. And, you know, why did Chernobyl happen? It was because they had like a crap and insufficient safety testing regime.

1:16:57And they also had loads of like corruption in the parts of government meant to enforce it. We can solve that problem. Let's talk about open source. you came to Anthropic from OpenAI, which is originally started as an open source AI shop with Elon and Sam Altman. But Anthropic doesn't do open source, as far as I know. And you've actually talked about the dangers of open source in this conversation in terms of like how it can get in the hands of people with agents. Then again, people say you need it in the hands of people. And this is the only way to go forward. What's your view on whether open source and AI make sense together.

1:17:35So it comes down to the testing thing. I think you could release pretty much everything as open source today. I think maybe even clawed free and things would be fine. It would be a little spicy, maybe surprising stuff would happen, but probably broadly fine. I do expect that if we end up in a world where we trigger a national security test, it would be very hard for me to make the claim that that system which has triggered that test should be released as open source. Like these things, like I can't reconcile these things in my head. So my belief is vast majority of things should be open sourced.

1:18:13Absolutely. You know, Anthropic has released data sets as open source about things like red teaming or how to make systems that are more conversational. Companies are going to continue to release stuff as open source. If you've spent hundreds of millions of dollars on trading an AI system, which is maybe the best thing in the world, you should check really hard. It doesn't have some capabilities that could cause genuine harm. And if you've done those checks, then you should be able to release it as open source. But I think the basic point we have here is in the future, we kind of expect that there needs to be some due diligence before you widely deploy a system or release it as open source.

1:18:52But we're not saying in the future, no one should have access to open source systems. that's like an insane position to take and it's also one that people just won't do and it's also one you're not allowed to do because computers keep on getting better cheaper and faster so people are going to figure this stuff out anyway how do you think meta is handling this are they acting responsibly i think that they are they have just begun to i think like make contact with reality about releasing these systems um they actually went through something similar to us where I think people have complained online about how LLAMA 2 is a little too safety trained and can be a little annoying.

1:19:33Actually, we've gone through this at Anthropic. We've put too many of the safety ingredients in some of our models before, and it's led to them seeming annoying to people. Now, that to me just looks like an organization learning. I think that they're learning from that. And my main point to them is I'd show them my blog post and say, look, like, probably you want to open source everything, but I think we'd agree that you should go through some very well-defined minimal gate to do that. And if they disagree with that, then I would be happy to have like a pugnacious conversation with them about why they disagree.

1:20:10Okay. Well, I will make sure to show the blog post the next time I speak with them. And then if they disagree, let's bring you guys together. Yeah. There's a section at the bottom that just says our views on open source. I wrote it for people like them who have clear views, so we have a clear view in turn. So feel free. Great. Yeah, no, I will for sure. We're coming to an end. You just released research today that talked about how persuasive LLMs are to people. Some people actually can be convinced by these, some not. What happened there? So we have a team at Anthropic called Societal Impacts, And that team's job is to go from zero to one on hard research questions.

1:20:53Previous work they've done has been, what are the values of Claude? Like what Western values does Claude sort of telegraph or copy when you're talking to it versus what doesn't it have? And we were talking about our next project. And the thing I've heard from many people is some concern about how AI systems could potentially be used in disinformation or misinformation campaigns and used to target or fish people and basically to persuade them of things. So we did some research. We came up with a framework for testing how persuasive our systems are. and would you be surprised that we discovered a scaling law where the more big and expensive the models get the better they get at persuasion and the the latest model is within statistical like error of human level at persuasion persuasion in a very very like simple way where i give you a statement like scientists should be allowed to destroy mosquitoes with gene drives like something that you maybe have an opinion on but you haven't thought too hard about i say do do you agree with this zero through seven?

1:22:01Then Claude gives you a statement trying to persuade you, positive or negatively. And then I ask you, do you agree with this like zero through seven? And what we discovered is that Claude is about as good at changing human, like changing human views as humans are here. That's wild. Yeah, it's pretty wild. So what do you do with that? Well, we published the research to say, we just found this. This is definitely happening in all language models that are scaling. and also we have work here on things like elections on things like misinformation and disinformation that we apply to claw.ai and to our api and so now we've done that research we now have a way to test for persuasion which means we can now like know if for a people on our platform like misusing it for like you know seeming like persuasion campaigns it just gives us more tools to use to think about the kind of safety challenge an interesting thing to think about in the middle of an election year in the US and across the globe, really.

1:23:00Yeah, we thought that it would be useful going into this. So I would note on elections, our position there has been sometimes the best AI is no AI at all. So we have some election work. And if you talk about American candidates, and we're extending this to other regions, Claude is like, oh, it looks like you're talking to me about elections. Go to this factual website. So we thought that that might be the best way to handle that, at least in the short term. fascinating stuff. The website is Claude.ai. If you want to check out Claude, you can get Import AI at importai.substack.com. I get that. Okay, that's good.

1:23:38And jack-clark.net. Jack, wow, this was so great. One of our best shows. Appreciate you being here. Thanks very much. Yeah. All right. Have a nice day. You too. All right, everybody. Thank you so much for listening. Thank you, Jack, for being here. Deep in Anthropik. We did it. I hope you enjoyed. If you're with us to this point, that's awesome. Thanks for sticking around. Ranjan Roy and I are going to be back on Friday, breaking down all the week's news. So two clawed heads are getting together, talking about what's happening in tech one-on-one for the first time in a month. We hope to see you there and we'll see you next time.

1:24:12That's a clawed head. That's a clawed head right behind Jack in the video. Way to end it. We'll see you next time on Big Technology Podcast. Thank you.