191. Is It Already Too Late to Control AI? (Anthropic Co-Founder, Jack Clark)

19:41Rory Stewart:But Jack, just to challenge for a second, one of the things we keep hearing is safety. I mean, certainly it seems as though that's what is driving Elon Musk in 2016. He's worried that Dems for Services' system's not safe enough. And then there seems to be an anxiety that open AI isn't safe enough. I mean, so somewhere here, and this is one of the things that I noticed when I'm talking to the founders of these companies, and I'm saying, why are you taking these risks? Or why are you traveling at this speed? Often the answer is, well, I have to get there before this other person. And the other person was often one of their friends 10 years ago.

20:19Rory Stewart:And I've got to get there before them, because they're very dangerous. And unless my models they're ahead of them, we're going to be in trouble. And then it moves on to, and anyway, the US has got to get ahead of China because they're also very dangerous. So there's a sense that we're in a kind of America's cut race where we're speeding along and the storm's coming in and the winds and the sails and everyone's saying, well, I can fix the safety, but I can't fix it in a way that's going to slow up my ship because I've got to get there ahead of them. I mean, I'd say there was an era where you wanted to run a range of experiments on different organizational designs, different research agendas, that era has happened.

20:53I think we're entering an era where you actually need to do serious coordination, including between governments. It's great that China and the US actually recently discussed AI at the high level summit, but also you're going to have to bring companies and governments and other parts of society together to talk about this because the era to keep having all gas, no brakes is probably entering the rear view mirror. It's not sustainable to keep this going. And we have to somehow force a larger conversation on what do we want to do with this technology and how do we want it to be developed. So if you were writing the report card on the world for how we're doing on this, where would you get to?

21:32So for listeners, Jack and I have known each other for quite a long time, before Anthropic, but we start really getting to know each other better when I was working in UK government. Jack was obviously doing his role in Anthropic, and we worked together on creating the UK AC and the first AI Safety Summit. And these were sort of early efforts to try and dip our toes into this coordination. That was nearly three years ago. Lots has changed. How do you think we're doing? Well, now we have the UK AC evaluating both Mythos and GPT 5.5 from two separate companies on cybersecurity challenges, which the UK AC has built in partnership with the UK intelligence community.

22:11Rory Stewart:Explain to me what an AC is. An AC is the AI Security Institute, formerly the AI Safety Institute, but they changed the name. Very good. Okay, on we go. And the AC has built cybersecurity tests, which neither of our companies have seen. We, our technical staff, believe in the legitimacy of the tests AC has built because we trust the the talent there, which is extraordinary. And the AC has built a test that governments around the world can trust for cyber risks, because it wasn't built by a company, it is them impartially testing our systems. If you'd said to me in 2020, we're going to build an entirely new function within the UK government that does frontier testing of AI systems, the most powerful AI systems that will ever be built, and they will invent their own tests that won't come from the companies, and these tests will be better than the ones the companies build, I would have said, that's impossible.

23:10There is no way the government can build that function. And yet it has. And that function is being replicated in countries around the world. It's been replicated in the US. They could do with more money in the US, but we'll get from that.

23:23Rory Stewart:So obviously, you were right at the core of building this thing. How do we answer Jack's challenge here. It seems a bit weird, doesn't it? I mean, given these companies have literally are spending hundreds of billions of dollars and are able to pay single programmers $200 million a year, how on earth does a government put together something that can actually run independent tests that actually do any good? Because I can imagine the American companies swaggering around and saying, forget it, you're never going to have the money, you're never going to have the talent, this is never going to work.

23:51Well, and some of them did, not Jack, but I remember a very well-known person in AI saying to me, I just can't believe that you'll be better than us at machine learning evaluations. I remember saying, no, but I really hope we're better at bioweapons. And so joking aside, that is part of the answer, I think. And to Jack's credit, I think he saw this very, very early that both from a legitimacy perspective and a capacity perspective, there are some things that only governments can do. The reason I sort of slightly messed up my life in 2023 to get involved in this is that I really believe that it was both first order good.

24:25It's just a good thing for the world that these things, AI Security Institute exists and can do this work. But I also think the second order effect of building state capacity in the UK to have a lot of AI experts in government actually understanding what's happening would have a bunch of second order benefits for the UK. Now, to your point, could Anthropic, OpenAI, Google pay such people more to do? Absolutely. But I think if you believe, as I do, and I think Jack does, that this is going to be the most important technology of our lifetimes, probably ever, then I think you need to believe from a pure democratic legitimacy point of view, that it can't just be something that's done to governments.

25:04Governments have to start to build the skills to do that.

25:06Rory Stewart:And how do the governments get the money to pay people in a competitive rate if they can all go off and work at Silicon Valley for much more money? Well, two things I'd say. One, it's extraordinary how quickly the amounts of money have changed. I remember securing the first 100 million for AC or what became AC in 2023. And it seemed like a colossal amount of money at the time. And obviously now it's not. But I do think, and this is slightly off topic, but I think it's worth saying, I do think that to me one of the lessons is that when governments actually have a real clear mission with a degree of urgency and top-down support from the prime minister, both Rishi Sunak and followed up by Keir Starmer, actually there are a lot of people, even people whose opportunity cost is earning huge amounts of sums in big tech firms, that are willing to take up the challenge.

25:56So I'm not saying hiring AC has always been easy. I obviously don't work there anymore. But I definitely think the caliber of talent that you've been able to get through the mission has been very impressive.

26:05Rory Stewart:Okay. Well, Jack, the problem that we've got is that AC is not the United Nations of which Demis Fassabas dreamt in this interview that he gave to you. It's effectively voluntary, right? And it's the UK government. And the UK government is not the US government. It's not the Chinese government. One of the things, you're right that when you're being optimistic, you say, isn't it lovely? that US and China discussed AI safety. Actually, if you look at that summit, what's shocking is how little Xi Jinping and Donald Trump discussed AI safety. That's 50 % of the world's economy. Those are the two countries that have all the foundation models that matter in the whole world sitting in them, and they're not really doing it.

26:45Rory Stewart:And so it doesn't, I think, ultimately really matter if plucky little Britain gets on and pushes ahead with its stuff if 50 % of the global economy it's not playing ball. So push on this where how do you trust a plane that has taken off in another country with a different governance system and different regulatory system to your own? It's because you have common standards on things like aerospace safety and testing authorities which exist in each of these different countries or sometimes at the regional level. And planes are able to take off in countries, including countries which are at war with one another sometimes, and land in each other's countries because you have reciprocal technocratic standards organizations that are actually facilitating some shared notion of safety.

27:26Now, the things like the UKAC, which have been replicated in the US, China has its own efforts here, other countries have their own, is exactly the beginnings of what you need for some of this notion of what standards look like. Now, is that going to cover everything? No. For some of the larger risks, you're going to need bilateral or multilateral agreements between countries and high-level diplomatic discussions.

27:50Rory Stewart:Give us examples of some of those bigger risks that would need something bigger. I would say the question of what you do about proliferation of national security capabilities that touch on cyber or bio or like nuclear is exactly the kind of thing that traditionally has been the domain of nations trying to talk with one another about non-proliferation regimes, safety regimes, testing regimes. That is clearly an area where we're going to need to have some agreements with teeth eventually. So let's just dig into one of them, which Matt raised, bioweapons. Yes. Right. Explain a little bit to the ordinary, intelligent listener to our podcast, what are the potential threats of bioweapons and why you would need a particular structure to deal with it?

28:29The best way to think of it is that biology is inherently dual use, right? It's the science of what our bodies are made of. Now, the experts that can build things like vaccine candidates also have the same expertise needed to build things that could cause terrible havoc in the realm of viruses. Why don't they? Well, the world doesn't incentivize for this. The world doesn't want this to happen. And also there are things like biological weapons treaties. This has been a topic of discussion among governments for a long time. And also the basic goodness of people. There aren't that many people that want to just visit harm on others.

29:11There are some, but not many. And those that do typically don't have the capability. Those that, well, the number of people, this gets to the point, right? The number of people who want to visit harm on others in the world is relatively small for this kind of horrendous act, and they typically aren't trained biologists. And if they are, carrying out acts of terrorism usually doesn't require you to not be a lone wolf. It requires some amount of coordination. Now, the risks of AI systems are universal educators. And if you take either an individual or small set of people that want to commit some act of bioterrorism, and they have the ability to access a universal educator which is versed in every aspect of biology, then suddenly those people have been accelerated.

29:55And they've been accelerated without paying the coordination costs or conspiracy costs that typically allow us to find groups of maniacs in the world. Additionally, at the state level, states are controlled in this domain by many agreements between states to restrict this area because there is no interest in things that have the potential for vast collateral uncontrollable damage. States prefer military capabilities that can be a lot more targeted. A bioweapon is almost the definition of a thing which is very hard to target and has huge spillover effects. So for bio, you need to basically solve the two challenges.

30:31One, how do you make it hard for individual actors to access knowledge, like street-level AI that allows them access to knowledge that would allow them to cause harm? And I think that problem has been worked on by industry and government for many years now, so far, with some effectiveness. The harder challenge in our future, though, is what happens when we have capabilities that are the same as or better than the best single group of biologists in the world, which is what we have right now with systems capable of cyber offense and then the question becomes okay how do you think about access at the state level or the corporate level for these capabilities which are now are now transformative at the high end as well should we switch to cyber then because it's very live and obviously anthropic has been in the news on this on this topic so do you want to do 30 seconds on mythos for the two percent of listeners who haven't you know seen it all over So Anthropic trained a model recently called Mythos.

31:31It is a standard AI model that uses standard techniques, same as the ones that the other frontier labs do. And it's very, very good at a range of skills, including aspects of cyber offense and cyber defense.

31:47Rory Stewart:And it's a general model. It is a general purpose model. It also would be quite good at writing Shakespeare's sonnets. It's great at creative writing. It's great at coding. It's great at biology, but it's similar to a few months ago where AI systems got good enough at coding, but suddenly loads of programmers started using them. It went through some hard to predict point of being sufficiently good at cyber, but it gets interesting from the point of experts. And again, just to explain to you, by cyber, you mean the capacity to launch cyber attacks, cyber crime? I mean, the capacity to find bugs in software like Firefox or Windows or your iPhone and hack into it, which is then the key ingredient to cause to carry out hacks or cybercrime.

32:29And can you say a little bit, because obviously we're already here in this world. This is not no longer a thing that might happen in future. It's already here. And Anthropics already had to make some decisions about how to deal with this. And one of them is this sort of structured access. I don't know if that's how you describe it. We are running an experiment right now into how do you take systems that have this capability and you try to diffuse them into the world in a way that is defense dominant. Because what we know now that this system exists is, okay, at some point in the future, it will be systems like this will proliferate.

33:01Many people will train them. The kind of water level of hacking skill in the world will have risen generically as a consequence of these systems. How do you deal with that environment?

33:10Rory Stewart:Just to understand, presumably there are two different things. One of them is that the thing will be much better at finding the bug or the little backdoor into your system. The second thing is that it presumably could mount many, many more attacks very, very quickly and respond very quickly to the defense. So instead of some dude sitting in their backyard trying something, failing, and then trying again, this thing could be generating tens of thousands of these attacks and learning all the time. So there's an amazing opportunity here, right? You have systems that now you can turn onto the world's most important software and ask it to find bugs in it.

33:48And indeed, it is finding thousands of bugs in widely used software at a rate far faster than what these organizations, because we've shared it with third parties, you know, like JP Morgan, Microsoft, others have found in the past.

34:00Rory Stewart:Okay. Now, again, sorry, I keep interrupting, but I just sort of bring the audience along. We were told two years ago, don't worry about this, because exactly the same moment as the cyber attack capacity increases, the cyber defense capacity will increase. Who told you that? Well, I don't know. Well, not. No, no, but I mean, these are people that you know well. And in fact, some of them you were talking about before we came into the room, but I don't want to drop the minute. But I literally was with them two and a half years ago. I raised exactly this problem. And I was told very, very straightly by these people, don't worry about it, because cyber defense will increase at just the same speed as cyber attack?

34:35Well, I mean, it's the issue with, say, cyber attacks or, say, biological weapon attacks is the defender has to be right all the time. The attacker only has to be right once. So these things don't have a relationship of being like symmetric at all. They're extremely asymmetric. And so a lot of what we're trying to do is figure out, okay, as these capabilities arrive, how do you give defenders an advantage? The main thing you can do is give them time, which means finding ways to release these systems to what I think of as a, you start with a small circle of organizations, and then you try to learn how to expand the circle over time.

35:10We are at our small circle of organizations today with Glasswing who have access to Mythos. The goal is expand it over time, such that they can use this to raise the kind of defensive posture of the world, and also get intuitions for how you can use AI systems to change cyber defense. And it feels like Mythos, even if from a capability perspective, it's another point on a relatively smooth curve. I think that for large organizations, it crossed some threshold of relevance where I think even just in the last few weeks, far fewer leaders of large organizations within the public sector and the private sector are like, yeah, but it's all tech bro hype.

35:48It feels like it's been a helpful thing for getting the world to take AI seriously. Maybe the way to think about it is your CTO has to care about coding capabilities and that your software engineers are getting accelerated. But your CEO, general counsel and board needs to know about cyber attacks and vulnerabilities. And so Mythos raises to all of these organizations, AI just got real in a domain which you all care about beyond the technologists.

36:12Rory Stewart:Okay, quick break and then back for more. When you finally find your thing, you want the whole world to know about that thing. So you use a thing called Canva to make it an even bigger and better thing. Whether you want to create flyers for that thing, make presentations for that thing, or design merch for that thing, you can do anything. So people can see your thing, feel your thing, love your thing. The next thing you know, it's a thing. Canva, the thing that makes anything a thing. The right window treatments change everything. Your sleep, your privacy, the way every room looks and feels. At Blinds.com, we've spent 30 years making it surprisingly simple to get exactly what your home needs.

36:55We've covered over 25 million windows and have 50 ,000 five-star reviews to prove we deliver. Whether you DIY it or want a pro to handle everything from measure to install, we have you covered. Real design professionals. Free samples. Zero pressure. Right now, get up to 45 % off with minimum purchase. Plus, get a free professional measure at Blinds.com. Rules and restrictions apply.

37:19So can we just go back a minute and talk about, you talked rightly about this sort of, this capability will broadly diffuse. Right now, Anthropic has it, arguably OpenAI has something close to it. Maybe Google does, but this is very small. And as you say, you've been able to do the structured access program. You've chosen who gets it. As you say, everything we know about AI progress to date suggests that that won't be true for very long. What's your current take?

37:47Rory Stewart:Sorry, what won't be true for very long that a tiny handful of companies have models that capable? You're assuming that quite quickly, in six months' time, Chinese companies will catch up or whatever. In somewhere between zero and 12 months, I think we can expect, but I'd love you to push back if you don't agree, an open source equivalent capability, that's likely from China, but possibly somewhere else. One, do you think that's true? And two, given what you said about attackers only have to get lucky once, defenders have to get lucky every time. What do you expect the real world consequences of that to look like, say, a year out?

38:21Yes, I think that in the order of a year, another model will arrive that proliferates generally. Absent, that is also a policy choice. So it could be the case that government, like the Chinese government, says we shouldn't proliferate an open-way model that's capable of cyber hacking. There are very incentives and disincentives on this side. what happens to the world well you will see likely a rise in some amount of of hacks you will also see a step change in how organizations approach computer security i think the place that you end up with is one where computer security looks more like the white blood cells in your body where you have many many ai systems running as rory said all the time at speed patrolling your organization in the software and continually finding and fixing bugs.

39:10And it will be a new, more robust way to do computer security than before. Just as when our own immune system encounters a new virus and we have no defenses for it, some people get sick, the same will be true in the cyber environment. We will see bad hacks likely proliferate due to AI and the world will go through some period of adjustment. I do think that on the other side of this, you end up with likely a more robust world from a cyber capability than we've had before.

39:36Rory Stewart:Let me be challenging for a second and unfair, maybe. I guess the anxiety, if you were listening to this, is that's maybe a little bit too confident and optimistic. You've sort of described a scenario which is a bit bumpy, but we come out the other end and things are a bit better. Somebody listening to that might say, well, isn't there a scenario where actually that could be pretty catastrophic? That period you've just described before the white blood cells get going and some companies fail to adapt, some do, might actually be a description of AI models unleashing tens of thousands of unbelievably aggressive, effective cyber attacks, which could do shattering damage to the global economy.

40:20Some might say that this is a scenario that you could encounter as well. Some of this is a choice. It is a choice not just on the part of companies, but how seriously governments take this and how aggressively governments and companies work together to go around critical infrastructure and other providers and secure it okay my follow-up question

40:39Rory Stewart:yes your follow-up challenge my follow-up challenge right so you basically that the narrative we got is you chose you you realized you had this thing you chose not to release it thank you right but again listening to that it's a bit like well these guys kind of built a nuclear bomb and they decided that was a bit dangerous. So they decided on their own volition not to let anyone have it. That's a bit worrying because that implies there isn't at the moment a government regulator or somebody who told you, you can't release this. Because now we have to gamble that it's not just you're being good guys.

41:16Rory Stewart:Apparently, the guys running Gemini have to be good guys. The guys running OpenAI have to be good guys. The guys running Grok have to be good guys? I mean, are we in a world in which we're essentially saying that a bunch of voluntary good guys build these amazing weapons and then they just choose not to release them, but nobody can tell them what to do? I think the world we're in is one where AI companies are building the equivalent of nuclear power plants. And we just had our first case where when we upgrade the plant, a nuclear bomb also falls out of it. You say, oh, this has some implications.

41:46Now, as a society, do we want there to be more nuclear power? Of course we do. Do we also want to manage the risk of the nuclear power, of course. Do you want to deal with a potential proliferation problem of nuclear power plants spitting out nuclear bombs? Absolutely. Do you need laws for that? Yes, of course you do. Like I'm not, I'm not sitting here saying, leave it to industry. No, the situation I've described is like extremely unusual. And I think the thing which we're trying to do with Mythos is we're trying to tell the whole story, which is, hey, Good news. It keeps being the case that as we make these AI systems better, they are more capable at many of the things that we want them to be capable at, like discovering vulnerabilities in code, advancing science, advancing our ability to push forward healthcare.

42:33And also, it turns out that as you make them better at this, dual-use capabilities are now showing up for the geostrategic. This means we can't treat it like a normal technology. We were going to have to change our mindset. by sharing it with a set of companies and with organizations like the AC we've made it so that you don't need to trust the claims of the originating company you can ask them or you can ask for AC and that has generated I think the best information for the world that this is legitimate and it gives us time to work out what we need to do and of course as part of that you should have some regulation that says you don't get to choose whether or not to release a nuclear bomb but but But you also don't want to have a regulation that prevents the nuclear power part, because then you're going to be where the world found itself after overreacting to nuclear power issues in the 70s, where we just stopped building it in large swaths of the world and lost out.

43:25Let's up the ante a little bit on that. Oh, we're going to up the ante. I'm very – well, so you recently wrote what I think is a brilliant essay that everyone should read on this idea of recursive self-improvement. So you'll finesse it better than this. But effectively, AI that can improve itself, and therefore you could get some sort of intelligence explosion. We can get into whether that's the right way of describing it.

43:46Rory Stewart:Just explain it again for the audience. Why would it explode? So effectively, right now, AI progress is bottlenecked on Jack's very smart colleagues and their counterparts. And they need to keep coming up with good ideas, and they need to keep improving the machine. Jack's written a great essay about the timelines to a machine improving itself, which would remove at least one bucket. neck in that process. And when it's improving itself, it would be as though the machine had 100 ,000 of the best software engineers in the world working 24-7, and suddenly it would be able to improve much more quickly.

44:15And so this, you know, to Jack's, you know, I think really productive analogy of building power plants that spit out nuclear bombs, you know, I think you can up the ante and say, you know, that is like the next level of that. Then you're building power plants that build themselves, which also have the property of occasionally spitting out nuclear bombs. I suppose, like, some people would say, well, you know, Rory said, you know, is there, you know, we want to make a choice. There's regulation. You say, yeah, we should make a choice. I suppose like one more profound worry would be, do we have a choice?

44:43Not in the sense that, you know, I mean, in some like technical sense, of course we have a choice, but there is this sort of, you know, some of my, you know, much more, you know, sort of left-leaning friends would talk about, you know, the sort of techno-capitalism, you know, that actually there is no choice here. We kind of have to do this at some level. We are just - Do what? We have to allow this to happen.

45:06Rory Stewart:Because the techno-capitalists want us to allow it to happen. So I think there is a point of view that actually the choice runs counter to all our incentives. You've already brought one of the arguments that's used. If we don't do it, someone else will do it. That's true both within countries, between competitors, and it's true across countries. Imagine that we self-deny the right to build nuclear power plants that build other nuclear power plants that spit on nuclear bombs. China will do that. Russia will do that. And so there is this sense of like, I'm really interested in like, where is that choice?

45:36Now, I actually think that we are not star incentives and that virtue is a thing and we can make choices. But what do you say to people that say, it sounds like you're describing something where actually there is no choice. We have to do this. Well, there's a choice which is basically on a spectrum between maximizing individual sovereignty and maximizing safety or what might critically be called paternalism, where maximal safety is no one gets access to it other than maybe just the government or the safe organizations. And maximum liberty is everyone gets access to it. We trust in the ability of people to experiment with this.

46:13Now, obviously, both of these are ridiculous positions. Like if basically no one gets access to it, you get essentially none of the benefits and you centralize power into like one or two entities or a small handful. If everyone gets access to it, you're now rolling like a whole bunch of loaded dice with the lives of people around the planet. There must be paths through this that look more like gradations of access to the technology that we allow to come to market. So you're saying essentially that we have to figure out that coordination, both within the industry and maybe more importantly, and to Rory's previous skepticism, more challengingly with China.

46:50And I think I'll get it wrong. I think you said you thought there was a 60 % chance by the end of 2028. Yeah. So we're talking, you know, two and a half years to coordinate, you know, the within industry thing is quite challenging, but with China on not only that we do this, but how we do it.

47:08Rory Stewart:But paradoxically, I think coordinating with China is easier than coordinating between these bunch of people sitting in Silicon Valley. I think that I'm less worried about China than I am about how on earth you get. I think countries have the enlightened view that they're around for a long time. And there's an interest in reducing chaos. And I think industry has the incentives of we might be around for a very short time and we're in intense amounts of competition. So I agree with Rory, but actually, I think it may be easier. I think the Chinese Communist Party is relatively rational. I think if China is convinced that this thing represents some existential threat to the Chinese Communist Party, to humanity, to etc., and they're six months behind, they have every incentive actually to come up with some regulation.

47:52Rory Stewart:But it takes two to tango. the people that I find much more difficult to understand coming on board is I don't really wake up in the morning and think Elon Musk, Mark Zuckerberg, Sam Altman. These are the guys that are really going to want to sit around. I was actually thinking of you last week when, when, when Sam Altman and Elon Musk dutifully turned up to court to give evidence in this thing. And I think like one thing, Jack, I'd love to know whether you, given your, some of your recent experiences, whether you agree with this in the abstract, you might want to talk, not want to talk about specifics, but actually one encouraging thought on that is actually, American companies show up to court and they sort of broadly do what judges say.

48:29It's a low bar, Matt. That's good. And they do what their governments tell them to do. And actually, if anything, I think what we've learned over the last six months is that governments will be very assertive. Well, the US government will be very assertive. It has a ton of levers at its disposal to make companies do what they want to do. And the idea that the US government couldn't force coordination between these three or four private companies, I think that seems very unlikely to me. My bigger worry is maybe the CCP, if feeling it's dealing with someone that's coming to the table, will make a deal.

49:03But actually coordinating those two countries to do a deal is not trivial. I would say, let's just look at the general landscape. All of the frontier AI companies ended up doing bioweapon classifiers. All of them have ended up sharing lots of details with one another. Now, it's not been made mandatory. but I think that's almost beside the point because you can show that industry like self-coordinated onto, hey, let's not randomly proliferate things like bioweapon risks into the world. That feels like an even lower bar than show up to court, but yeah, yeah. Hey, but we passed the low bar. We didn't proliferate bioweapons and we show up to court.

49:38Okay, like that's two wins. We'll take it. Now that to me is at least a proof that you can do this basic form of coordination. It does require two to tango. It requires there to be greater of political will to do something tougher on regulation. We have a regime of transparency and transparency reporting right now, which is almost like, tell us about your manufacturing processes and details of the labels you've put on your AI systems. Clearly, it'll go further. And in the same way, we have aircraft, automotive, and food testing for safety before you ship it to consumers. I'm a recent dad. Lots of my kids' toys get tested effectively to make sure that when my kid inevitably eats it.

50:18It's not fully covered in that.

50:19Rory Stewart:I'm worried by this, Chair, because actually the story of that regulation was terrible. There were a lot of kids getting poisoned with lead in their toys before that happened. The story of all regulation has this property, but surely we can learn that the fact you say that is a reason for optimism, right? Or pessimism because, unfortunately, the examples you've given are much, much more slow moving, and the tolerance rate for failure is much higher. So food safety, you know, humans been eating food for 150 ,000 years, a current species, right? And we've learned a lot about what kills us and what doesn't, and a batch kills someone here, and eventually we get up for food safety.

50:58Rory Stewart:But you're talking about a technology which in two and a half years can be spitting out bioweapons, nuclear weapons, etc. And again, getting to the aviation safety is a slightly different story. I was with aviation safety people this morning. China is completely obsessed with planes not crashing, which is one of the reasons why technological development in China is really slowed. I mean, one of the reasons why it's going to take them eight years to build a jet engine and then another seven years to roll it out is they're terrified about planes falling out of the sky. We're not talking about that kind of industry.

51:34Are you not encouraged though, Rory, by how active the interest of the US government is in this from a national security perspective.

51:41Rory Stewart:I'm, well, okay, and we can come on to that in a second because I think that actually raises another type of question. But let me just sort of put out my thing, right? Okay. What I worry about is that if in two years' time something really catastrophic happened, I don't know. Critical national infrastructure goes down. Critical AGI, mad, tens of thousands of mad cyber attacks, bioweapons release, etc. And somebody replayed these conversations that we're having. They would not be that impressed because the gap between the catastrophe and the sort of reassuring stories about, well, the US government's taking national security seriously and the companies have voluntarily done stuff on bioweapons and we all know we need to get to regulation.

52:32Rory Stewart:It doesn't feel like there's quite the sense of urgency. I mean, I had this recently with one of the big companies saying to me, after an hour and a half argument, listen, I agree with you, we should be regulated, but you've got to regulate us, all of us together. If I were you, I'd be going out there and doing the regulation, but you can't expect us to do it because we're in a race with all the others. But what I don't see on the other side is where are people generating all the details of what it is that they want to test these companies on, regulate them on. In fact, often what I hear is the companies playing devil's advocate and saying, well, come on, you want to regulate us, tell us what you want to regulate.

53:05Rory Stewart:You're not telling us what you want to specify, right? I disagree quite strongly. I think it is the ability and the choice of companies to do stuff beyond what is mandated by regulation. We do this already today where we commit to a range of safety testing of our own products and we publish it. And we say, we think that ultimately this should be what regulation should look like but we can generate information ourselves at all of the frontier companies there are people that care deeply about the safety of their systems i mean how could you not when you're building it it's not you don't get to stare directly into the heart of like the ultimate cyber hacking machine and say oh well this will be fine famous people watched up on hype man right there's a moment where those guys think there is a non-trivial possibility that when they trigger the first atom bomb they could get a chain reaction that blows up the whole universe.

53:55Rory Stewart:And they do it. I mean, we know that about human nature. We know that humans can be very worried about things. We know that 30 % of the engineers in Google could be very anxious about these things. But we also know from things that have gone wrong in the past that companies can make catastrophic mistakes, notwithstanding the goodwill and the seriousness of the engineers. What does the future looking back want? It wants you to have some kind of mandated safety testing thing, which everyone has to do and everyone has to go through. And it wants you to have some notion of sharing details about these risks with society, and it wants you to do something where society pre-positions to get advantages or deal with risks, or to say no if the risks seem intolerable relative to the gain.

54:35All of these things have the shape of beginning to happen now, and we as a company are in vocal, often, support. So vocal, in fact, that it sometimes causes us issues with other parties in this space. And I believe that governments will ultimately act. It will just feel down to the wire because my sense, and you two are experts in this, that governments, it takes a lot to move them to action. And it takes a lot of evidence before a crisis for them to do anything before a crisis. But it's surely the goal. Do you think this applies? We've talked a lot about what I would call national security type risk today for understandable reasons we for the rest of this mini series we spoke a lot about the economic impacts obviously i think your new role uh anthropic or newish role at least in part is very focused on this well first maybe just headline like what do you think you know there's a lot of talk about jobs apocalypse um there's a lot of talk about differential impacts within and across countries where do you think we are on that and then i'd love to dive into a little bit like what do you think governments kind of should do on that?

55:41We're somewhat where we were with AI and national security a few years ago. There's an instinct that things are about to happen that are important. There is no real measurement or testing infrastructure built within governments to do this. And the companies have only just begun, including with the Anthropic Index, to share information. But all of those are the ingredients from which you can build a telemetry system to basically tie, to make causal claims of if AI company does X, why happens in the economy. We absolutely have the ability to generate that data to do that across companies and government.

56:14It should be regulated that companies share information.

56:16Rory Stewart:Tell us a bit about this index. So the Anthropic Economic Index looks at all of the ways that AI is being used by our customers in a privacy-preserving way. And it joins that with what are called O-Net job classifications, which things like the Bureau of Labor Statistics use to classify changes happening in the economy. And what this allows you to do is look at the economic activity happening on the AI company platform and join it with the same economic data used to reason about the economy writ large. If we have any hope of being able to make strong claims about the impact of AI on the economy versus CEOs laying off people and saying it's due to AI, but rather it was due to COVID overhiring, which is a sin many commit, then we need to set up these kind of data sharing systems that absolutely can be done and is being done now.

57:03I'm in here in England just meeting with people with the new AI and Economics Institute in the UK government, which aims to do just that. But we don't know what the shape of the future economy is. All I can tell you is, is I can't reconcile the capabilities of these systems with the economy staying as it is today. Like clearly like massive changes will happen. Everyone that tries to predict this tends to be wrong in like ways that are comedic and outrageous in the future. If you forced me to predict it, I'd say clearly you get productivity multipliers on things, that AI touches, clearly you get the emergence of new companies that are able to do a lot more with way fewer people relative to previous generation companies.

57:42And probably you are going to have some issue with early just out of school hiring, because those are the people that have almost the least set of skills that are most replaceable by AI systems. Beyond that, it's very hard for me to say what the shape of the future economy with AI is, because I don't know if the productivity multipliers compound and also create new industries. I don't know what the shape of like, whether these new firms that are doing all of this, this new business generation, if they proliferate in a much larger number than normal business formation. But you don't think that your vision of RSI, this recursive self-improvement is incompatible with the world of human labor?

58:18Or you do? I think that under something like RSI, the economy grows so much that it's like humans sit on top of an economy that's hundreds of times larger than the one today. You know, A lot of economic doctrine is that what you end up doing is you end up validating and verifying the outputs of automated processes. That's what happens in large chunks of the world around us in things like manufacturing and pricing risk and figuring out as people how you make agreements based on the risk of what you do. See the insurance markets, see things like bond markets, which essentially model risk at the country level.

58:49I think that there will be ample employment for people in new jobs and specialisms we can't imagine that sit on top of this much larger economy. But en route to that, you're going to see like massive, massive changes in the structure of the economy and in jobs. But it's very hard to predict what those changes will be. I think you can just bet there will be massive changes with confidence.

59:11Rory Stewart:Connecting those two stories together, in fact, three stories, mythos, risk and jobs. One of the things that worries me, and you were talking about the US government regulating, is that the US government could wake up in a couple of years' time and say, Mythos 15, we believe for national security reasons is too dangerous to release outside the United States of America, could launch these horrible cyber attacks. At which point, these frontier AI models become proprietary within the United States. Europeans can't access the latest cutting edge frontier models. And then there's a huge sucking sound as all the economic value is sucked out of Europe towards the United States, where these AI native companies, your trillion dollar company with three employees, are set up in the US on the basis of US frontier models.

1:00:01Rory Stewart:And then you make a lot of money and we sit around in Europe hoping that you're going to feel that you've made so much money that eventually you're going to, in the way that I'm sure Donald Trump would love to, share generously with the rest of the world the proceeds of your wealth. Let's do a deal. Yeah. I think this is a not impossible scenario. It's a very worrying one. I think there's a couple of things that we need to work on. one is it's again hard to me to reconcile the shape of this future economy with I guess current ways that we try to like tax or control corporations especially AI corporations the picture looks more to me like well these companies including us are going to have computers all over the world but computers are going to be where lots of economic activity is taking place there must be some way to more directly target that in terms of targeted forms of taxation and I'm not a tax expert.

1:00:52I'm not claiming I have the answers. I'm just saying a basic intuition is that's where the thing changing your economy is. And it has a body and it's geographically distributed outside the US. Surely you can target that. Sounds like a good argument for investing in domestic compute. It is a hugely good argument in investing in domestic compute infrastructure. The second part is with technologies that end up getting classified for military use or being deemed to be relevant to national security, there is always this tension of sovereignty and building and other things. And I have been saying to governments around the world, since I started working in AI policy in 2016 or 17, you guys should build a big computer.

1:01:33We're building big computers and it's giving us like outrageous amounts of leverage. Have you considered building a big computer? I think the choice of building a big computer, which you could do AI systems on, is still there for the sort of European community, plus England. it is a choice. And the problem with this choice is that the numbers get more outrageous each year. So the best time to start it was last year. The second best time is now. A bad time is next year. And you see the picture. I think that has to be a part of this.

1:02:05Rory Stewart:What have you learned over the course of the years debating this stuff about what is productive and what isn't productive in these conversations? One of the things that worries me sometimes if I talk about safety to someone like you who's been doing this for 10 years, because you've heard it all before, your mind will be closed on certain kinds of arguments. And somebody will say, how about the bioweapons going to blow up the world? And you're going to be like, oh, for goodness say, not again. I've been talking about this for 10 years. And here's my answer. Where do you begin to feel the conversation is a bit dead and inert?

1:02:39Rory Stewart:And where do you feel the really live questions are? I think the positive questions are, okay, what are the exact regulations that you need to do. We had a productive one here where you were sort of pushing me and saying, well, that doesn't really cut it. What is actually necessary? I think it's useful to just get people to label specifics. I think it's also good to actually talk about feelings. This might be the fact I've lived in California for too long, but I now say stuff like this. But as someone working on this technology, of course, I'm very excited by it. And I work on it because fundamentally, I think humanity has a huge range of challenges ahead of it this century.

1:03:12And getting through them requires us to figure out smarter ways to generate power, various science breakthroughs, a huge range of medical treatments. AI can absolutely help us do that. But I'm scared of it. I'm scared of the technology that I'm building. And I'm scared of how it is governed less than like, the toys I buy for my kids or the food I buy from the supermarket. It seems like an insane situation to me. And I think actually getting people that work on this to say how they feel is good, because it makes us accountable for it. Like I'm, I'm saying I'm, I'm scared and worried about this because it makes me even more accountable to solve that, you know, for everyone else as well.

1:03:49Well, maybe there's just a very last one, given that you mentioned your kids. Number one question I got after doing this mini series with Rory at the end of last year was, I was the optimist by the way, on the show, you might be surprised to hear. You know, what do you think about what it means for your kids? What are you doing for your kids? Like how you think about educating them? What do you want them to do? How do you think about that? I mean, I think that the sort of childlike wonder and curiosity I have a three-year-old that children have is remarkable. And it reminded me of when I had that as a child and how school methodically beat the curiosity out of me as hard as it could.

1:04:23And the thing that AI gives you is a machine that is leveraged directly by one's intuition and curiosity. and basically having areas of like huge passion and having curiosity about the world are things that are massively leveraged by AI technology and having, you know, rote skills or things like specific career plans are things which are almost disadvantaged by it. So I'm trying to teach my kid. I mean, you can't really teach a three-year-old anything, but in a few years, encouraging this culture of curiosity, encouraging obsession, because the way that I've been most oriented to this technology is I have this passion project of writing newsletters and writing fiction within it and actually having a personal creative practice and hobby has been one of the best ways for me to both use AI in ways that kind of delight and excite and empower me, but also one of the best ways for me to feel calibrated about how good it is because I have something I deeply understand.

1:05:17And occasionally now I'm like, oh, it finally wrote a good bit of dialogue for a story. Okay, you know, it's good to get calibrated. And I think it's just reminds you of the amazing excitement where, you know, in a very realistic sense, we have taught sand to think. Bizarre. Bizarre stuff is a fur. And I think being awake to that is important as well. Thank you very much. We love the teaching sand to think.

1:05:40Rory Stewart:Thank you. Thank you. And look forward to speaking again soon. We will be talking soon anyway in other setups, but thank you. Thank you.

1:05:51Rory Stewart:Well, we hope you enjoyed that. So much to get into. So many other questions we could have pushed harder. And I think we need to keep pushing harder because Jack, as one of the co-founders, one of the biggest companies in the world, is an example of an individual with incredible personal influence and power that could totally upend our economies, our national security, our public service, and in fact, most of the future of humanity and human society. It's a very few people, and we really need to make sure they're thinking clearly and honestly. So we hope you enjoyed that, and look forward to hearing what you made of it.

1:06:22Rory Stewart:And don't worry, I will be back next week.

1:06:30Thank you.