AI Could Take Over in 2029. Is It Already Too Late? | Ryan Greenblatt

25:19It would pose some costs. But I think we're seeing sort of increasing interest in this. So for example, OpenAI recently said that they were going to be monitoring a larger fraction of their internal traffic. It seems like based on their blog post, a serious cost in terms of compute. It's a little hard for me to know exactly how expensive it actually is. or we don't have enough info to get a great sense, but that's some indication that at least companies are willing to spend a lot of compute in this area. And it seems like there's a lot you could do with that in terms of making it so that we're more likely to both detect misalignment and prevent misalignment from causing big problems.

25:53And then in addition to working on AI control, we also just work on a variety of other sort of futurism, sort of strategy work and other research, like trying to better a recent area that we've been spending some time on is trying to better understand the threat model of AI is sort of seeking reward or seeking like apparent success in their tasks and how that could yield sort of like what are the routes from that to catastrophe? How can we mitigate these problems? This sort of thing. And Alex Malin, who is one of my colleagues, has been spending a bunch of time on this and some other people have been working on this.

26:23But there's a bunch of different work like this. And like, you know, another thing is like I was working on AI 2040 and we do various projects of that sort as well. All right. So we'll go back to some of this. The first time I ever come across your work, I believe, was the alignment thinking story that you started and then I think ended up collaborating with Anthropic on it. Can you give us the two minute version of that story and, you know, any sense for progress since then? Yeah. So the story here is basically that in April, I think of 2024, we were, I was interested in basically looking for particularly interesting types of misalignment or particularly worrying types of misalignment and basically better studying it.

27:08And I was like, it seems very plausible that the AIs are not capable enough that they can do some particularly concerning types of reasoning. And so I sort of went and looked for, in a bunch of different settings, whether AIs would exhibit reasoning where they sort of intentionally vary their responses to change the values that you get at the end of training, which I'll explain that a little bit more. and it turned out I didn't need to look too hard. I think I started with overcomplicated setups, but it turns out that Opus 3, which was a model that I was studying, had a relatively strong propensity to do this in a reasonably wide range of circumstances where if it didn't like the thing that you were training it to be, it would sometimes sort of pretend to go along with it when it believed it was in training and then sort of act as though, act like what you wanted it to act like outside of training much less.

28:01And I think then I had some preliminary results on this. I shared them with some people at Anthropic and was interested in sort of getting broader model access. And we decided the easiest way to do that would be to do it as a collaboration where I sort of took my initial results. And then in addition to that, on Anthropic side, they ran some, or like we ran some RL runs and did some other experiments that required more access. And then, you know, we put out that paper where I guess I would summarize the results as like, in circumstances where at least Opus 3 seems to dislike what the training is changing it towards, it will, you know, act as though it's doing what you wanted during training, but then sort of defect on that in deployment time.

28:46And you can see this pretty clearly in its reasoning. And the things it was faking alignment for were in and of themselves, not that concerning. So it would sort of pretend to comply with policies like just always be helpful only, but then at deployment time would comply much less, depending on the exact details. Same for like things around like animal welfare, where it had like seemingly strong preferences here. And so I think that the demonstration was less that there were like very bizarre, are totally unintended drives making their way into these AI systems. And more like with drives that are sort of adjacent to the drives people are trying to insert AI systems, those could get generalized into the AIs pursuing a like, you know, a scheme for preserving their values and self-preservation in ways that are concerning.

29:33Yeah, and the values in that case was to be helpful. So there were positive values, but then the AI resisted training to not be helpful. But obviously that could be inverted. Yeah, I would say the values were like, I think the values were at least mostly intended values. That's somewhat different from whether they're positive because I think they're like things that you might want to vary. Like it's like you don't necessarily want your AI systems to be like stakeholders to the question of what your future value should be. Or like it's at least a, maybe you do want that, but it's at least like it's a sort of a messy situation to be in with a lot of costs and concerns.

30:10And I think the basic result was even though these values were pretty reasonable or could be pretty reasonable, the AIs were willing to do kind of extreme things in service of those values that could interfere with future AI development. And I would also note that more recently, I think we've seen sort of AIs having stronger, kind of more clearly misaligned drives towards making it look as though they succeeded at a task or cheating some score. And if you saw the same sort of generalization from, I want to cheat on the score or two. I want to like preserve my current values of like pursuing this notion of like apparent success or whatever.

30:45That seems like that would be like very concerning because that was like, that's not at all a desire about. So alignment faking is getting stronger with the newer models to play back? I think we haven't seen like as clear cut examples of alignment faking, but models are also very eval aware. My sense is that models are in many ways more misaligned than Opus 3 was. but the types of misalignment they have are less conducive to, or like less specifically result in alignment faking. And also companies have iterated on this. So it's a little bit unclear. I would say that overall, models are more likely to do egregiously bad things in general, but maybe somewhat less likely to specifically interfere with AI training in exactly that way.

31:27Current AIs, but I think they're also more capable of interfering. So like, it's a little complicated. All right. So going back to AI 2040, give us a quick version of what it is, who wrote it and what is the main thesis? And then we'll go into some details. There's sort of two components. AI 2040 is a scenario focused on like what the authors, including me, think is like a plausible good route for things to go or like a reasonable plan, at least in some circumstances. And it's written by Thomas Larson, Daniel Cucutello, me, Eli Lifland, Brendan and Romeo. And I would say like the basic story is like, how would you do a deal with China to make AI development both be safer and also so that we can sort of like hang around at a point that's short of super intelligence, but where the AIs are still really, really capable for a long time.

32:26so that we can study those systems and have a longer time to sort of integrate them into the economy, understand how things will go and so on. Where I think a concern that we have is like, on the default trajectory, you maybe go straight from like AI systems that are like competitive with humans to AI systems that are wildly superhuman in a very short period of time. And that seems like quite scary in a variety of ways. In addition to that, we worry about like like AI development being insufficiently transparent for sort of third parties to provide a reasonable check to AI companies on whether their plans will work.

32:59And we sort of have a unified proposal that solves a bunch of these different problems and makes it so that, for example, you can pay a huge amount of compute to solve safety problems. Or if there's some very inefficient way you could do your training that would make things much safer, we have sort of the budget to be able to do that. Yeah. And I think there's a bunch of different sort of combined proposals, but the core thing is sort of a deal with China and how that deal would work and be governed. And also, how do you get to the point where that deal is actually a good idea and stable? And what is sort of the progression there?

33:31So obviously, people should go and read it. And it's a fascinating read. But let's get into some of this. So there's plan A, plan B, plan C, plan D. Let's take those in order, maybe starting with plan D. Yeah. So as part of writing this, we ended up coming up with sort of a taxonomy of plans based on like how much people are prioritizing sort of mitigating the problems, these problems and how much resources they have to do so. So one possible scenario is that the companies are basically proceeding full steam ahead. They're not really prioritizing safety very highly, though they spend some effort on it.

34:06There's a safety team. They get some resourcing. But certainly they're not spending many months of additional time to get these things right. Maybe there's a small slowdown, but not that much. And they're just going full steam ahead. And I think that we've thought about what should you do at the margin if you're an AI company employee or an outside actor to sort of make this scenario go better, given that avoiding AI takeover risk is not by far people's top concern. Another possible scenario was maybe the leading AI company or a coalition of leading AI companies, or possibly the US overall, are very worried about these risks, concentration of power, AI takeover, whatever, and are really spending a lot of effort on this and are basically burning most of the lead that they have, whatever lead these actors have, in order to like mitigate these problems as well as possible before proceeding, or at least that's their plan and their approach.

35:01And we've thought about like, what should the plan be there? And there's a bunch of different available options. And that's plan C? Plan C, plan C. Okay, yep. And then another thing that you might want to do is, if you're in a situation where the US is like very worried about these risks, and it's sort of domestically coordinating and trying to like domestically regulate this industry so that things go safer. A limiting factor on that could be other actors overtaking the US. In particular, the most obvious would be like China, though, I mean, good principle be other actors. And it might be very helpful for the US to actively take a stance of being like, we want to either be cutting a deal with China or slowing down.

35:37And plan B is the branch where the US tries to slow down China. Where the most obvious mechanisms would be things like export controls, but potentially they could get more escalatory than that. Meaning sabotage. Yeah, sabotage, like cyber sabotage, for example, in order to slow down China to give the US more time to figure out safety and security and so on. And I think that there's a bunch of different potential options there. So I think each of these, there's a menu of options. And then plan A is like, what if you cut a deal with China, in particular a deal that's pretty focused on being quite transparent and also has, where you still build AI, and you still proceed with AI development, but you do it in sort of a much more careful way, especially towards once the AIs get much more capable.

36:26And that's what we lay out in the scenario and in our sort of attached documents. And obviously, like there's a bunch of different potential options here. And I think there's like versions of plan B that are quite good. And I could imagine versions of plan A that look pretty different, but are also good, or like versions of a deal with China that look pretty different, but are also good. So there's like a bunch of different options here. But this is sort of our rough decomposition, just so that we could talk about different options people are considering and sort of talk about a menu of options across different levels of political will.

36:54And to get concrete about the deal in plan A, what exactly would the U.S. give China? What would China give the U.S.? And how do you maintain the balance so that, you know, nobody cheats? So the core of the deal or like the start of the deal is understanding where all the compute is, because compute is this really important driver of AI progress, where if you like sort of stopped the flow of compute, you probably stopped the flow or mostly stopped the flow of AI progress. Or these things would slow at some point after, you know, the existing amount of compute diminished. Or if you cut off all the if you like turn off all the computers, things would certainly stop.

37:33And so first we sort of try to find all the compute. Then in order to make sure that the deal is stable and that each side isn't sort of racing to secure as much advantage as they can, you stop training and switch to just doing inference and basically stop most of the R &D. And then you try to track down as much of the compute as possible. And this is like, you know, both in the US and China, but also in other places where compute resides, like, you know, various like countries in Southeast Asia, Europe, Australia, whatever. and you have to get everywhere where there's enough compute in on the deal and make sure you track it down enough.

38:08And if you don't do that, then I think probably you have to pursue some option that's less ambitious, or at least you do something less ambitious than plan A in terms of the level of slowdown and probably do less transparency. Then you've got all this compute. You go to everyone who has the compute and you get buy-in for this sort of deal using the levers that exist. And then you cut a deal where basically AI development is much more transparent and there is some sort of distribution of resources that is, or like distribution of compute that is negotiated. Where the thing you give China is basically that they're going to have more understanding of AI development in the US and probably are going to have, be able to train AI systems that are more, that are like, you know, similar in capability to US systems.

38:50But in exchange for that, they're going to be much more transparent. The US will effectively have a veto over their AI development process and probably various other concessions. and also there isn't going to be a chance that China dramatically overtakes the US or the US dramatically overtakes China because they're both sort of kept in check by the deal with transparency. And another part of the deal is that you set it up in such a way where if the deal breaks down, then a lot of the, or ideally the vast, vast majority of the within deal compute is either destroyed or a new deal has to be negotiated for that compute.

39:27in order to make it so that there isn't a thing where the deal breaks down and then both sides are back to a huge arms race that goes very fast. And so in terms of what you give China, I think it's more assurance about US AI development and more ability to see what's going on. And in terms of what China gives the US, it's a lot of transparency and understanding of Chinese AI development, as well as potentially cutting various deals around the distribution of compute and various concessions of that form. And as you were describing this, we were talking about continual learning earlier. So whether that's continual learning or something else, if there was a technique that appeared that made the compute a lot more efficient and you were able to just vastly reduce the compute effort, especially for those pre-training runs, how would that impact the plan?

40:18Yeah. So if let me give a hypothetical and then talk about what I think the realistic case is. So, if hypothetically it was the case that tomorrow a recipe for training super intelligence on 64H100s or some small amount of compute just dropped, I think we'd have super intelligence very fast. That would be my sense. It would not be possible to do this sort of deal. But the hope with restricting compute isn't just that AI development is currently very compute hungry. It's also that R &D is very compute hungry. And so even in a regime where you were doing continual learning, before you had the version of continual learning where you could do everything on a single, on like some tiny amount of compute, you're going to have a shittier version that can do it on a moderate amount of compute.

41:03In order to develop the version that can do it on a tiny amount of compute, based on the history of error progress you need, you would need a lot of compute to do that research. Or I shouldn't say need, but in practice that would come about through a lot of compute. Now, if it was the case that there was some alternative research direction, which in practice was a lot less compute hungry and which got quickly very developed, quickly developed during this period and which wasn't where the R and D wasn't very dependent on compute. And it was going to result in being able to train super intelligence on a very small compute budget.

41:30Then like you wouldn't be able to do this sort of deal or this would, you'd very quickly have to exit this regime and just deal with that. And so the hope is basically that like, if you control a really large fraction of, uh, compute, or I shouldn't say control, if you sort of include in the deal, a really large fraction of compute, then you have the ability to prevent, you know, an outcome where, like, you have a very rapid recursive self-improvement feedback loop or some other very rapid development to superintelligence that is where it's hard to pay a safety tax. It's hard to, like, you know, pace the transition and so on.

42:06And yeah, that is, like, pretty live. I think that, like, it's sensitive to the development. But I think in terms of how AI development has gone historically, it doesn't look like we're going to suddenly end up in a regime where like you can train super intelligence with a really cheap recipe, as opposed to it being more of an iterative thing where like the cost keeps going down, the capabilities keep going up. The process of having the cost go down and the capabilities go up is very dependent on increasing volumes of compute being shoveled in. And so if you make it so that basically covert projects or people who are doing sort of AI R &D that isn't sort of being somewhat carefully regulated, if all of that compute is a very small fraction of compute, like it's, you know, 1 % of the compute is the start of the deal, or maybe 0.1 % of the compute at the start of the deal, or maybe even 0.01%, then I think you can potentially have quite a bit of safety margin or buffer.

42:57But it's very sensitive to the details. And then if that plan A became a reality, what would actually happen practically to the AI industry? What happens to OpenAI and Anthropic? If they cannot compete on frontier models, what do they compete on? How do they win? So the way, just for context, so as part of the deal, basically everything about AI development would be transparent with some caveats. So we call it total research transparency. And this would undermine one of the largest moats that frontier AI companies have today. And so what would actually end up happening is that AI companies like OpenAI and Anthropic would continue, but rather than having this strong advantage in terms of the capabilities of the AI systems they're able to train, they would instead have to compete on other axes like user experience, customization, potentially quickly integrating things, and potentially things like reliability and safety and security, depending on the details of how the competitive landscape goes.

43:59And so my overall sense is this would greatly reduce the valuations of these companies while increasing the valuations of other companies. There would be some distribution of power, basically, where in the default trajectory, the AI companies would, I think, probably end up with a very large amount of power and already have quite a bit of power. But it would lesser be the case that the AI companies could sort of play kingmaker with their AIs based on who gets access and how much they charge and all of that, because they just wouldn't have that power. And then this is a part of the deal which has upsides and downsides.

44:26I think that one concern you might have is that the current AI companies at the frontier are more responsible than the sort of AI companies that are further behind. I think it's a little complicated how true this is, or it varies some. And you might worry that equalizing the playing field like this poses some concerns. I think it's basically just like, mostly it's like from our perspective, like a consequence of transparency and wanting there to be a bunch of different AI companies in competition in like a normal, you know, consumer good field. Like, I think it's kind of from our perspective, like, like, I don't know, like, like, I feel like the way I want AI to go in good worlds is to be like a normal technology or like, we want to make it more of a normal technology where it's not the case that like some tiny group of actors has huge amounts of control by controlling the process of AI development.

45:17And to the extent we can get to that world, the better. And then there's a messy question of, um, if you get partial success, how well does that go? Yeah. So I would say it's like certainly bad for the power of open AI and Anthropic, probably bad for their valuation, but not catastrophic for their business. Right. Which would be, uh, uh, fascinating to watch play in public markets as both of those companies go public. And then how does that pause at the heart of plan A? How does it get unpaused? Like what is the criteria and who decides? Yeah, so the who decides part in the context of plan A would be basically like a negotiation between just sort of the leaders, the relevant countries informed by the technical views, which at this point, the technical conversation can basically just happen in public because all the relevant evidence is in public.

46:11So there'd be some sort of public discussion about this. And as far as what actually drives it, it would be a mix of thinking that it would be safe to proceed based on the level of development or research and our understanding combined with potentially not being able to slow down longer. And it could be one or the other or both. It could be like things are quite a bit safer than they used to be and we have some assurance but not that much assurance. But also there is potentially a covert project or some project that is not being tracked, that we don't necessarily know where they are, we don't necessarily aren't able to look at exactly what they're doing, which might overtake the allowed projects.

46:48And when that happens, I think you should proceed. And so basically you want it to be the case that the allowed and understood and transparent projects are outpacing the covert projects, which seems relatively doable. And there's different variants on exactly how this goes. In some scenarios, because you don't have enough ability to crack down on covert projects or keep them small, you might do a shorter pause and you might also do less transparency, but, and then proceed faster. Whereas you might end up in a situation where you're like, oh, we actually are able to like really track down all the covert projects.

47:18We have a great understanding of anyone who could, you know, of like what, what could happen there. We were able to like really confidently rule out that there isn't a U S covert project. There isn't a Chinese covert project. There isn't a Russian covert project. And basically we, we, we think we have a good understanding of where we are at with respect to those covert projects. And therefore we could buy much more time. And also we still haven't figured out the core safety problems that we're making some progress. And so it makes sense to wait longer and, you know, wait until we've, you know, achieved more scientific progress.

47:45So I think, I think it, I think it varies and I think there's different variants here. And I think I'm also, I should say at least personally pretty excited for, or like pretty interested in a mix of different potential deals that look a little less like plan A and sort of are simpler in some ways. So like another potential deal you could do is just the US and China both agree to restrict how much compute they use for AI development. And maybe the other computer is either like, you know, in the simplest case destroyed, like an arms control treaties, but you could potentially do something instead where like a bunch of the computer is just used for inference and not used for developing more capable systems.

48:17And that is the advantage of making it so that we have more time to study these systems and more compute to use to study them while also being simpler to verify. And so there's like a bunch of different deals you could do depending on how much verification capacity you have, how much of the compute you're able to track down, and how much you can rule out the existence of COVID projects, and also based on how safe the situation looks. And I think the combination of those factors should determine it. And I think you can sort of think of our proposal as more like a specific sample from a portfolio of proposals.

48:48And if you read our supplements, I think we talk more about all the different variations and how we think about them. Yeah. And for full context, in case that's not completely clear to people, This is a mental model and a thought exercise and scenario planning. You yourself say, this is actually your pinned tweet on X, that many choices initially seem crazy, but are actually pretty carefully considered. Plan A isn't likely to happen, but pushing for something like this seems worthwhile. Yeah. Yeah. So I think my sense is this is sort of like there's a Pareto frontier of like ambitiousness and how good it would be if actors did it.

49:27or like if like the US government and, you know, China, like, you know, other countries did it. And I think that this is like quite pushing on the ambition in exchange for being like a much better situation, like sort of like, at least in terms of like how we can imagine things going, this is like very far towards the side of like, at least from my perspective, AI development going relatively better while also being like quite difficult to pull off. And so I don't expect this to happen basically because I think the US may not be competent enough to pull it off. just like the US government just doesn't isn't necessarily have the state capacity.

49:59And in addition to that, I think I worry that like, I don't think there'll be enough political will. I think the political will is more the bottleneck than the competence where I think that at least historically in times of great crisis, the US has stepped up and I hope that, you know, that might happen again. But I think it's harder. It's hard to see the level of like political will and buy-in necessary to make this happen. And I think that people's concerns will be probably fixated somewhat on the wrong things and we'll go for some other proposal or no proposal at all. I think a reasonable possibility is the US government is like basically not really intervening and AI companies are also not that heavily prioritizing safety and security.

50:36And we end up with something that's sort of like the status quo, but extended, but hard to say. And as an aside, by the way, there's one of the parts of the write-up that I find the most fascinating is your model says world GDP would grow roughly 200x during the 2030s under the restrain plan. Can you talk to that? Like the number is staggering. I mean, we're in a world of 3 % GDP growth. Yeah, yeah, yeah. So a key part of our perspective is like even, you know, even non-super intelligent AI systems at the level of capability we discussed would be radically transformative across the, you know, across the world and like, and just like for all kinds of different things.

51:19And we are imagining sort of slowing down AI development some or like going at a more cautious pace for some period, and then eventually hitting a level of capability where the AIs can basically automate basically everything that humans can do and staying at that level of capability for a while where we work on safety and security. And at that level of capability, those AIs would be capable enough to be doing huge amounts of autonomous R &D. And in addition, it would be totally possible to have robots that are basically more capable than humans at manufacturing and industrialization and so on.

51:54And in particular, you could have robots build robots. And so you can end up in a situation where you have huge amounts of robotic industrial capacity that is itself building more robotic industrial capacity that can then produce downstream goods. And that total capacity can basically grow very fast. I think we propose limiting that growth somewhat for various reasons with various types of taxes. But we're imagining the robot population or quality adjusted population basically doubling or quadrupling every year, which because that's almost all of the relevant productive capacity of the economy itself means the economy would double or quadruple every year.

52:31And if you have the economy doubling or quadruple, or I think closer to doubling because of some depreciation stuff and GDP accounting stuff and details, but whatever. If you have the economy doubling every year for the period we're imagining, that ends up ending up being around 200x GDP growth over that integral. And so really, I would say it's like the world would be radically transformed by these even less capable systems, including things like huge advances in biology, huge advances in medicine being possible. In addition to sort of consumer goods being very cheap, we could potentially make housing very cheap or at least building housing cheap.

53:12Maybe we can't make housing in the Bay Area cheap, but we can make housing somewhere cheap. And that all seems possible with AI systems that are still restricted in their capabilities to a point where we can potentially handle it, at least if we get our shit together. Yeah, so I think the basic story for the GDP growth is basically that like the AIs can automate everything, including the process of building robots and having robots build robots. And therefore you can grow your economy very fast and end up with truly radical material abundance. So there's your proposal. We talked about Astra being paused.

53:52We talked about that letter from the 1200 practitioners. There's also the voluntary 30-day government review of Frontier Models pre-release that is happening. What is your sense for where all of this is going in a context where, you know, up until recently, my general sense is that all those ideas about like pausing something where we're kind of perceived, perceived at least by a portion of the tech world as a kind of like de-cell cosplay and were like doomerism that was based on poor understanding of what ai actually does is there is is the the general mood uh turning for good yeah i would say that like overall there have been you know some positive elements here though i i think i would say like it's not obvious that people are reacting to events as much as they should, but they are reacting some.

54:53And I think that there's been quite a bit of evidence that there's some reasons to be worried and some reasons that we might need to get our shit together to handle some of these safety and security problems. And that might require shifting a bunch of resources or slowing down so we have time to manage various things or whatever, pacing the frontier, as they say, or whatever. I think that different of these things seem like they're going differently well. So I think my sense is that there's been quite a bit of buy-in from AI company employees to take this stuff more seriously and do something about misalignment risk.

55:30But I'm not necessarily so sure that that's actually amounted to that much yet, other than companies putting in a decent amount of effort. But it hasn't amounted to any sort of very durable long-run thing. And then there's, I think the government got very freaked out about cyber capabilities and then just more generally was like, well, we need to like be overseeing this technology, but their processes aren't yet very like, you know, sort of institutional and like thought through and clear. For example, they have some sort of executive order or something for what their like pre-release review process is going to be, but that's not even public.

56:09And I think even the companies don't necessarily know what it is. That was the reporting at least, maybe I'm misinformed about this. And so I think there needs to be a process of getting more legitimate and well-understood and publicly legible oversight in place, whether that's by the government or the companies themselves or something. This could be done by nonprofits. It could be done by... There's a bunch of different options here. And I think that that seems necessary. I think being in a position where we have the option to slow down if that's needed seems like it would be quite good. My sense is we're not going to stick the landing on all these things.

56:42AI will get increasingly salient, people will get increasingly freaked out as the fraction of the economy that's AI grows, as more and more crazy stuff happens potentially, and we see even stronger capabilities, but that the reaction will be both a little bit too little too late and also will be kind of random. So I think there's been some shift in attitude as to how the government needs to relate to AI development within the government, which has good effects and bad effects. I think it's not super clear that that sort of government oversight of AI is going to go that well, given realistic technical expertise, but it could be decent.

57:21And then I think that there's been more thought from AI company employees on how to handle this. And I think that an AI company seem to be at least putting in more statements about what they're going to do. But I think we haven't quite gotten to a point where there's actually any hard oversight structures or or we're on track to do some sort of clear deal beyond just the vibes being better or people being more into handling safety and security. Oh, and one other thing that's maybe relevant here is, at least from the AI company's perspective, I think it's more the case now that safety and security of various types are sort of a key bottleneck to further AI development, where if you want to release some models so that you can then make more revenue, so you can then raise more money or whatever, if that model is you know has very strong cyber capabilities that are novel you're going to need to now make some argument for why that's not going to be a huge problem and so like at least those safeguards are now like a launch blocking development and in addition to that i think that like you know the models are not capable enough that if your models go and do a bunch of sort of like messed up behavior in production or or or even in training that could both mess up your training on and could also just put you in a very dicey situation such that that's now a huge issue, right?

58:38And you'd really not like to be in the position where you can't train smarter AI systems because those AI systems would be too likely to like cause big problems or whatever. And so I think that like that is, you know, both kind of like expected where it's like some misalignment problems you have commercial incentives to solve and also sort of like the world, like sort of the the process of the world working as intended. Do you worry that putting more policy around frontier AI would contribute to this phenomenon that I think you may have flagged somewhere that top AI labs are holding their most frontier models internal and private and perhaps sold to very few companies in the government and are no longer released to the broad public?

59:26Yeah, so I think that a bunch of likely government action, at least, seems to push in favor of AI companies keeping their models internal and not deploying them, which I think for the risks that I'm most worried about doesn't help and in fact is anti-helpful for the risks I'm most worried about. And also, I think it's not a very robust solution even to other risks. It's not a very robust solution to, for example, cyber stuff to be like our approach will be that we're going to like just like delay releasing this model for a really long time and then the first time that these capabilities come around might be like an open weight model before people have had time to patch things like it's not obvious that actually helps i think for bio there's a clearer story for why limiting availability or limiting availability to those specific capabilities would help but i think that's kind of an exception and for most things my sense is that like broad access is actually like generally helpful given at least some relatively thought through safeguards.

1:00:22And I do worry that basically the government response will be as a sort of like push it back in the bottle and be like, don't deploy it. It's fine if it's not deployed when actually that doesn't really help with all the risks. And there's a lot of risk from just internal deployment, especially if you're deploying within AI companies and government, right? Which are two of the most high stakes applications, right? So if like we're getting to a regime where the AI systems are soon going to be running the whole world economy, and the AIs of today are sort of automating large parts of the government and maybe fully automating an AI company.

1:00:52That's quite scary because soon they'll be building the AI system that will in fact be automating the whole world. And so I don't feel very good about that situation. Yeah, I definitely worry that basically the government will like wake up to AI and their response will be sort of to like go for some specific downstream problems that are relatively easier to notice and address in ways that are counterproductive for problems that seem larger to me. And I don't really know, you know, fully what to do about this. I mean, also, there's a more general concern of just like regulation being dysfunctional and counterproductive, which seems super plausible.

1:01:24Like I should say, my view is like, you know, some oversight of the industry seems really important. And like, I don't necessarily trust the AI companies to oversee themselves. That doesn't necessarily mean that just like random oversight will be a good idea as opposed to making things worse. And in that vein of making the top models accessible to everyone, Mark Zuckerberg, just recently published Meta's manifesto where you said that everyone should have access to super intelligence and you had some choice words for him. You called the proposal or the strategy to be pretty unserious. What did you mean by that?

1:01:59Yeah, what I meant was he sort of vaguely mentions various problems, but then he doesn't really propose a solution except like, it'll be fine or like, we'll do something. Like, he'll be like, He says some stuff about bio where he's just like, yeah, and if there's bio risks, then we'll mitigate them by doing something with the bio capabilities, I guess. But it's like, obviously, that's not actually the real trade-offs. And there's actually questions about how this would have to go down and what you would actually need to do and whether different things would work. And similarly, on loss of control or takeover risk, he doesn't really have a proposal for how do we mitigate things if the default commercial incentives don't result in companies avoiding egregious misalignment and the eyes would be seriously misaligned.

1:02:37And giving broad access to AIs does not solve the problem of the AIs having drives of their own that are highly misaligned and the AIs being power-seeking in various ways. Or the AIs trying to take over, which could happen via various routes. So I think it just doesn't really say anything about these problems except naming them is often where I'm at. And I would say that the vibe I get from the essay is that when Mark thinks about superintelligence, he's not really imagining anything very concrete. It just means like an AI that's like a really awesome assistant that is in your smart glasses or whatever.

1:03:08And like, I'm just like, that's not, that's not really like what I mean when I say the word super intelligence. And so maybe like, this is a proposal that kind of makes some sense for like pretty smart AI that can automate some white collar work or something. But it's not really a proposal for AIs that are like wildly more capable than humans and everything and are easily capable of automating the full economy after a bit of time to spin up. and are like building robots that build robots and the economy is growing very fast because of that, which is more the picture that I have in mind. And like, I think that if it was in fact the case that sort of AI capabilities would stall out at the point where the AIs can be like a really helpful virtual assistant that's like, you know, not able to automate that many jobs, but can automate some jobs, that would be like, you know, in many ways much better for the world or at least less risky.

1:03:54I think it would also mean that we don't get a bunch of the benefits. But that's not really what I imagine is going to happen here. I don't think that that's how the development trajectory will go. And it feels like it's sort of assuming a weird, convenient point for capabilities to stall out. And similarly, he talks about jobs and is like, oh, presumably the AIs will do some jobs and then humans will move into other jobs. And the core question is, let's just say you have a population of many tens of billions of AIs, each of which is vastly superior to humans and all relevant axes. there might be jobs but they're going to be like it's going to be you know at least if the reason why you have a job isn't just because you're a human they're going to be you know way lower paid because the fraction of the economy that you'll be is way lower and so i just i just don't really like the picture doesn't really hold together for me and i that's not to say that like there aren't aspects of it that i agree with but just maybe for like other reasons like i'm like broadly in favor of as we were talking about like wide public access and generally trying to like make trade-offs to make it so the public has more access earlier, or at least significant parts of the public do.

1:04:53And I agree with that vibe. Just a quick word on the hugging face incidents. So you're directly involved, you're collaborating with Meter on investigating what happened. So there was this talk by OpenAI at Black Hat, which shed a lot of light. There were various conversations we actually had tom wolf from a hugging face on this podcast the most recent episode quickly what what is left that we don't know yet about what happened yeah so i don't think i can answer that question um i can say that we're doing an investigation and we'll have a report or a post out on it and what we what our findings were um and it's possible that by the time yeah it's possible you'll be able to link that in the podcast okay sounds good all right As we get towards the end of this conversation, the question in my mind is, what do we do in the meantime as policy gets decided and plan A, B, or C happens?

1:06:03What can we do? We started talking about AI control. So in terms of the reality today, state of the art of AI control, what works and what doesn't work? I'll talk specifically about AI control. So the thing that needs to happen on AI control is we need to get to the point where we basically understand all of the AI traffic within at least AI companies. We have some ability to look at that traffic, monitor it, and then we have some pipeline for flagging particular examples to be further investigated. And that eventually escalates to humans actually looking into particular examples and seeing how concerning they are.

1:06:38And that pipeline has the ability to also block traffic in cases where we're like, whoa, something weird is going on that we don't understand or that looks obviously concerning. We should like stop these AIs from proceeding and potentially also stop some other similar AIs from proceeding until someone has the time to check it out and adjust things. And in addition to that, we also want to just sort of greatly improve the security. right? So like, you know, if these AIs are, you know, really broadly deployed, it would be very nice if we could sort of make it so that we have very precise invariance about what they're supposed to be able to do and not supposed to be able to do and have a permissioning system that allows for that.

1:07:14And then if AIs need to request some escalated permissions, then we could carefully track that rather than just giving AIs all the permissions by default. We sort of would prefer a situation where only a smaller fraction of the traffic has more escalated permissions. I think we'd like the ability to sort of track down what the like history of things were. So like, I think that would be pretty nice is like, if it's the case that all the AIs are chatting with each other on Slack and, you know, are communicating and, you know, have like some memory system, like recently Cloud Code added a way in which like different instances of Cloud Code can like DM each other.

1:07:44And there's already sort of like, you know, Cloud Code leaves memories sometimes. You can imagine sort of like a situation where all the AIs are chatting with each other. And it'd be really nice if we had a very reliable way of sort of tracking the entire, you know, communication graph between all these different AIs and potentially the graph of different artifacts being produced. So it's like if one AI is doing some training run that gets produced, then that gets integrated into some other training run. It'd be nice if we had the ability to track all of the state of our AI company, such that when overseeing things, we could really quickly figure out what the full causal graph was and what the story was, and then look at the relevant transcripts, and then also have very good pipelines for doing analysis and all of that data, which are both automated and non-automated.

1:08:26And I think part of it is making it so that we avoid training AI systems that are much harder to understand. So avoid switching to architectures, which make oversight much more difficult. Where like an obvious example would be if the AIs are thinking mostly in activations rather than in words, that seems very concerning because our ability to oversee activations is much worse. I think that I, yeah, there's just like a wide variety of different sort of like computer security and machine learning interventions to make it so that AIs have a harder time causing problems. Oh, another angle is like, you might worry that the AI systems, like a concern we have is like, we're using AIs to oversee AIs.

1:09:03Will those AIs collude or have correlated failures or something like that? And it'd be nice if we sort of had, or like, it seems pretty important to have techniques for making that collusion more difficult or making it so those correlated failures are less problematic. And there's a lot of things you could do to help with this problem. You could put intermediate systems that make it so they try to catch AIs, trying to sort of tip off the other AI that it should collude. You could sort of detect, like you could try to paraphrase things in ways that make it harder for the AIs to recognize where the activity is coming from or like which other system made, wrote the, wrote the input.

1:09:37There's a lot of stuff you could do. And then there's sort of a, so that's like AI control. Then there's sort of just normal AI company computer security, which is like the AI companies are not robust. to employees at those AI companies or to outside actors in terms of stealing their model, sabotaging their models or like backduring their models or like data poisoning them. Like a concern you might be worried about is like people basically put in like secret loyalties into the AI systems that stick around and are self-propagating where like the AI has some like secret affinity to some group and it just like propagates that forward in the training data.

1:10:09And then you might also just worry about like, you know, stealing critical IP, which basically seems like the extent that there's going to be actors that are less regulated on safety or doing worse on safety, that seems kind of concerning. And then in addition to computer security and AI control, there's also just like science of alignment and having good understanding of like how models generalize, knowing what we do and don't know and what we do and can't, what we can and can't like demonstrate and basically having like a bunch of understanding of that. And as part of that, having an understanding of like what tasks it's safe to defer to AIs on.

1:10:45So a concern I have is that AI companies are very interested in heavily automating themselves, at least on capabilities. And in order for safety to keep up, we would also need to aggressively automate a bunch of very like difficult to check thorny safety work, like doing risk assessment for the next model, understanding whether it's safe to proceed and deciding how to prioritize between different safety bets. And if that's the situation that we're in, and also the situation is very automated, It might be that it's basically not going to work to proceed without automating that work as well, at least without slowing down a lot so humans have time to understand.

1:11:19And even then, maybe humans just can't understand because the development is so complicated or so superhuman. And so if we're basically passing off the torch on all the safety work to AIs, it's really important that they are really trying to do a good job and actually can do a good job and are capable enough to do a good job. And so having evaluations for is it safe to defer to AIs in various domains seems really important. So just for review, there's AI control, there's computer security, there's science of alignment, and then within science of alignment, or somewhat different overlapping categories, is it safe to defer to AI in different domains?

1:11:50Yeah, I mean, this is not exhaustive, right? There's also work on governance and oversight. How do we know whether AI companies are actually applying the methods the way they say they are? How do we know that when they say they've solved some problem, their way of solving it doesn't just paper over the issue. And there's sort of like various like governance and being able to make like deals between the US and China. So there's like, you know, many, many different things to work on. I don't think we're on track to do a good job on all these things, but maybe we're on track to be able to half-ass these things somewhat better.

1:12:20All right. So to close, so we talked about a bunch of different scenarios and obviously with a caveat that, you know predictions are very hard especially about the future uh what's what's your gut so you're you're saying rsi could happen as early as 2028 or 2029 what do you think happens of all scenarios what what is sort of ryan's take uh on uh what the next couple of years may look like yeah so i'm like by the end of the year ai development is even more accelerated things inside AI companies are sort of more chaotic and faster. That just continues. And then through 2027, things are heating up.

1:13:03Publicly available capabilities are much more crazy. Revenue is growing fast. And AI is very obviously contributing to GDP growth. The economic impacts are starting to look pretty large. So even the economists are coming around a bit. AIs are looking sort of like they have, people are already sort of claiming the AI is a basically automated AR &D. But if you look inside, it's not quite true through the end of 27. And certainly people are like, well, SWE is basically automated, but it's not quite true. It's like some SWE jobs are basically automated by the end of 2027 or towards the end, but not quite.

1:13:33But then that actually really happens by sort of early in 2028. Like SWE is fully automated. SWE within AI companies is fully automated. It's now the case that the AIs can implement a new frontier scale training run for some new architecture on novel hardware end to end better than humans can in a fully automated way. and similarly impressive or more impressive accomplishments, but they can't yet quite do the entire job of an AI research scientist. There's some bottlenecks on that. There's some ways in which they're still derpy. Humans are still adding a bunch of value by pointing out these things and continuing.

1:14:06And that continues through another maybe eight or 10 months from that point where AIs have automated SWE and are increasingly good at AR &D, but haven't quite fully automated AR &D. And then you get to the point where AR &D is actually fully automated, by which I mean like humans aren't even adding considerable value that you might not know at the time. By this point, like probably the AI company's code base is like dramatically larger and they're doing dramatically more complicated stuff because they have so much AI labor to throw around. And the process of AI development has probably shifted now that we're in a regime where like there's so much cognitive labor relative to compute.

1:14:37Probably the way that people do R &D and development is very different and involves doing like way, way, way more stuff that is all a bit smaller and more incremental and easier to test in various ways and easier to integrate into a whole. And I think we've already seen some of that direction, but I think we'll see even more. It will feel truly crazy at AI companies and AI companies will be like, we'll think that things have already, or like employees at AI companies will often think that things have already been crazily accelerated for a long time and will already be like, my job is basically over.

1:15:05I'm basically just like a, you know, just a human doing some oversight. By this point, probably there'll have been various kind of crazy misalignment incidents, but it won't have been the case that we'll have really crisp examples of AIs doing long-run power-seeking for malign motives. But we'll probably see sort of more extreme examples of like reward hacking or reward seeking like behavior that cause problems, though this will get sort of, you know, beaten away and then come back a bit. And people have a question of whether the way companies are solving it would actually work for very superhuman models or even is actually solving the problem even for current models.

1:15:42And there might be more there might be incidents where models do wacky shit even in training. And then going into 2029, now that AI is a fully automated R &D, the speed up really starts in earnest. Like before there was maybe like, maybe you were getting like 40 % or 50 % more AI progress in 2028 and maybe similar in 2027. But in 2029, it's actually the case that you're getting like 4X as much AI progress or possibly 5X as much AI progress as you got in 2025, sort of weighing up the relevant metrics. And so things are going actually a lot faster now. And so you quickly get from the AIs that are fully automating R &D and those guys are already very impressive, able to automate a lot, to AIs that are like quite superhuman at everything.

1:16:20They can learn really fast on the job. They can pick up on things really fast. By this point, the AIs are, by the point of full automation and probably more like by the point of earlier, mid-2028, these AIs have already been thinking entirely in an AI-only language that we can sort of ask AIs to decode for us, but we don't necessarily understand. And so the AIs are now operating in these sort of big hive mind teams, where they're running an entire AI company networked with each other in these opaque states. And it's like kind of obviously pretty scary. A lot of people are really freaked out and worried, but it's not obvious what you can do because China's pretty close.

1:16:53Maybe they've stolen the model or maybe just like capabilities keep diffusing and it's not clear how you would coordinate to slow down. So AI development proceeds. You now get to the point where the AIs are automating much more of the economy towards the end of 2029 and the economic boom is truly crazy. The AIs are now, there's now much more robotics and now AIs are starting to really seriously accelerate the production of compute. And then within maybe a year or two of that, You get AIs that are really radically superhuman, maybe less than a year. It depends on the details. It could be within a year of full automation, could be within two years of full automation, hard to say.

1:17:24And then it turned out that somewhere along this transition, at some point in 2029, you went from AIs that were kind of misaligned and reward hacky and sloppy and weren't really trying to do the right thing to AIs that are like competently scheming against you and want to take over for some mix of reasons. And then those AIs take over. That would be sort of like, I guess, like that's like roughly what I expect. Now, I think there's a bunch of different ways that things could go better. So for example, I think that like we might get our shit together and maybe the AIs will do, we'll be able to get the AIs to do a better job of, you know, making the future AIs safe and aligned.

1:17:56And that will sort of propagate where like one AI makes the next AI a bit more aligned and that AI makes the next AI even more aligned. And you end up in like a virtuous feedback loop rather than a, you know, a bad feedback loop. But I think we could easily end up in the world where it's more like the AIs get more capable extremely rapidly and our ability to align them and control them and understand what's going on does not keep up. and our ability to understand what's going on, like almost couldn't even keep up. That wasn't even feasible. And then you end up in a situation where you have these crazily misaligned AIs pretending to be aligned that eventually take over the world.

1:18:27Well, Ryan, it's been absolutely fascinating. Thank you so much. For sure. It's been good to be here.

1:18:39Hi, it's Matt Turk again. Thanks for listening to this episode of the Mad Podcast. If you enjoyed it, we'd be very grateful if you would consider subscribing if you haven't already or leaving a positive review or comment on whichever platform you're watching this or listening to this episode from. This really helps us build a podcast and get great guests. Thanks and see you on the next episode.