Washington Hones in on AI Safety, What Are AI Doomers Proposing | Diet TBPN

0:01John Coogan:What are the AI doomers actually proposing? That's the question I was trying to answer this morning. For a while. 20 year sentences. That's one. AI 2027 predicted that this month, you know, late in 2026, Congress would wake up and that is in the journal. Congress is suddenly waking up to the AI doomsday threat. And so this is happening all over the place. Was it Matt Damon who was caught on TMZ being interrogated about his thoughts on AI risk? There are now protests. I believe AI 2027 predicts a 10 ,000-person anti-AI protest by the end of the year. I was trying to figure out how big this protest in Chapel Hill, North Carolina, earlier this month was.

0:50John Coogan:I think it came in sub-10K, but certainly tracking to it. It was about 150 people at this protest in Chapel Hill, but just two ooms away from the prediction from AI 2027. So I wanted to dig into what the actual proposal is, because the joke for a while has been just like everyone, when they're pressed on this, they just say, we got to talk about it. We got to talk about this. And the actual proposal is super concrete in AI 2040. And it's very interesting just to hear about how they want to slow things down. So the thing that I think a lot of people online who are like, yay, an anti-AI sentiment that's going viral are gonna be depressed about is that like, this is not stop AI at all.

1:36John Coogan:AI 2040 is like, keep the inference flowing. The current models are great. Also, let's keep doing capabilities research, but we just wanna reach super intelligence by 2040 instead of 2028 where we're not necessarily prepared. So they want it to be highly controlled by governments, nation states, highly secured. And there's a whole bunch of very, very tactical recommendations that they make around that. So the first mechanism is an AI pause. They want to pause training. They don't want to do any more new frontier training runs or R &D experiments. And to enforce this, they're calling for to apply inference-only verification to essentially all major AI data centers.

2:21John Coogan:So anyone who has more than 10 ,000 H100 equivalents, roughly$100 million of equipment, and that is pretty easy to figure out. You're just like, big building over there. Let's send the inspector inside. Oh, says NVIDIA on all these chips. Count them up. There's over 10 ,000 of them. You've got to apply for this permit. You've got to tell us what you're doing, right? Very easy to enforce, at least in the United States. And with an international body, you could kind of do the same thing internationally. The whole goal, you can only inference the current models and you have to verify your workloads with an independent auditor, probably the government, maybe there is some sort of non-governmental organization that is doing this.

3:01John Coogan:There is a whole bunch of different solutions that you can pull from across nuclear non-proliferation work that has happened in the past. Major countries, they want them to declare AI compute inventories. tell everyone not just your local population but also the international community how many how many warheads you got how many h100 equivalents do you have where are they everyone shares this that's gonna be a really tough sell because international agreements are really really tough sells it's much easier to have a groundswell of support for something that happens in America America changes we have a system that we don't really have the international to quickly implement that in a way that doesn't allow for a lot of defects.

3:44John Coogan:They want to physically remove high bandwidth east-west networking inside of data centers so you can't do large distributed training runs, but you can still do inference. For any new AIR &D data centers, they want entirely new facilities built from scratch with nation-state level physical security and verification. But for the next run and the one after that and the one after that as it gets crazier, we want it in a new building built inside a Faraday cage so you can't communicate it from the outside. We want a bunch of physical controls. It's like going to a nuclear facility, highly verified who gets in the building and when.

4:22John Coogan:Air gap communications. This is one interesting proposal that they have that really shows how deep they thought this through. They want the R &D data center to be connected externally. If you want to communicate with it and you want to tell it what to do, okay, train the next running or do whatever, they will have a bandwidth capped connection at one meg per second. So you can send little instructions, but if you say, send me the weights because I'm taking them somewhere else, it would take you like five years to exfiltrate it. So interesting hardware solution to this. How do you actually go and implement that?

5:01John Coogan:There's going to be a whole bunch of other things. But interesting that they're thinking about the width of the pipe. So it would be very obvious if you're stealing the model weights because it's like, wait, this one meg pipe has been at full tilt for months. What's going on here? Someone's taking the stuff out of the data center. When frontier model weights move from an R &D facility to an inference facility, They want it to be placed on physical storage devices encrypted independently by both the US and China. So both countries have to sign off and physically escorted by representatives of both countries to the destination.

5:33It's a tall order.

5:34John Coogan:One's a tall order for sure. Yeah. And they actually want frontier models to be made deliberately larger than compute optimal. So they want the weights to be 100 terabytes instead of honing them down to something that's just one terabyte that could actually be moved around a little bit easier. Yeah, I mean, first of all, a lot of this tracks with the kind of regulation that we had pushed for beginning about two years ago around podcasts, wanting podcast studios to be air-gapped, wanting Faraday cages around podcast studios. A locked briefcase with an SMB7, SM7B in it. And in order to unlock it, Patrick O'Shaughnessy and David Senra both need to give you codes to independently verify that this podcast is worthy of being recorded.

6:18John Coogan:I like that one. Yeah. It just makes sense. High level value valve they see as being like the most effective in controlling the speed of capability improvements is compute caps. So there is a world and we're going to the Bernie Sanders thing because it's already getting like sort of twisted, but the big hammer is just chip controls and data center build out slow down. That's the easiest thing. And I think that's like the biggest valve that they're going to be, that we're going to see twisted around to actually slow down capabilities. And so the goal is to allow models to get better mainly by adding hardware rather than inventing better algorithms which can leak to secret projects.

7:02John Coogan:So the goal is like, okay, well, we know that this model is capable of this, so we want this much compute over here. Okay, you've done well. We're allocating more compute as opposed to this one weird trick that AI doomers hate. Yeah. So the goal here is not to go back in time. It's so interesting because when you look at, I mean, anytime you have, you know, really, really hardcore government regulation and international coordination around issues like this, you're going to have a bunch of unintended consequences. And one thing that feels obvious around this, if these policies were to be rolled out, is that you would effectively create an incentive for millions of individuals or groups globally to be in secret, like trying to find entirely new breakthroughs that are.

7:50Yeah. And again, this incentive already exists, but it kind of pushes a lot of the idea that humans are just going to be like, oh, I'm not I'm no longer going to try to create the God model because.

8:04John Coogan:Yeah. Like there's this, you know, big global organization that's that's sort of policing it. Yeah. I just. I mean, that's the same thing we see with nuclear non proliferation. There's always a discussion about what countries are getting the bomb and how far along are they and wars break out over this. The game's not over just because you create a framework. But ultimately, GPUs and computers are infinitely more widespread than nuclear materials. Yeah, but you still got to marshal them all together. Yes, there's some weird scenario where there's a Python script that's AGI that can run on your laptop.

8:46John Coogan:But I think most people are convinced, at least in this crowd, that scale is a prerequisite. And I mean, we were joking about like, it would be so like, SSI, Ilya Sutskiver's new Neolab, recently got a big cluster from NVIDIA. And we were like, the most bullish thing you could do if you're this secretive Neolab would be like, We're actually selling our compute because we've discovered a more compute optimal way to reach AGI and we don't need a lot of compute. But of course, even Ilya is like, it's time to scale up. I need more compute because it seems like even if he's taking a completely orthogonal approach to innovation and research, he still needs a lot of compute.

9:28John Coogan:And so it does feel like everyone is sort of with the consensus that it's going to be a big building with a lot of energy, big heat signature, definitely visible from space and pretty simple to track, at least in the short term until people start building crazy underground facilities. And then you're back to, you know, nuclear nonproliferation. But their goal is at least to like, you know, try, try. Yeah. And then the other side of this is, does AI development actually become something closer to the Manhattan Project where, you know, a hundred researchers are working with the government in secret because you can't just assume that other countries are going to slow down or do any of these things?

10:14Yeah.

10:15John Coogan:But in general, I think the proposal is don't go back in time. Definitely not stop everything in its tracks. It's a slow down with the goal of scaling gradually. They do actually want to reach super intelligence. They just want to do it by 2040, hence the name of the project. So the goal is gradually scale into top human expert capability around 2035. Now a lot of people are saying, oh, we might get this by 2029, 2028, 2027. And they see that as too fast. So they want to push that out to 2035. then wait five years with AGI and then unlock superintelligence in 2040. This is their initial proposal.

10:57John Coogan:Of course, there's a lot that could change over the next decade. And I think to zoom out, overall, if you're worried about X-Risk, the AI 2040 plan does feel like a concrete path towards slowing down. The conversation definitely gets dragged down into PDOOM estimates and trying to narrow down exactly how a human extinction scenario plays out. And that can be, I feel like that's almost a sideshow because in a democratic society, amongst humanity, it doesn't really matter the mechanics of getting to 10 % PDOOM or any of those. It's just like, if everyone feels that way, something will happen. This is a concrete plan of what that might look like.

11:38John Coogan:And that's valuable to understand in this case. So for the safety skeptics, it's easy to see how this level of control over what you can do with computers is authoritarian or anti-libertarian. Even if we're talking about$100 million computers, there's a lot of people that say, like, I should be able to do math on my computer. I can do whatever I want. Let me do cool things. I'm excited about this. That limits your freedom. It might create regulatory capture for a few major players. it might crash the stock market or delay economic gains that come in the good ending where alignment is solved and x-risk plummets.

12:12John Coogan:You can imagine a situation where in a few years if x-risk fades into the background, you're like, yes, there's still a risk, but it's the same risk that we face every day with like an asteroid hitting Earth. Like no one, it doesn't really change anyone's behavior. That would be sort of the good ending, in my opinion. So it's a balancing act. And for most of these slowdown proposals, I personally have a hard time blackpilling about them. in the sense of like if all of this gets implemented, how frustrated will I be? Like the models are good. I would like better models. I want safe models. But at the same time, like there is this massive capability overhang.

12:49John Coogan:The current models can do a lot of interesting work. We're finding new uses even for like non-leading edge models. There's a lot that can be done. So I don't believe the doom doomers who are dooming about what the doomers are planning. I find that unconvincing right now, but people are starting to lay it out more. Brad Gerstner, Jensen Wong, David Sachs are talking about the other side of this equation. Yeah. There's also a big question around what, do the 2040 people have a point of view on robotic and physical AGI? because it seems like even if you pause efforts towards RSI, well, if we add billions of robots into the world that are just running on today's model, that also presents today's, like, you know.

13:40John Coogan:I don't think they're worried about that. Yeah, but to me that - A billion robots with GPT-6 level intelligence and years of alignment work that is currently happening, that's fine. It's the next, next, next thing, the super intelligence, the thing that might have its own goals. I mean, we're talking to someone from OpenAI's robotics team, hooking up Astra to a robot, a paintbrush, and a camera. We talked about it earlier, painting. I don't think we're at a point where that poses a risk. It's the next model. It's the model with its own volition, basically, which a lot of people still aren't seeing.

14:22John Coogan:They're just like, yeah, the models keep getting better, but they seem to follow your instructions, sometimes too much, and then you need to worry about the paperclip scenario. But it's not that they want to do their own thing necessarily. I don't know. But people are going back and forth on this. Clem over at Hugging Faces, sorry, but asking Jacob about AI extinction risk is like asking your AC guy about climate change. Not saying it's necessarily uninteresting or wrong per se, but let's keep things in perspective and hear from the full range of expertise across the ecosystem. And Nathan Lambert says, banger.

14:57John Coogan:What was our take? AC guy might be right about how to change. Well, my AC guy would be like, I don't really know about that, but I just want to make sure when you're hot that we can run this AC cool. Yeah, the AC seems... I don't know about all that mumbo jumbo, but when it's a hot summer day, I don't want you to be worried about the heat, brother. I like that. Yeah, this is kind of an unnecessary shot at AC guys. AC guys are important. I don't know. It's funny. What did Bernie Sanders have to say? Is this real? Entities shall be subject to the corporate death penalty, and persons shall be subject to not more than 20 years in prison if they don't pause AI development.

15:47John Coogan:That seems pretty easy to comply with. I don't know. I guess how do you define AI development? Is prompt engineering AI development? Then you get caught because your model sort of did a little prompt engineering in the final stage and then you're guilty of this. Yeah, that could be a negative knock-on effect, I guess. It does seem aggressive. but his overall proposal is banning artificial superintelligence so no person or entity may develop or deploy superintelligent AI systems. He defines artificial superintelligence as an artificial intelligence system that exhibits or can easily be modified to exhibit capabilities that match or exceed human cognitive performance and capabilities across a broad range of domains or tasks.

16:34John Coogan:It sounds like the mission statement. It sounds like the explicit goal of like 17

16:48John Coogan:I don't like overthrowing or undermining the US government. I don't like overthrowing or undermining the I don't like overthrowing or undermining the US government. Pausing advanced AI development until a new federal AI regulatory body is up and running and then the new cabinet level federal agency will monitor frontier AI systems at all stages of the life cycle. Supervise the removal and the new of dangerous capabilities and supervise the destruction of artificial super intelligence. We're coming for it, which is similar to the corporate death penalty is a line that you don't hear a lot. Right.

17:21Usually, usually these companies just, you know, go bankrupt and wind down. But corporate death penalty goes pretty hard. Metal. It's kind of it's kind of metal.

17:32John Coogan:Yeah, you kind of got me with that one. Yeah. Yeah, rough, rough, rough situation. We'll see where it goes. Jamie Cox over at FluidStack, the co-founder of FluidStack, a compute provider, shared his convictions, sort of pushing back on a lot of this, saying that he thinks America should build more. They're pro-freedom, pro-democracy. They believe AI will bolster human flourishing. We support simple, clear, enforceable regulation frameworks that set simple requirements proportion to capabilities and risk with clear responsibilities and no unnecessary barriers to competition. Yeah, the real, you're going to see a lot of pushback from people who are like, the regulatory stuff is going to be like these 10 companies and I'm going to be number 11.

18:26John Coogan:And I'm basically getting the corporate death penalty then because I didn't make the cut to be one of the regulated, one of the approved companies. I'm still early in my stage. So there's a lot of nervousness, I'm sure. But Will Manitis said, we believe AI will make everyone rich, healthy, and free is novel. and interesting comms from the frontier uh he's he's endorsing this and i i agree i like i like this uh i like these convictions i think it's generally like a positive direction to move in not a direct response to the the proposals that are going out but we're gonna get a whole lot more of them uh where do you want to go to next uh over on tick tock they're sharing a photo of the whistleblower and saying in every worldwide disaster movie there's a dude that looks just like this that nobody listened to he really does look like he does look like an actor here yeah he looks good um but people are people are all and please stop calling him scary potter i've been seeing people i've been seeing people over on x calling him scary potter that's the goal the goal is to is to wake up china wake up congress wake up everyone door dash has entered uh the conversation They say two years at DoorDash.

19:42I do not say this lightly. We are extremely close to the burrito arriving before you decide you want it. We are not asking for a ban. We are asking for a pause. I don't know why they would ask for a pause.

19:54John Coogan:Yeah. That seems like very, very aligned to humanity and to their business, which I think is fantastic. The doom is very much contained to the frontier lab work. Everyone in the application layer who's applying the models, diffusing them, they're like, I can't get this thing to work right. I got to get forward deployed engineers to teach people how to use this thing. Everyone deeper in the stack is like Jim Rainbow over Jim O 'Reilly or Jim Riley. Yeah, Jim Riley. Jim Riley. O 'Reilly Auto Parts. Jim Riley over at Charleston AI says it's a whole lot of mumbo jumbo. That was your word. He just he's just happy to get, you know, eight hours back.

20:36John Coogan:Yeah, yeah, yeah. And then, yeah, everyone deeper in the supply chain like Jensen and all the different semiconductor manufacturers are are not particularly on this side. And then you also have Wall Street, who's just like, what's the enterprise acceleration? So lots of different groups around the table that need to be brought on board to this movement. The discourse truly is fascinating. So what is Tyler Cowen calling for? He's banging the table saying, bet on this. Tyler Cosgrove. Are you named after Tyler Cowen? Is that your namesake? Tyler Cowen says, if you have very pessimistic fears or predictions about AI, name the market prices that will support or confirm them.

21:21John Coogan:This is what taking this seriously means. And I love Tyler Cowen. I'm not sure this matters because you're not going to be around to collect it. Yeah, this is what Deep Dish employer says. Tyler, why would short-term existential risk affect market prices in any meaningful way? Spell out the exact mechanism. Contracts that pay out if everyone dies aren't worth anything to me. Yeah, I don't know. Yeah, I think he's just, he's calling for like, he wants to like make a falsifiable claim. Like, is this, am I able to tell if your claim is like true or false? And so it's like very hard with these scenarios.

21:56John Coogan:Isn't it an unfalcible claim though, just by definition? And like, you just have to like accept that and move on. Yeah, but then it's like so hard to have any like real discussion. Was nuclear any different? Like the threat of nuclear apocalypse, the threat of World War III, this was a very motivating factor for decades, most of the 20th century. people made real decisions based on it, based on where to do business and where the conflicts were going to be and the motivation for nuclear treaties and non-proliferation. You can make financial decisions, like think about nuclear war, right? You have a bunker.

22:38That's like a decision you make. Bunky? Is anyone making like AI bunkers? No, because they think it's going to be so totalizing that the bunker actually doesn't do anything. Exactly.

22:46John Coogan:Yeah. So if you think it's so totalizing, then you don't make the bunker. And so saying, hey, you don't have a bunker is not proof that the person doesn't believe what they're saying. There is the other side of this, which is Paul Cristiano, who has been worried about risk. He recently just joined the board of OpenAI. And Tyler Cowen has this quote, if you're a doomer, or why aren't you short the market? And Paul Cristiano is a two X levered long, and he's short the, isn't he short the bond market or US treasuries, right? So he has 5 % of his net worth in Tesla, 90 % of his net worth in AI bets, and 100 % of his net worth in normal investments.

23:31John Coogan:No Tesla options. That sounds like a scary place with lottery ticket biases and the crazy Tesla investors. And then Eliezer Yudikowsky says, am I correctly understanding? You're 2x levered. And Paul Christiano says, yeah. And so he says he's personally short the US 30-year debt. I think that just means you have a mortgage. This doesn't seem like a doom-based bet. This seems like this bet also pays out in just the good ending and AI is real and delivers value. So your AI bets perform well, the market performs well, and money slides from US debt to data centers and AI build-out debt or something like that.

24:08John Coogan:So So he is putting his money where his mouth is, but it doesn't feel like a representation of doom by any means. People have been creating AI agents of fruit flies. Have you seen this? Yes. My understanding is Google basically mapped out all of the neurons in a fruit fly. Yeah, like the entire... In 3D, they got the entire structure, and now people have been able to recreate that in software, in a simulation. Yes. I think in theory, you can replicate all of the flies' decisions or whatever. It's like movements. Yes. And this is notable because many people have said, I have the mind of a fruit fly.