AI's Rising Risks: Hacking, Virology, Loss of Control — With Dan Hendrycks

27:48And I don't see – it's very unclear when those arrive. Could be a few months from now. Could be a year from now. It's a little less – I'm a little more suspicious. Maybe it would even take two years for that. So that's something for us to get prepared for, figure out how we're going to deal with that, try and make safeguards increasingly robust to people trying to maliciously use it in those ways.

28:16But, yeah, I think much of the risk source comes from being able to take one of these AIs, let's say one of these DeepSeek AIs, let's say it's DeepSeek agent version, and it's able to actually do these cyber attacks. Then you could just run 10 ,000 of them simultaneously. And then some rogue actor could have it target critical infrastructure. Then this is causing quite severe damage. So for critical infrastructure, this could be like have it reduce the detector or the filter in a water plant or something like that. Then the water supply is ruined. Or you could target these thermostats in various homes because often some of the more advanced ones are connected to Wi-Fi.

29:01And then you sort of turn them up and down simultaneously. And this can just ruin transformers and blow them, and then they take multiple years to replace. Things like that. But they aren't capable of doing that sort of thing currently. So it's more of an on-the-horizon type of thing. But I'm not feeling the urgency with that currently. I'm more concerned about it. I think there's more the geopolitics of this, like making sure that states are aware of what's going on in AI, like they're at least able to follow the news and things like that in some capacity. I think things like that feel somewhat more urgent to me than trying to address cyber risks.

29:53There are things to do, though, and I think we should create incentives beforehand. Maybe I'm too much of an optimist for my own good. But when I hear you talk about this, I also get a little bit excited about the capabilities of these programs. Because, for instance, if AI can enhance the function of a virus, AI can probably create a vaccine and make medical discoveries. If AI can hack into the infrastructure of some country, find exploits and turn the thermostats up and down, then AI could probably do incredible amounts of very beneficial coding and computer work for humanity. So if we do get to that point, it seems to me like there's going to be these maybe two poles here.

30:31One is the potentially scary and destructive stuff that you can mitigate with some of the controls that you talked about, but also amazing opportunity. Yeah, so it's – and the thermostat thing was for messing with the electricity and that causing strain on the power grid and destroying transformers. Just for clarification in case – but yeah, I think you're pointing at that it's dual use. So I'm not saying AI is bad in every single way. It's like other dual use technologies. Bio is a dual use technology. It can be used for bioweapons. It can be used for healthcare. nuclear technology is dual use.

31:10There's civilian applications for it as well. And chemicals too. And we have managed all of those other ones by selectively trying to limit some particular types of usage and restricting the capabilities of rogue actors to some of these technologies and making sure there are good safeguards for the civilian applications. And then we can actually capture the benefits. So it's not an all or nothing type of thing with AI. It's what are surgical restrictions one can place so that we can keep capturing the benefits. And so, for instance, with virology, that's a matter of you add the safeguards and then the researchers who want access to those can speak to sales.

31:53That's basically a resolution of that problem, provided that you have the models kept behind APIs. And so now on this dual use part, though, there's an offense-defense balance. So for some applications, it can help, it can hurt. And maybe it helps more than it hurts, or maybe it will hurt more than it will help. So in bio, I think that is offense-dominant. If somebody creates a virus, there's not necessarily a cure that it will immediately find for it. If it would help a rogue actor make a somewhat compelling virus, now that could be enough to cause many millions to die. And it may take months or years to find a cure.

32:36There are many viruses for which we have not found cures yet. And for cyber, in most contexts, there's a balance between offense and defense, where if somebody can find a vulnerability with one of these hacking AIs, then they could also use that to patch the vulnerability. There There is an exception though where in the context of critical infrastructure, there the software is not updated rapidly. So even if you identify various vulnerabilities, there will not necessarily be a patch because the system needs to always be on or there are interoperability constraints or the company that made the software is no longer in business, these sorts of things.

33:15So critical infrastructure is a sitting duck and so in that context, cyber is offense dominant. But in normal context, there's roughly a duality. And for virology, I think that's largely offense dominant. So before we go to the nation state element of this, I need to ask you a question about the actual research houses themselves. Every research house says they're a concern with safety. From open AI to XAI, everything in the middle. Maybe not DeepSeq. We'll get to DeepSeq. Yet, they're the ones that are building this technology. And I find it a little strange that you have companies that are saying, it's weird, we have to build this and advance this technology so we can keep people safe.

33:59I never really understood that message. Yeah, I don't know if it's to say that we need to keep people safe. I think it's more that the main organizations that have power in the world now are largely companies. And so if one's trying to influence the outcomes, one basically needs to be a company is how many of them will reason. They'll think that, yeah, you could be in civil society or you could protest, but this will not determine the course of events as much. So there's sort of many of them are buying themselves the option to hopefully influence things in a more positive direction. But most of the effort will be to stay competitive and stay in this arena.

34:41So I think over 90 % of the intellectual energies that they're going to spend is actually, how can we afford the 10x larger supercomputer? And that means being very competitive, speeding this up, and making safety be some priority but not necessarily a substantial one. So I do think there is sort of an interesting contradiction or something that looks like a contradiction there. But I think if we think back to nuclear weapons, nuclear weapons, nobody wants nuclear weapons. If there'd be zero on Earth, fantastic. That would be a nice thing to have if that would be a stable state. But it's not a stable state.

35:19One actor may then develop nuclear weapons and they could destroy the other. So this encourages states to do an arms race and it makes everybody all collectively less secure. But that's just how the game theory ends up working. So you get a classic, what's called a security dilemma. Everybody's worse off collectively. But – and even if you took it seriously, you say, yes, nuclear technology is dual use and potentially catastrophic. And we need to be very risk conscious about it. You can agree with all those things, but you still might want nuclear weapons because other parties will also have nuclear weapons.

35:57And unilateral disarmament in many cases just didn't make game theoretic sense. in the way that an individual company pausing their development while others race ahead doesn't make game theoretic sense. So I think this just points to the fact that game theory is kind of confusing, and so you're getting some things that are seeming contradictions that if you use a nuclear analogy, go, yeah, I suppose that makes sense, and it's just kind of an ugly reality to internalize. Doesn't that discount the fact that these companies, if they want to influence the way things are going, they are going to be it's like you're one and the same.

36:35Yes you're influencing but without you this wouldn't be moving as fast as it is. It is interesting for instance think about Elon Musk right? Obviously he has you in two days a week to work on safety inside XAI but he's also putting together what million GPU data centers to build the biggest baddest LLM ever I don't understand Well if he didn't then he would be having less influence over it so it's it's not something that I would envision everybody would just sort of voluntarily pause. So subject to companies not sort of voluntarily rolling over and dying, then what's the best you can do subject to those constraints?

37:13But the competitive pressures are quite intense, such that they do end up prioritizing focusing on competitiveness and other priorities, like what's the budget for safety research, it will be generally lower than would be nice to have if this were a less competitive environment. Do you think Elon is more interested in restoring this original vision that he had for OpenAI, making everything open source, making it safe, I would imagine? He founded OpenAI with Sam Altman as sort of a beachhead against Google. because he was afraid of what Google was going to do with this technology. So I'm curious if you think that XAI is along that mission, or is he more interested in the sort of soft cultural power that comes with having the world's best AI?

38:09For instance, like you can change the way that it speaks about certain sensitive political issues. It can be anti-woke, which we all know is sort of where Elon stands. So what do you think his true interest lies in building XAI? Well, I think the – and I won't position myself as sort of speaking on behalf of them here. We won't put you as Elon spokesperson, but you are in there. Yeah, so I think that the mission is to understand the universe. And so this means having AIs that are honest and accurate and truthful to improve the public's understanding of the world. So we will be getting in a very fast-moving, trying situation with AI if it keeps accelerating.

38:53And so good decision-making will be very important. And us understanding the world around us will be very important. So if there are more features that enable truth-seeking and honesty and good forecasts and good judgment in institutional decision-making, those would be great to have with the hope is that Gra could help enable some of that so that civilization is steered, steering itself more prudently in this potentially more turbulent period that's upcoming. That's one read on the mission statement. But I think that's the objective of it is understand the universe and their different sub objectives that that would give rise to.

39:36I think it's ability to help culture process events without censorship or political bias one way or the other is a stated objective. And I think that would be indispensable in the years going forward. Do you buy that that's what they're doing? Because we also heard the same thing from Elon when it came to buying Twitter, now X. but I think community notes has been quite good. But that was something that was built under Jack Dorsey. I'm not going to take sides here. I'm going to just observe empirically what I've seen. We know that Substack links have been deprioritized because it was seen as a competitor with Twitter.

40:22We know that Musk, I think according to reporting, changed the algorithm to have his tweets show up more often and his tweets took a strong stance towards supporting Donald Trump in the election. So to me, the idea that like hearing again from Elon, and again, look, I respect what Elon's done as a business person, but hearing again that he has a plan to make a culturally relevant product that's free of censorship and politically unbiased, I don't know if I believe that anymore. So I don't know about some of the specific things such as the waiting thing or something like that, profile things, for instance.

41:01I think that overall in terms of cultural influence and people being more disagreeable and doing less self-censoring has been successful. I think that was the main objective of it. So I think that X had a large role to play there. So I don't know. I think in terms of shaping discourse norms in the U.S., that seems to have been successful in my view. Yeah, I'm not saying pre-Elon Twitter didn't censor, which is probably the wrong word because that's usually from the government, didn't sort of shape the definition of speech to its own liking. It obviously had a progressive approach and moderated speech on a progressive approach.

41:47I just don't think Elon is not using his own influence when it comes to how he runs X. But you and I could speak about this forever. Yeah, this isn't even my sort of wheelhouse as much. But yeah, I mean, it's sort of like since, you know, you brought it up. So, oh, OK. I mean, just the non-biased and truthful thing. So it's worth it. I mean, it is if there are like ways in which it's like extremely biased one way or the other. That's that's useful to know. This is a thing that is continually trying to be improved, at least for XAI's GROC. And I think that all of the sort of product offering could get quite better at this.

42:36But I'm not speaking as a sort of representative there or anything like that. But I guess maybe in my – I guess right now in my personal capacity, I think that there's things to improve on for all these models. in terms of their bias. All right. We agree on that front. You hinted at it previously, but you talk a little bit about how companies, basically how you don't think it's a good idea for there to be an arms race here. And certainly there is one between the U.S. and China. We know the U.S. has put export controls on China. China has in some ways gotten around them through very creative procurement processes that go through Singapore.

43:16We can probably say that with a pretty good degree of confidence. Then, of course, we see the release of DeepSeek and some other AI applications from China. And everyone's trying to build the better AI so that they have the soft power, like we spoke about, to effectively, A, control, to influence culture across the world. But also it's an offensive capability and defensive, like you're saying. if your country has the ability to manipulate viruses or to do cyber hacks, you become more powerful and you get to sort of, you know, potentially put your view of the world, implant your view of the world on the way that it operates.

43:51You have a paper out that's sort of arguing against this arms race. It's called Superintelligence Strategy. It's with you, Eric Schmidt, who we all know, former CEO of Google. I think he just started. He's taking over a drone company. So you Alexander Wang, the current CEO of ScaleAI, who's been formally on this show. Talk a little bit about why you don't think it's a good idea for countries to pursue this arms race. You say it might be leading us to mutually assured AI malfunction, not mutually assured nuclear destruction. I think that's what you get that from. Yeah, so the strategy has three parts, one of which is competitiveness.

44:30but we're saying that some forms of competition could be destabilizing and that you may be irrational to pursue it because you couldn't get away with it. So in particular, making a bid for superintelligence through some automated AI research and development loop could potentially lead to one state having some capabilities that are vastly beyond another state's. If one state gets to experience a decade of development in a year and the other one is the year behind, then this results in a very substantial difference in the states' capabilities. So this could be quite destabilizing if one state might then start to get an insurmountable lead relative to the other.

45:23So I think that form of competition would be very dangerous and because there's a risk of loss of control and because it might incentivize states to engage in preventive sabotage or preemptive sabotage to disable these sorts of projects. So I think states may want to deter each other from pursuing superintelligence through this means. And this then means that AI competition gets channeled into other sorts of realms, such as in the military realm of having more secure supply chains for robotics, for instance, and for AI chips having reduced sole source supply chain dependence on Taiwan for making AI chips.

46:11So states can compete in other dimensions, but them trying to compete to develop superintelligence first, I think that seems like a very risky idea, and I would not suggest that because there's too much of a risk of loss of control and there's too much of a risk that one state, If they do control it, uses it to disempower others and affects the balance of power far too much and destabilizes things. So – but the strategy overall, think of the Cold War. Before you go on the strategy, like my reaction to that is good luck telling that to China. So I think it's totally – so for deterrence, I think if the U.S.

46:50were pulling ahead, both Russia and China may have a substantial interest in saying, hey, cut this out. pulling ahead to develop superintelligence, which could give it a huge advantage and an ability to crush them. They'd say, you don't get to do that. We are making a conditional threat that if you keep going forward in this because you're on the cusp of building this, then we will disable your data center or the surrounding power infrastructure so that you cannot continue building this. I think they could make that conditional threat to deter it. And we might do the same, or the U.S. might do the same to China or other states that would do that.

47:28So I don't see why China wouldn't do that later on. Right now, they're not just thinking about superintelligence and advanced AI. So this is more of a description of the dynamics later on when AI is more salient. But it would be surprising to me if China were saying, yes, the United States, go ahead, do your Manhattan project to build superintelligence, come back to us in a few years, and then tell us, you can boss us around because now we're in a complete position of weakness and we'll be at your mercy and we'll accept whatever you say or tell us to do. I don't see that happening. I think they would just say, move to preempt or deter that type of development so that they don't get put in that fragile position.

48:13Are you in like the Eliezer Yudkowsky camp of bombing the data centers if we get to superintelligence? Well, so I think I'm advocating or pointing out that it becomes rational for states to deter each other by making conditional threats and by means that are less escalatory, such as cyber sabotage on data centers or surrounding power plants. I don't think one needs to get kinetic for this. And I think that if discussions start earlier, I don't see any reason things need to be escalating in that way or unilaterally actually doing that. We didn't need to get in a nuclear exchange with Russia to express that we have a preference against nuclear war.

49:00So I think - Thank goodness. So indicating or making conditional threats through deterrence seems like a much smarter move, then, hey, wait a second, what are you doing there? And then bomb. That seems needless. Yeah, I'm not into that solution either. But what you're talking about is sort of assuming that there will be a lead that will be protectable for a while. But everything we've seen with AI is that no one protects a lead, right? Well, if there's... So one difference is that when you get to a different paradigm, like automated AI R &D, the slope might be extremely high, such that if the competitor starts to do automated AI R &D a year later, they may never catch up just because you're so far ahead and your gains are compounding on your gains.

49:55Sort of like in social media companies, Eric will use this analogy, where if one of them starts blowing up and growing before you started, it's often the case that you won't be able to catch up and they'll have a winner-take-all type of dynamic. So right now, the rate of improvement is not that high or there's less of a path for a winner-take-all dynamic currently. But later on, when you have the ability to run 100 ,000 AI researchers simultaneously, this really accelerates things. Maybe open AI has got a few hundred, maybe it'll say 300 AI researchers. So going from 300 AI researchers to orders of magnitude more world-class ones create quite substantial developments.

50:44This is something that isn't new. This is something that Alan Turing and the founders of computer science had pointed out, that this is a natural property of when you get AIs at this level of capability, then this creates this sort of recursive dynamic where things start accelerating extremely quickly and quite explosively. Okay. We managed to spend most of our conversation today talking about present risks or risks in the near future. We should focus a little bit more on intelligence explosion and loss of control, and we're going to do that right after the break. Did you know your credit card points and miles can lose value to inflation?

51:24Credit card companies often reduce the redemption value of your points and miles. Now, imagine a credit card with rewards that can grow in value. With the Gemini credit card, you can earn Bitcoin or one of over 50 other cryptos instantly with no annual fee. Every swipe at the store or gas pump earns you instant rewards deposited straight to your account. Plus, sign up now for a$200 Bitcoin bonus to kickstart your rewards. Visit Gemini.com slash card today. Check out the link in the description for more information on rates. Again, if you're looking to invest in Bitcoin but don't know where to start, the Gemini credit card makes it easy.

52:03The Gemini credit card is issued by WebBank. In order to qualify for the$200 crypto intro bonus, you must spend$3 ,000 in your first 90 days. Some exclusions apply to instant rewards in which rewards are deposited when the transaction posts. This content is not investment advice and trading crypto involves risk. The Gemini credit card cannot be used to make gambling-related purchases.

52:28You're used to hearing my voice on the world bringing you interviews from around the globe. And you hear me reporting environment and climate news. I'm Carolyn Buehler. And I'm Marco Werman. We're now with you hosting The World Together. More global journalism with a fresh new sound. Listen to The World on your local public radio station and wherever you find your podcasts.

52:55We're back here on Big Technology Podcast with Dan Hendricks. He is the director and co-founder of the Center for AI Safety. Dan, it's great speaking with you about this stuff. Let's talk a little bit. You've been sort of talking about it in the first half, but I want to zero in here on this idea of intelligence explosion or what you talk about as basically having AI autonomously improve itself. just talk through a little bit about how that might happen and whether you see that being something that is actually probable in our future yeah i mean the the basic idea is just imagine automating one ai researcher one world-class one then there's a fun property with with computers which is there's copy and paste so you can then have a whole fleet of these it's it's well you With humans, if you just have one of them, maybe they'll be able to train up somebody else who has a similar level of ability.

53:51So this adds a very interesting dynamic to the mix. And then you can get so many of them proceeding forward at once. And AIs also operate quite quickly. They can code a lot faster than people. So maybe you've got 100 ,000 of these things operating at 100x the speed of a human. How fast will that go? Maybe conservatively, let's say it's just overall 10xing research. But 10xing research would mean, say, like a decade's worth of developments in a year. So that telescoping of all these developments makes things pretty wild and means that one player could possibly get AIs that go from very good, world class to being vastly better than everybody at everything and a super intelligence.

54:45something that towers far beyond any living person or collective of people. So if we get an AI like that, this could be destabilizing because it could be used to develop a superweapon potentially. Maybe it could find some breakthrough for anti-ballistic missile systems, which would make nuclear deterrence no longer work or other types of ways of weaponizing it. So that's why it's destabilizing. And so states then, if they're seeing, oh, don't run this many AI researchers simultaneously in these data centers working to build a next generation or superintelligence, because if you do so, then that will put us in – that will make our survival be threatened.

55:34And so them saying, them deterring that would help them secure themselves. And they can make those threats very credible currently. And I think we'll continue to be able to have these threats be credible going forward. So this is why I think it might take a while for superintelligence to be developed because there'll be deterrence around it later on. And then maybe in the farther future, there could be something multilateral. But that's speaking quite far out in very different economic conditions. In the meantime, with the AIs that we'd have in the future, those could still automate various things and increase prosperity and all of that.

56:14So we'd still have explosive economic growth if you had something that was just at the average human level ability running for very cheap. So I think that those are some of the later stage strategic dynamics. And I don't think we can get away with or I don't think any state could get away with trying to build a super intelligence, go build a big data center out in the middle of the desert, a trillion dollar cluster, bring all the researchers there and let's not invite the other states to go. What do you think you're doing here? You were at the White House yesterday. Well, this is largely just sort of speaking about some of these strategic implications.

56:56Are they receptive? Yeah. I mean it's a – this isn't a – there's always interest in thinking what are some of the later term dynamics, what things should happen now and whatnot. But this is – yeah, I think when people think White House, it sounds – Well, we're the president. So there's the – well, yeah. So there's the Eisenhower building, which is part of the White House, kind of not. But that's where everybody works in the White House. And I think some of the things we're speaking about here, like virology advancements, things like that, But there's just a lot of things to speak about and think what things make sense or what things to keep in mind going forward.

57:47Yeah, I guess I'd rather an executive branch paying attention to this stuff than not. Yeah, that's right. Yeah, and what are the sort of ways that help maintain competitiveness? Because how people normally think about this, they'll think it's all or nothing and good or bad thing. And instead we're saying, no, it's dual use. So that means there are some particular applications that are concerning and there are other applications that are good and you want to stem the particularly harmful applications and what are ways of doing that while capturing the upside. Right. Okay, so the intelligence explosion part of this conversation inevitably brings up the loss of control part, where to me I think the thing that when people think about AI harm, they're always worried that AI is going to escape the simulation or whatever it is and act on its own and try to basically ensure that it preserves itself.

58:40We've seen it recently, and I think I brought this up at the beginning of the show, where Anthropic has done some experiments where the AI has run code to try to copy itself over onto a server if it thinks that its values are at risk of being changed. Is this – so it's fun to think about, but it's also like probably just probability. Like if you run it enough times because it's a probabilistic engine. If it was like, oh, it's only one in a thousand of them intend to do this. Well, if you're running a million of them, then you're basically certain to get many of them to try and self-exfiltrate.

59:13And so are you worried that this self-exfiltration is going to be a thing? I think from a recursive automated AR &D thing, I think that has really substantial probability behind it of a loss of control in that situation. So you're worried about this? So there's that. But I would distinguish between that and these sort of things that are not super intelligences or things that are not coming from that sort of really rapid loop. and like the currently existing systems. I think that the currently existing systems are relatively controllable. Or if there is some very concerning failure mode, we have been able to find ways to make them more controllable.

59:57For instance, for bioweapons refusal, we used not to be able to make robust safeguards for them two years ago. But we've done research with methods such as called circuit breakers and things like that. And those seem to improve the situation quite a bit and make it actually prohibitively difficult to do that jailbreaking. And so maybe we'll find something similar with self-exfiltration. So I think people generally want to claim that, like, oh, current AIs are not controllable. And I think that they're not highly reliably controllable. They're reasonably controllable. Maybe we could get some – or it seems plausible that we'll get to have increasing levels of reliability.

1:00:37and so I'm sort of reserving judgment. It'll depend more on the empirical phenomena so I think everybody should research this more

1:00:50and we'll sort of see what the risks actually are but there are some that seem less empirically tractable or things that can't be empirically solved like this loop thing like how are you going to you can't run this experiment 100 times or something like that and make it go well. you're making a huge attempt at building a superintelligence and has destabilizing consequences. This isn't something that's totally unprecedented. And for that, you have more of a one chance to get it right type of thing. But with the current systems, we can continually adjust them and retrain them and come up with better methods and iterate.

1:01:24So it is concerning. It would not surprise me if this would really start to make AI development itself extremely hazardous. instead of just the deployment, but instead inside the lab, like you need to be worried about the AI trying to be breaking out sometimes. That's totally in the realm of possibility. But yeah, I could see it going either way. Yeah, I mean, this personally freaks me out because, yeah, if you see the AI trying to deceive evaluators, for instance, or you see the AI trying to break out, you really can't trust anything it's telling you. And we had Demis Asabes on the show a little while ago, And he's basically like, listen, if you see deceptive behavior from AI, if you see alignment faking, you really can't trust anything in the safety training because it's lying to you.

1:02:13There is truth to that. Are you seeing deceptiveness at Grok, by the way? Oh, yeah, yeah. So we have a paper out last week. We're just measuring the extent to which they're deceptive. and in the scenarios we have, like all the models were in these sorts of scenarios under slight pressure to lie, not being told to lie, but just some slight pressure. Then some of them will like lie like 20 % of the time, some of them like 60 % of the time. So they don't really have this sort of virtue sort of baked into them, the virtue of honesty. So I think we'll need to do more work and we'll need to do it quickly.

1:02:49So I'm sort of speaking in a more nonchalant way about this, but I can't get worked up at every single risk because they're also just, ooh, be at 11 all the time. So there are some that I'm putting in different tiers than other risks. And this is a more speculative one. We've seen these sometimes get surprisingly handleable. But, yeah, it could end up making things really, really bad. We'll see. We'll do things about it to make that not be the case. Okay. Thank you. Two more topics for you, then we'll get out of here. The Center for AI Safety. Who's funding it? Well, so there's not sort of one funder.

1:03:31It's largely just various philanthropists. The main funder would be Jan Talon and Jan Talon, who's a Skype co-founder. There's a variety of other philanthropies or philanthropists. It's the generally for – so, for instance, Elon doesn't – I've never asked him to fund the center. So that isn't to say I don't get any money from Elon. My appointment at XAI, I get$1 a year. At Scale AI, I've increased my salary exponentially to where I get$12 a year,$1 per month from Scale. But I'll try not to – or I'll try to avoid having some complicated relations with them just so that I can – You're doing the work?

1:04:28Not feel on behalf of any of them in particular. Okay. So you're basically doing the work for them for free? Well, but it's useful. Right. It's useful to do. And I mean, yeah, I think the main objective is just try and generate some value here as best as one can.

1:04:48by reducing these sorts of risks. Yeah, I think it's a good arrangement because it enables me to do have a choose-your-adventure type of thing. Like, oh, now I think the politics or geopolitics, this is more relevant. So now I can go off and learn about this for some months and then work on a paper there. Compared to if it's like, no, you've got to be coding 80 hours a week. That's your job. That would be quite restrictive. I couldn't be speaking with you. I'm glad you're here. Thank you, Alex Wayne. So let's talk a little bit about this funding, because I think that after Sam Altman was fired and then rehired at OpenAI, there was a sort of skepticism around effective altruism's impact on the AI field.

1:05:35Even Jan Tallinn, I'm reading from his statements, right after the OpenAI governance crisis, highlights the fragility of voluntary AI-motivated governance schemes. So the world should not rely on such governance working as intended. Now, Jan is, of course, associated with EA. EA is basically leading the conversation around AI safety. Is that good? So I think that in terms of Jan, I think he's funded organizations that are EA-affiliated. I don't know if he'd call himself that, but whatever. People can ascribe labels how they'd like. I think that the, I mean, I've tweeted that EA is not equal to AI safety.

1:06:18I think that EA community generally is insular on these. So I lived in Berkeley for a long time when I was doing my PhD. And there was sort of a school, a sort of AI risk school that was, had very particular views about what things are important. So malicious use, for instance, when I was talking about malicious use in the beginning of this thing, they're historically really against that. Yeah, yeah, they're only loss of control. Don't talk about malicious use. That's a distraction. And so that was annoying because I'd always been working on robustness as a Ph.D. student where the main thing was malicious use.

1:06:58so yeah I ended up leaving Berkeley even before graduating just because of the sort of relatively suffocating atmosphere and the sort of central focus on there'd be some new fad and you'd have to get interested in that elk eliciting late knowledge this is the important thing that you have to focus on you have to focus on inner optimizers there's lots of these speculative empirically fragile things so for instance this alignment faking stuff that you're seeing. There's some concern there, but I'm not totally sold that this is a top-tier type of priority. But in these communities, this is all that matters currently, roughly speaking.

1:07:39This involuntary commitments from AI companies. I think voluntary commitments from AI companies are also a distraction because the companies will, you should expect most of them by default to just break those sorts of commitments if they end up going up against economic competitiveness. So I think it's a distraction, relatively. And so I think there are many people who think that EA's broadly, their influence on this sort of thing, has not been overall positive. I think at least for me and other sorts of researchers in this space who've been interested in AI risks, the amount of pressure to adopt some particular positions on this would be extraordinarily high and I think quite destructive.

1:08:26So I'm very pleased now that in the past year or so there's been a lot more diversity of opinion which has been quite important. And I think this is just because the broader world is getting more interested in AI. So a lot of these fixation on this is the one particular risk, this is the most important risk and everything else is distraction, that just doesn't work when you're speaking with the – or interfacing with the real world. There's a lot of complications. And AI is so multifaceted. So you can't – in your risk management approach, you can't just be focusing on one of them. Right. So you're not an effective altruist.

1:09:08I don't think of myself as that. I don't particularly get along with this school of thought, this sort of Berkeley AI alignment monolith. And I'm pleased that people can be more independently operating in this space now, which I don't think was the case for many, many years, including basically the entire time I was doing my PhD. And there'll be many people like Dylan Hadfield-Manel, a professor at MIT, who was also at Berkeley at the time, very suffocating. Rohan Shah, researcher, very suffocating. They all feel this way. Okay. Let's bring it home. We've been talking for more than an hour about AI safety as if it's controllable.

1:09:55But open source is really putting up a pretty valiant effort in this field, keeping pace with the proprietary labs. And, of course, open source is not controllable. What do you think about that? I mean, we just saw DeepSeek, not to go back to it all the time, but it effectively equaled the cutting edge at the proprietary labs and put the weights on its website. So how can we possibly have a relationship of safety with AI if open source is out there exposing everything that's been done? So I haven't been endorsing open source historically, but I thought that releasing the weights of models didn't seem robustly good or bad.

1:10:39So I sort of was like, it's fine. It seems to have complicated effects. There's an advantage to it, which it helped with diffusion of the technology so that more people would have access to it and sort of get a sense of AI and sort of increase literacy on this topic and just increase public awareness. and get the world more prepared for more advanced versions of AI. So that's been my historical position, but this depends on it should always proceed by a cost-benefit analysis. So if, for instance, they have these cyber capabilities later on, yeah, I think that would be a potential place to be drawing the line on open-weight releases personally, in particular the ones that could cause damage to critical infrastructure.

1:11:34You could still capture the benefits by having the models be available through APIs, and if they're like software developers, they have access to these more cyber-offensive capabilities. But if they're a random user, they don't. If they're a random faceless user, they don't.

1:11:50And likewise for virology. Once there's consensus, once the capabilities are so high that there's consensus about it being expert level in virology, I think that would be a very natural place to be having an international norm. Not saying a treaty because it would take forever to write and ratify, but to a norm against open weights if they are expert level virologists. For the same reasons that we had the Biological Weapons Convention. Russia and or the Soviet Union and the U.S. got together for the Biological Weapons Convention. US and China did as well. We also coordinated on chemical weapons with the Chemical Weapons Convention and the Nuclear Nonproliferation Treaty.

1:12:30States find it in their incentive to work together to make sure that rogue actors do not have extremely hazardous, potentially catastrophic capabilities like chem, bio, and nuclear inputs. So I think something similar might be reasonable for AI when they get at that capability threshold. Dan, I am at once kind of reassured that people are thinking about this stuff, but also more freaked out than I was when we sat down. But I do appreciate you coming in and giving us the full rundown of what to be concerned about and what maybe not to be as concerned about as we think about where AI is moving next.

1:13:14So thank you so much for coming on the show. Yep. Yep. Thank you for having me. This has been fun. Super fun. If people want to learn more about your work or get in touch, How do they do that? I guess this paper or strategy you've been speaking about is at nationalsecurity.ai, and I'm also on Twitter or X or whatever it's called. You should know your work with Ewan. X.com. X.com slash Dan Hedricks would be another way of following the goings-ons as the situation evolves. We'll keep trying to put out work and seeing what's going on with these risks, and if we come up with technical interventions to make them less, then we'll also put that out, too.

1:13:49So, yeah, that's where you can find me. Well, Godspeed, Dan, and we'll have to have you back. Thanks again. All right, everybody, thank you for listening, and we'll see you next time on Big Technology Podcast.