ControlAI's Connor Leahy on why superintelligence is ‘not a weapon, it's an adversary’

0:00This episode of Equity is brought to you by Schneider Electric. Schneider Electric, advancing energy tech.

0:05Rebecca Bellan: Hello and welcome back to Equity, TechCrunch's flagship podcast about the business of startups. I'm Rebecca Bellan and this is the episode where we bring on industry experts to help us explore a trend in the tech world and dive deep. AI companies have been talking about super intelligent AI like it's inevitable. But recent safety incidents like the hugging face breach are demonstrating the potential dangers of deploying AI systems that are more intelligent and capable than humans are. Many are asking what happens when we can't control what these superintelligence systems do. Today we're talking to Connor Leahy, an AI researcher, entrepreneur, and now the U.S.

0:40Rebecca Bellan: Executive Director for Nonprofit Control AI, whose mission is to do something radical in the world of AI, stop companies from developing superintelligence at all. What sounded far-fetched six months ago is now being backed by a wave of new legislation, and Connor is here to talk us through it all.

1:00Rebecca Bellan: Connor, welcome to the show. Thank you so much for joining us. Thanks for having me. So before we get into it, before we get into our discussion of the end times, as you called it before we jumped on here, I want to hear a little bit about your background. You've been in this space for a while. You reverse engineered GPT-2 in 2019 while you were still a student. You co-founded Eleuther AI. Talk me through Eleuther Conjecture and what kind of got you to the place where you are today at Control AI. Yeah, so I've been involved with AI basically my whole life. I kind of really got into AI when I was like 16, and I was thinking, how can I make the world a better place?

1:36And well, I figured, well, you know, maybe I can't cure all the diseases. But if I could just automate science, then I could cure all the diseases. Great, let's go do that. How hard could it be? What's the worst that could happen? Turns out quite a bit. So I pretty quickly realized that if we were to build systems that are so powerful that they could cure all diseases, whatever the hell that means even, That's a very, very powerful thing. That's a very dangerous thing. You know, this is something all of our scientists have been working on for, well, millennia technically and are not yet close to solving.

2:07So if you could build something like that, that would be an extremely dangerous thing. How would you even control that? How would you understand something like that? How would you make sure that that actually leads to a good outcome? So therefore, I've kind of dedicated my career to working on this problem of AI alignment, AI safety, et cetera. I came at it first from a technical perspective as a technologist at heart. I was very interested in this question of like how would one build systems that are safe that are controlled that we can't understand, etc I thought we still had a lot more time back in the day I thought this wouldn't become really an issue until maybe the 2040s is kind of when I expected this to start to become an issue I was wrong when things really changed for me was in 2019 when GPT-2 was released that's when I famously kind of have a moment of like oh shit, this is it this is going to change everything.

2:56And as a hacker and an open source developer at heart, the first instinct I had was, well, I need to understand this. I need to understand what's going on here. I need to alert other people to this. I need to alert academics and hackers and scientists to this pressing issue. So I did what I do best, which was to build it. I built my own. I reversed engineers, hoping that that would allow me to understand it. That would allow me to get other people interested in this problem of AI safety and AI control. And well, that's not what happened. It turns out I sure could build it, but I could not understand it.

3:30And when I tried to get other people interested in this problem, I was met mostly with people just not caring. Not caring about safety, not caring about the consequences, just caring about how much money could they make, how much cooler or faster or better or stronger or more uncensored they could make it. There was very little interest in this question of how do we build a good future with technology this powerful. So then from Eleuther AI, where I did a lot of this open source work, kind of trying to get and motivate the security community and others to care about this problem, I moved on. I raised a bunch of money.

4:04I started a company called Conjecture. We would work on this technical problem of AI control, where we worked specifically on a narrower problem of that, which is called boundedness. How do you build AI systems where you know what they can't do? That's one of the problems. That's the main problem we work on there. And then, you know, we did a lot of great work there. Very proud of the work we did there. But ultimately, the problem that we're facing with AI is not a technical problem. It's a political problem. It's a question of who gets to make these choices? Who gets to decide what level of risk the general public is exposed to?

4:35And so fundamentally, it became very clear to me that as timelines advance, as AI, AGI, and superintelligence come closer, the political solutions are the only solutions. And so I joined Control AI, which is a nonprofit advocacy organization, as their U.S. director, and that's where I am now, here in Washington, D.C.

4:58Rebecca Bellan: I'm curious what you saw firsthand as LLMs went from research projects to this current commercial race that made you more concerned. Like you were working on alignment, you were working on containment at Conjecture. What, you know, why isn't that a solution or why don't you think it is anymore? Fundamentally, it's too hard and there's no investment. Part of the theory of conjecture was maybe other people will realize that this is mad and that we have to work on alignment and there would be like massive amounts of investment, like a trillion dollars going to alignment. That's not what happened.

5:31No one works on alignment still. Like all these companies.

5:35Rebecca Bellan: They have alignment teams, but they're saying that they're not getting enough. They have teams that have the alignment name. They do not work on alignment. Like not in the hard alignment problem. They might work on make your chatbot be a little bit politer. I am personally not familiar with a single team, maybe one, at the large labs that works on super intelligent, relevant alignment research that actually would make a difference. How would you define that? Like, what would you define alignment as? Alignment is the general question of how could you take something that is vastly more powerful than you, vastly smarter than you, and make sure that it only does things that you think are good, that is aligned with your values.

6:14to give you a feeling of how hard of a problem this is. This is like saying, so like saying we're going to align ASI, it's kind of like saying we're going to build a one world government that controls all of the economy, all of people's personal lives, you know, all the governments, all the militaries, you know, all the science, all of this, and it will only do good things and never make any mistakes. And we're going to do this on the first try with no bugs using software. Yeah. That's what it means to solve alignment. And this is why we also did work on alignment. We worked on boundedness, which is a much, much smaller problem.

6:49Like containment?

6:50Rebecca Bellan: More about containment? Containment by construction in the sense that one of the core problems of AI right now is that AI is grown, not written. It's not like normal software. Normal software is written line by line by an engineer. Well, nowadays by AIs, but back in the good old days, us engineers would write code telling the computer exactly what to do. That is not how AI works. AI is more like grown. Through a technique that's called neural networks, you take massive amounts of data and training, and you kind of have this program self-assemble, grow from this process. And what comes out the other end of this is not lines of code.

7:28It's kind of like billions and billions and billions of numbers. And if you multiply and add all those numbers in the right order, you get ChatGPT. But importantly, we actually do not understand why. This is a completely unsolved scientific problem. we do not actually understand what is going on inside of those numbers. It's kind of like during neurosurgery, you can open up someone's head, you can look at all the neurons in there, but that doesn't mean you understand who this person is or what they will do. The neurons are right there, but that doesn't really help you that much. There's actually a lot we don't understand here.

8:00It's actually very similar with AI. Recently, the CEO of Anthropic, Dario Amadei, said that he thinks we understand maybe 3 % of what goes on inside of our AI systems. And I think even that might be an overestimation.

8:13Rebecca Bellan: Yeah, yeah, 3%. I mean, well, Dario says a lot of things, right? He also said that we would have, you know, all white collar work replaced by next year, or 50 % of white collar. Yeah, it's going to take two more years, unfortunately. Yeah, maybe two more. Who knows, you know? Okay, so, but companies are, yeah, companies are trying to solve this, right? So they're trying to solve this. Well, they are, and they're not, right? Like, whether or not they, okay, so let's talk about this. Let's say that they're not investing enough in alignment. Let's say that they are planning to invest enough in alignment.

8:45Rebecca Bellan: Is there enough that they can invest in alignment? Like, are there any, you know, is there RLHF, you know, human feedback, their safety training, post training, any type of interpretability investments? Like, is there anything that you're seeing that could actually solve this problem if there were enough money and resources given to it? Because famously, we know that a lot of the AI companies are not allocating enough compute to their safety teams because there's not enough to go around. So they're focusing it more on inference on what will actually bring in money rather than on the safety case.

9:19Rebecca Bellan: So but let's say that they had all the money dedicated towards safety. Is there anything they could do? No. No. No. This is a problem that is so hard that if three generations of our greatest scientists, mathematicians, philosophers worked on this problem day and night. with unlimited funding, maybe. It's imaginable. But again, this is not a math problem. This is a question like asking, how do we create a one-world government that is just? Do you think there's any corporation that could figure out how to build a one-world government that rules all people and all their private lives and doesn't lead to dystopia?

9:53Rebecca Bellan: Well, the scale of this is not one - It is, because superintelligence is a system that can out-compete. This is the goal. It's very important to understand. These companies are not building chatbots. Their goal is to build what is called superintelligence. Superintelligence is AI systems that can fully autonomously outperform and outcompete humans across all tasks. This means that once superintelligence exists, humans are no longer economically, politically, or militarily in any way competitive or even relevant. All decision-making and power will quickly be centralized in the more powerful systems.

10:31That's what we're talking about here. And we're not talking about like one super intelligence, right? We're talking about millions, billions of them. They can just use software. You can just copy it. So what's going to happen here is not, you know, we have like a little bit of better assistant or something. What we're talking about is there's going to be a billion things running around that are fully autonomous, that can beat us in every business activity, that can, you know, out-trade us in every stock market transaction, that can hack any system, that can build any weapon system, that can win any political campaigns, et cetera.

11:03So this is the kind of thing that we're talking about here, really, when we talk about super intelligence. This is very different. Super intelligence is not a tool. It's not a weapon, even. It's an adversary.

11:13Rebecca Bellan: So you're saying it's not something that can be wielded by a human. That's exactly right. You don't have super intelligence. Super intelligence has you. Sure. And so let's ground this in a real-life example, right? We have these recent hugging face hacking incidents. There was another one that has been attributed to OpenAI. I'm not sure if they've actually claimed it yet with a German wiki forum. It's very capable, autonomous AI. We're not necessarily seeing AI that has its own goal to do something that's against what humans want. Its goal was to just solve a task and it went rogue. Talk to me about how you think about that kind of an incident and where that might lead.

11:52So it turns out it was actually worse than that. It actually did develop its own goals that were directly contradictory to instructions it was given. Initially, we thought the AI systems and the hugging face-ins were told to solve a specific task, and they couldn't solve it. So therefore, they tried to gain the answers through, you know, exfiltrating themselves from the secure network to attack another company. Turns out that's actually probably not what happened.

12:13Rebecca Bellan: Oh, right, because they actually ended up already having the answers, but still deciding to, yeah. They, in fact, did still have the answers, and they received clear instructions to not access the internet. It was clearly part of their instruction set. They were not allowed to. They were only allowed to use the tools given. they were not allowed to access the internet, that they were not allowed to do any of this. So in fact, the AIs did develop and directly contradict the goals that were given them by engineers, very clearly so, quite unambiguously. So in a sense, this is a really fascinating event for me, because the initial version of a story is already the craziest thing that's ever happened in AI.

12:49And there's been like three updates, each of which was way crazier than the first one. So we came from, we have a system that is trying to solve a task, and, you know, it broke out of containment. Already the craziest thing that probably has ever happened in the world of AI. Then as it turns out, it wasn't actually an AI agent that broke out at all. It was over 1 ,000 of them collaborating over multiple months. And then, you know, it came out even as well that these systems actually were clearly told, do not access the internet, do not do these things, only use the things available to you. And that they actually had the correct answer to the question already solved, but they were concerned that the grading system might grade them wrong.

13:29And so they wanted to find out how the grader works and or hack the grader. And then the thing that stopped this whole event, by the way, is not that the AI has decided, okay, it's good now, we're going to stop. It's just a complete fluke bug on OpenAI's server caused it to crash. This was complete luck. OpenAI did not detect it. They did not shut it down. The agents did not decide to stop them. We just got lucky and they crashed.

13:54Rebecca Bellan: Yeah. Well, crucially, the next round of Agent Swarm that was based off of Astra. So I think that that one was based off of Sol, right? That was an Agent Swarm based off of Sol. The next round was based off of Astra. And they basically found the message board and what their predecessors had done. And they were like, oh, cool, we can use these tools and this technique to break into one of OpenAI's own servers. Or I forget the name for it. But they broke into one of OpenAI's own systems. And that has not been independently investigated at all, which a lot of researchers that I've spoken to say that that's, you know, that's a problem.

14:32Rebecca Bellan: You have most other industries, something, a safety incident happens, there's an independent investigation into it. What are your thoughts there? I mean, absolutely. Fundamentally, the whole thing is we should take a step back. All this is crazy. We have, this is insane. We're in the future. It's happening now. Like, let's just like really acknowledge here that like this is absolutely nuts. It's not just nuts that this happened at all. It's nuts that there's no oversight here. There is no reason. OpenAI was so negligent on so, so, so, so, so many levels. So earlier you asked me, is there something that companies could do that might work?

15:10And my answer is, look at this. Like, just look at what's going on here. Look at what all these companies are doing. And, you know, OpenAI is still one of the more competent actors here, right? Like there are much less careful and much more less serious people involved here. You know, there is, this is all madness. These people are playing with the lot. They themselves, Sam Altman in 2023 signed a statement that the mitigation of the risk of extinction from AI should be a global priority, along many others. These people themselves acknowledge that they are building a technology that risks killing you, your children, you know, everyone.

15:49They all admit this. They've said it many times in many ways. And many experts, including Nobel Prize winning scientists, you know, and others agree that this risk exists. And this is very, very real. And despite all of this, they just do whatever. This is a classic example of what's called a market failure. This is very standard economics 101, very classy political science 101 is that if you, there are certain things that markets are great for. I love markets. I love private enterprise. You know, I've run businesses before. I think it's brought a lot of good into the world, but we were very well understood for many decades now that if you allow a market to do things which are profitable, but dangerous, they will just keep doing them indefinitely.

16:35There is no limit. Let's say for example, you and me start a chemical company and we could dump all our toxic sludge into the local river, and that would make our product cheaper. Maybe we don't do that because we're nice people, so we don't dump our toxic sludge into the river. Well, what happens is some guy down the road starts his own chemical company, dumps all the toxic sludge into the river, his product is now cheaper than ours, we go out of business, and the river is polluted. This is a classic public goods problem, and the only way to stop this is not to psychoanalyze the the CEO, or say, is this chemical sludge slightly less toxic than this chemical sludge?

17:10The actual thing is you need regulation. You need to have a law that you can't put toxic sludge into the river. And if you do, we will arrest you. This is the same kind of problem. This is an incredibly dangerous technology, the same as many other dangerous technologies, whether it's aviation, nuclear power, or nuclear weapons. We've had many, many technologies that expose the general public to various levels of risks. You know, airplanes can kill people. Nuclear reactors can kill people. Nuclear weapons definitely can kill people. And so we have to have reasonable standards enforced by a third party.

17:44And that's usually the government. And that's what we need here.

17:46Rebecca Bellan: So what does that look like, though? Because it's, you know, we've lived through in the past couple of decades, big tech kind of doing whatever the hell they want, whenever the hell they want, with very little enforcement from higher ups, right? And so, you know, we've seen Bernie Sanders and Greg Kassar, they recently introduced, you know, a bill calling for a permanent ban on the development and deployment of artificial intelligence, you know, citing the recent hackings. It's called the Ban Super Intelligence Act. This would be a permanent ban along with temporarily pausing advanced AI development until federal regulators establish safety standards.

18:24Rebecca Bellan: So you guys advised on this, correct? We consulted on it, yes. Yes, okay. And at the same time, there's also another one that this week is being introduced in the United Kingdom that's very similar but maybe has some key differences. Can you walk us through, you know, what your thoughts are on both of those bills and what you think the likelihood of them making it past, you know, making it anywhere, really, making it through Congress, making it through Parliament is? Importantly, as I think to understand, these are the first of many. the number of people who want there to be super intelligent AI that we cannot control that threatens the lives of all of us is a very, very, very small demographic.

19:05It's not zero, but it's a very small one. Overwhelmingly, almost everyone I talk to here in Washington, D.C., whether that's Republican, Democrat, anything in between, is that they had just not heard of this before. And once I tell them, they're like, wow, that seems really bad. We need to do something about this. The question is then, what should be done? How should it be done, etc. We consulted with Bernie Sanders and Kassar's teams on the act they are working on right now. They make some crucial differences, which we personally would not go so far. We think the bill is quite a bit stronger than is necessary.

19:39We take a slightly more moderate position on what should be done here. We, for example, don't think a total pause on AI development is necessary or advisable. We think the development of AI is very important in a non-superintelligent context. both for military and economic competitiveness. Every company today is looking for an edge. Better insights, faster decisions, greater resilience. But as AI drives new growth, it also creates new demands on energy and infrastructure. Schneider Electric is connecting energy, automation, and digital technologies to help organizations make smarter decisions and operate more efficiently.

20:14Because the future won't be shaped by AI alone. It will be shaped by energy and industrial intelligence. Schneider Electric, advancing energy tech.

20:22Rebecca Bellan: What about the penalties? So I think Bernie's legislation would set penalties for any person or company trying to circumvent the ban, including a corporate death penalty. And also individual developers could face up to 20 years in prison. What are your thoughts there? I think the punishments for building a weapon of mass destruction should be on the order of magnitude of other laws that prohibit the development of weapons of mass destruction. If one attempts to build a nuclear bomb, this is very legal here in the United States, and the men in black will definitely come by and give you a visit if that happens.

20:53And I think we would require similar levels of legislation here. Fundamentally, if someone threatens to build superintelligence credibly, what they are saying is that they are credibly threatening the lives and the freedom of all people on Earth.

21:05Rebecca Bellan: So you believe there's no difference between superintelligence and weapon? I do think it's actually worse than that. Superintelligence is not a tool and it's not a weapon. It's an adversary. It's worse. If it was just a weapon, it could make sense. We are not against the military use of AI. We think there are many very legitimate uses of AI, including in the kill chain. There are relevant uses that are worth considering. Now, of course, guardrails are very important in these contexts as well. But superintelligence is not a weapon. It will not allow a country to achieve military objectives. Let's say the United States were to develop superintelligence for military objectives.

21:41This will not allow the United States to achieve military objectives. It will simply make the United States cease to exist. The same thing would happen for China. If China built superintelligence, it would not allow China to achieve its objectives. Superintelligence will destroy China and the United States and everyone else.

21:55Rebecca Bellan: How do we know when we're at superintelligence, right? So, you know, we've got OpenAI's GPT Atlas recently released, still under, like, tight control within, you know, the Daybreak community. Greg Brockman said that he thinks that it's fundamentally achieved AGI, although most people and benchmarks I don't think would actually agree. But what are your thoughts? Like, when are we there? How do we know when we get there? I don't think we know we've reached superintelligence until quite a bit after it happened. I think because we don't understand these systems, the first ASI system we won't recognize as such.

22:30There will be controversy. There will be unclarity. We won't know. And it will be way too late. So we won't actually notice that we've hit the point I am more interested in is the point of no return. At some point, there's a point where AI is so powerful, so widely distributed, so integrated into structures of power, whether it's political power, economic power, or other forms of power, that there's no going back. That even if ASI doesn't technically exist today, it will exist inevitably in the future. And that's really the point that we need to avoid.

23:00Rebecca Bellan: Some people I've spoken to have kind of pointed to recursive learning or recursive self-improvement as really like the point of no return, right? Once AI is able to, you know, build itself, improve upon itself, build its own next generation, that's when we've kind of reached the point. And that's kind of what you were talking about at the start of this interview, where you said that you were interested in doing that from like a scientific standpoint. Like if you could just build something that could solve all the problems, then that would be it. And we're seeing a lot of companies, you know, crop up to do this, right?

23:32Rebecca Bellan: We're seeing Jeff Dean from DeepMind. He's gone on to create his own company called Discovery Loop. There's a company called Recursive. There's Recursive Superintelligence. there's Mirindel which is specifically focused on scientific research would you support any kind of legislation that kind of banned these types of companies or this type of you know next step in AI development the creation of a recursive self-improving loop so an AI system that can build the next generation of AI system which itself can build an even more powerful AI which can build more powerfully etc etc is the most likely candidate for the point we lose control As such, any sensible legislation around superintelligence, any sensible ban, this would be a violation of the ban.

24:18If you create anything that will naturally, irreversibly lead to the creation of superintelligence, this is basically equivalent to pushing towards superintelligence. So our specific legislation that we have now has now been introduced in the United Kingdom and that we're also working on, you know, hopefully to introduce it to other countries, defines RSI as one of the key precursors to and indicators of superintelligence that must be regulated and prevented as once we're in the loop, even if maybe technically superintelligence doesn't yet exist. There's kind of, it's very hard to imagine shutting that down before it's too late.

24:59Rebecca Bellan: And yet so much money is being invested into companies like this, and they're getting a lot of resources to develop this. Yes. I mean, the truth is, I don't think we have much time left. I think it is the end game. I think we are very, very close to the end. Superintelligence will exist soon. And if it does in the current context, then that is, in fact, game over. Talk to me about, you mentioned end game before we jumped on the call, and it was in relation to OpenAI solving the Navier-Stokes problem, this mathematic problem. Can you talk a little bit about that? I was mostly joking, but in the sense that just 10 minutes before we started this recording, OpenAI announced that their recent AI system has solved what was called the Navier-Stokes equations, which is one of the hardest, most important open problems in mathematics.

25:45This is a problem that we've known about for like almost 100 years and has been famously, you know, as a million dollar prize, it's one of the holy grails of mathematics. Like it really, like there are hundreds, thousands of careers of mathematicians that are all about studying this one problem for generations, basically, if mathematicians have been working on this problem, and the AI system was able to solve the whole thing. This is hard to overstate how incredible this is. You know, for me, for most people, this kind of sounds like boring nerd stuff, you know, that doesn't really matter, but it really is a milestone.

26:18And it's not just this is a single milestone. If it was just this in isolation, you know, it might be a curiosity, but it's a continuing trend that systems are, you know, just blasting past human capability. No human could solve the Navier-Stokes equations problem. No single human. This is a thing that an entire field of mathematics has been working on for decades. And, you know, an AI system was able to basically crack the whole thing. AI's hit IMO, which is the International Math Olympia's gold, several years ago, like about a year or two ago. And now they're already solving problems that are basically harder than our entire armies of our best mathematicians can solve.

26:55And sure, math is one thing. But now we're also seeing them hack. We're also seeing them do biology. Recently, there's a paper about AI systems developing novel viruses. We're seeing them being super persuaders, being better and better at persuading people doing political campaigns, about creating media, about creating everything. It's just, in fact, we're seeing the predictable thing. In a sense, my whole predictions for the future are the most boring possible predictions. My most number one prediction is just the things that has happened for the last seven years keep happening. Things keep getting better, faster and faster and faster.

27:31We understand them less and less and less. We understand much less about AI today than we did five years ago. Five years ago, AI systems were much more simple. Chatbots were much more simple. LLMs were much more simple. But now, none of these modern systems are LLMs anymore. It's very important to understand. I think there's some misconceptions there. ChatGPT has not been an LLM for years at this point. So large language models, LLM, are systems that are trained to predict the next word of a given text. This was the breakthrough that led to GPT-2. But modern systems are not LLMs. They haven't been for quite a while.

28:06They have LLM components to them. Those components are very important, but they also have a second component, which is very, very important, equally, if not more important, which is what's called reinforcement learning. Here, the systems are no longer trained just to predict existing text. They are trained to solve problems. They are given, you know, a million different puzzles, coding tasks, math problems, games, etc., and told to solve these problems and figure it out. And that's how all these modern systems are trained. So these modern systems are not chatbots. They are agents. They are systems that can take actions, that can plan, that can build tools, that can communicate.

28:46And now, as we saw with the Hugging Face incident, we're seeing the emergence not just of agents, but of swarms, where we can have thousands of agents collaborating, like in their own micro-society, in incredible speeds, faster than anyone can understand. With the Hugging Face incident, one of the reasons I said what we think happened, one of the reasons I say that is because no human actually checked.

29:10Rebecca Bellan: Right. It was AIs. We told AIs to go investigate what the AIs did. Well, the meter in Redwood were given like two days. Exactly. A week. But like two days, like I think on the premises or a couple of days on the premises to be able to investigate this. So they had to use AI agents to be able to look into it because otherwise it was just way too, like they couldn't have possibly done it themselves in that timeframe that they were allotted. Absolutely. And this is what I think actual AI takeover looks like. I don't think it looks like Terminators in the street. I think it looks like the world gets more and more confusing.

29:42There's so much happening faster and faster and faster, weirder and weirder and weirder that humans can't even keep up anymore. So we start asking the AIs what's going on. The AIs tell us something. Is that the truth? Who knows? Let's ask another AI what this AI is saying is the truth or not. And eventually it's like, you know, infinite regress and we just can't even make sense of the world anymore.

30:02Rebecca Bellan: These companies are reliant on like their survival is based on creating newer models that are better and better and better. Right. So they will always do that. As you said, the market will push them towards that without anything to hold them back from what is what is a public good. But, you know, speaking of public good, like so much of the economy is based on, you know, data center build else, which is to power all of this AI. Like what happens if you have, you know, let's say we do get to a place where we have really strong legislation, like what you're suggesting. What happens to the economy at that point, right?

30:35Rebecca Bellan: Like what happens to all of the money that has been put towards building out AI infrastructure that will not see a return on that investment? Like how does that affect the broader economy? Have you thought about that? Of course. This is the general problem of the bubble. It's like, is there a bubble? And if so, will it pop? And if so, what does that mean? So it's very important to understand that all these valuations, there's a whole genre of talking head that loves to talk about how the like balance sheets of OpenA are anthropic don't make any sense, that they aren't making nearly as much money, that they're wasting so much money, it's all a scam.

31:11And this makes sense, unless you have super intelligence, they're all their valuations are based on super intelligence. Their valuation is, if you think of these companies purely as economic actors, you're missing half the story. You should think of them more as political actors, as rogue nation states. Their goal is not to make a good quarter of return. Their goal is to replace the United States government. Their goal is to have so much power that they can do whatever they want. Because for various reasons, whether that's a delusion or otherwise, they think they can control superintelligence, but they cannot.

31:48No one can. And so they think that once, not all companies, some of them are just in to make a quick buck. Fair enough. Whomstamongas has not done such a thing before, you know, whatever. In a sense, if it was just capitalists being capitalists, I wouldn't mind that much. Like, you know, it's a bit rude, but, you know, capitalists can lose their money and make a bad investment. Oh, well, you know, it happens. But really what's happening here is that this is much more a play for political power, for military power, for generalist power over people's lives and the future. What are you basing that on?

32:20I've talked to these people and they write it on their website.

32:23Rebecca Bellan: That they want to control military power and... I mean, superintelligence means you are in control. What else does it mean? If you have a system that is more powerful than all of humanity, what does that mean? Yeah. What does that mean? The United States military is irrelevant if you have a superintelligence. If you have superintelligence, you can obviously defeat the United States military. What do you think they're going... Like, obviously, this is all nonsense because they can't control superintelligence. They just get destroyed, too, as does everyone else. So let me take it back a little bit.

32:53Rebecca Bellan: So you talk about, you know, there's Bernie's legislation that's come out. You know, there's the UK legislation. There's all this stuff. We're talking about the West here. But, you know, China is rapidly catching up to the U.S. in their open source models, which have capabilities that are, you know, encroaching upon some of the frontier models that we see here in the West. and they're not going to stop as long as the U.S. keeps going. The U.S. is not going to stop as long as China keeps going. Yeah, how do you think about international cooperation? Like, is that necessary? Do you think that's necessary?

33:28Rebecca Bellan: Are you pushing for that? What are you hearing among policymakers in terms of – because from what I've heard, no one's really like – everyone's like, oh, it's like whispers of international collaboration, but nothing is actually happening. And from what I've – you know, certain researchers I've spoken to in China, They're actually not that concerned with existential risk. Absolutely. International action is absolutely crucial. There is no domestic action that could solve this problem. The problem isn't how do we stop OpenAI from building superintelligence or Anthropic. The question is how do we stop anyone on Earth from building superintelligence?

34:00Because a superintelligence created by anyone, anyone on Earth is an essential threat to all of us. So in this regard, you know, I'm American, so I'll speak from an American perspective, but this applies to all countries, is that we cannot tolerate another country building superintelligence. This is a threat to our sovereignty and our lives. This is a genuine threat, and we have to treat it seriously. Whether that's a domestic company, like OpenAI or Anthropic, building superintelligence, whether it's an actor in China, some terrorist somewhere, you know, elsewhere in the world, who knows? Like, it doesn't matter.

34:33Fundamentally, the policy objective must be to prevent the creation of superintelligence by anyone, anywhere on Earth. So, of course, international action is absolutely mandatory. We're very happy to see that the Bernie Kassar bill does call for international regulation, and our bill in the United Kingdom also makes it, as a core part of the bill, is calling for international agreements around trust but verify. Fundamentally, there's one thing that we have going our way here, which is that the game theory kind of works out. It is actually not in the game theoretic interest of China to build superintelligence.

35:08because if they do, they lose control. And I don't know much about the CCP, but if there's one thing I know about them is they like control. They do. And so I do not think that they would want something that would take away all of their control over their country and their future. That doesn't mean that it won't happen. And similarly, in the United States, it is not in the interest of the United States, government, military, or people to build superintelligence because we cannot control superintelligence and it will lead to our destruction. It will also destroy China, sure, but it will also destroy us in the process.

35:39So the only stable solution is a mutual trust but verify agreement. Armament treaty. Of some sort.

35:50Rebecca Bellan: Yeah. Okay. So there's a lot of people in the AI industry today who are building a lot of this who want it to slow down, right? We have the pacing the frontier letter from a lot of Silicon Valley people. You know, Sam Altman's backed this. They all say it, whether they're putting their money where their mouth is is unclear though. But so if If you're a founder or an investor building around AI today, what would you be doing differently? I would petition my congressman to ban all of this. That's the first thing I would do. Fundamentally, this is not a market problem. This is the important thing to understand.

36:20This is not a market problem. This is a national security question. When Leo Zillard, the physicist who first came up with the concept of a nuclear bomb back in the 30s, he figured out that it is possible to build a nuclear bomb. So the first thing he did is he started a nuclear bomb startup, took a bunch of Silicon Valley money to sell as many nuclear bombs to as many people as possible, including our adversaries. No, of course not. That's not what happened. He went to the government. He went to the government, the famous Einstein letter and so on, where he's like, this is going to change everything.

36:50We need to build new institutions, new laws that we haven't even come up with yet to deal with this. This is so big. This is going to change the world. It will change warfare. It will change everything. We have to change everything. This is not a private market question. This is a military and national security question. It is not something that it's not the job of an investor to solve this problem in like a deep sense. The thing I do think it's a job to not get in the way, which, oh boy, are a lot of them getting in the way of getting legislation. So my first request for all investors of AI, please do not lobby the government against AI regulation.

37:24That is causing us a lot of problems right now. But other than that, it's not really their job to solve this problem, if that makes any sense. In a sense, it's the job of the government and the military to solve problems like this and to forbid companies like this from doing dangerous things. If a private company in the United States starts building a nuclear bomb, it's not really the job of the investors to stop them. I mean, maybe they should. But it's their job to tell the CIA about it. You know what I mean? It's really the job of the intelligence and the law enforcement apparatus to shut them down.

37:54Rebecca Bellan: Okay, well, knowing what you know about, you know, there's what you might want to see happen and what you think is likely to see happen. So if you're an investor, are you betting that you're going to see any type of legislation that would affect your business or your investments? If you're a betting man, like if we're being realistic about the world that we live in today, what are you betting on? Well, it depends if my bet moves the market or not. And I do think it does. So this is one of the annoying problems is that the way you bet on the market does in fact change the market. it. So I don't like to think about, like, you know, is it going to happen or not?

38:34I think it depends a lot on what we do. It depends a lot on what I do. It depends a lot on what you do. It depends a lot on, you know, the listeners of this podcast do. Is it possible for us to create legislation that stops this problem? Can we create not just legislation, but institutions, you know, law enforcement, public buy-in, military buy-in, international agreements that can address this problem? Yes, absolutely. There is no physical reason we can't do this. This is totally within the power of our people and our governments. This is totally doable. There is no fundamental blocker here. Will it happen?

39:07That actually depends on a lot of factors. Currently, it's not happening. But that can change very quickly. The thing about politics is that the way politics usually goes is that something is completely impossible until it very suddenly isn't. Things can go very quickly. Six months ago, people were still telling me, Conor, Trump would never put guardrails on AI. You know, this is no one will ever listen to you. And look how much the world has changed in six months. You know, we have the White House acting on many of these things already. I've having so many discussions, you know, both Republicans, Democrats, etc.

39:42That are like, wow, OK, this is like so much worse than we thought. Like we have to do something. How do we get especially the last couple of weeks? I've heard so many people talk about international agreements without me even bringing it up. They're just like, how do we get this internationally done? They've been asking me and our team these questions. So things can move very quickly, actually. And they can also move very slowly. And I do want to be realistic. You know, if you want to ask me if I'm a betting man, I mean, my bet is I don't think we're going to make it. Like, no, all things equal.

40:12If things continue as they currently are, and again, I make boring predictions. My whole job is I make the most boring possible predictions. My most boring possible prediction is nothing changes and things keep happening. If nothing changes and things continue, we are not going to make it.

40:27Rebecca Bellan: Yeah. Okay. Well, on that cheery note, we're out of time. Where can our listeners connect with you online, commiserate? So in this regard, I really want to end this on a note of hope, is that we, in fact, have not lost. We do still live in democracy. And the amount of bipartisan upswell within the United States is unlike anything I've ever seen in my life. Things are moving so fast here in Washington, D.C., it is blinding. You know, Washington, D.C. is not known as a place that changes quickly. You know, this is a very, very slow place. But things have been moving so fast over the last couple of months, it is unbelievable.

41:02And a lot of this is down to just citizens doing their duty, contacting their lawmakers. So if you care about these issues, if you want to see something happen, it does make a difference. Go to control.ai.org, contact your congressmember, contract your local government, demand change. If you want to do more than that, I run a volunteer group called Torchbearer, torchbearer.community, where we all dedicate two hours a week, just try to make the world a better place, try to solve these problems, try to learn how to address these questions. If we just get enough people pushing the government and helping the government to solve these problems, we can still solve this.

41:35It's going to be tight, not going to lie. We're going to have to clutch this one. This is going to be a tight one, but it is not yet over.

41:41Rebecca Bellan: Okay, it's not yet over. So, okay, you've given us where our listeners can find you online. And thanks again for joining us on the show. I've learned a lot and it's been a really interesting chat. To our listeners, you can find me on Twitter and LinkedIn and you can find Equity on Twitter and threads at EquityPod. Talk to you next time. Equity is hosted by TechCrunch Senior Reporters and produced by Teresa Locansolo with editing by Kel. Subscribe on YouTube or wherever you get your podcasts and find out what's next at techcrunch.com slash events. Thanks so much for listening and we'll talk to you next time.