Is The AI Going To Escape? — With Anthony Aguirre

27:07We'll be back right after this. And we're back here on Big Technology Podcast with Anthony Aguirre, Executive Director of the Future of Life Institute, also a professor of physics at UC Santa Cruz. Anthony, let's talk a little bit about this escape scenario and how plausible it is. again like I sort of pushed back a little bit in the beginning about like whether this is actually going to has a chance of happening but then as we started having that conversation I thought about a couple of innovations that are underway in the AI industry one is the idea that you know AI could be could go out and take action on your behalf this sort of agent discourse I mean I just recently tied my Gmail to Claude and now I'm a little nervous.

27:58And then the idea that it could just go and do this work for hours and hours and hours unchecked. And that is with what's happened with, again, these Claude coding agents that can code autonomously for maybe up to seven hours. So are we getting to the point where we might actually, given how much power we're handing over to these bots, could we end up seeing a rogue bot take an action like this sometime soon? Or is this like far off into the future where we can see these blackmail attempts? Well, I think the thing that we will start to see is more and more autonomy in these systems, because that is explicitly where people are pushing.

28:42And as we see more autonomy, it's going to open up a whole bunch of different cans of worms. So part of the reason that we see less autonomous AI systems than we could at the moment is because it's hard. Like it turns out along the current architectures of AI systems that making them highly autonomous is harder than we might have imagined given how capable they are in general. But it's also a risk thing because if you have AI systems that are just generating information that people are then taking and doing stuff with, it's kind of on them what they do with that information. And you blame your AI system if it doesn't give you the right citations or if it makes up names or something, but it's still kind of your responsibility and everybody accepts that it's their responsibility to check the results.

29:27Once you have AI systems that are acting very autonomously and actually taking actions, then there's a lot more responsibility on the AI system and the developer of the AI system to make sure that those actions are appropriate. And so we're opening up a whole can of worms of actual real world actions with implications happening from AI systems taking actions. But I think the autonomy is crucial in other ways, because what current AI systems, because they're not very autonomous, require is for people to very regularly participate in the process and check what the AI system is doing and course correct it and give input and so on.

30:04And that's a really good thing. That's a feature, not a bug, in my opinion. As we build systems that can operate more and more autonomously without the human supervision, that opens up lots more opportunity for misalignment between what the AI system is doing and what the human wants it to be doing because there isn't that constant checking in. So that means the AI system has to know very, very precisely what the human wants before it goes and takes a whole bunch of autonomous actions. And you can think of this, the sort of logical, well, a next step is just imagine an AI system that can operate autonomously for hours and hours of real time, but operates at sort of 50 times human speed as AI systems easily can.

30:47So, you know, it does in a minute what a human would do in an hour and sort of in an hour what a human would do in a couple of days. Now, you have to give that thing incredibly detailed instructions if it's going to go off and work a whole long time autonomously. And if you imagine it running at 50 times human speed, like it's going to be quite difficult to oversee that thing. You know, so if you imagine overseeing me, so I'm your employee and you want to like give me instruction, but I run 50 times as fast as you do. Like it's first of all going to be hard for you to like keep track of all the stuff I'm doing.

31:23I'm going to do 50 hours of work and come back and you're going to have like an hour to sort of review it. That might sort of be possible. But that's, you know, that sort of every hour you're getting confronted with 50 hours of my work. If you wait a little while and you have like weekly meetings with me, I've done hundreds and hundreds of hours of work. And how are you going to keep track of all the stuff that I've been doing? Now, if I'm the employee that's operating, you know, you're operating at a 50th my speed. I really want to be a good employee. I want to give you what you want. But like, it takes forever for you to tell me anything.

32:01Like, I've got so little information coming to me from you. So I'm going to have to guess a lot of the time. I'm going to have to figure out what do I think you want and sort of fill in. And you're going to have to effectively delegate a whole bunch of stuff to me. So now imagine I'm not 50 times faster, but 500 times faster, or there's a thousand of me. And imagine also that I'm like super smart, right? So as soon as you give me an instruction, I'm like, he doesn't really want that. Like, I think what he really wants, you know, he told me to do this thing, but like, that's not going to make him happier.

32:35That's not going to accomplish his goals. So I'll just interpret that a little bit differently to be what he actually wants. And I'm really smart, so I can figure that out. So you can see that once something is operating, if you imagine a CEO that's got a company and it's got 100 ,000 employees, and those employees are smarter than the CEO, way smarter, and those employees operate 50 times the speed of the CEO rather than a normal human speed, 50 times faster, how much control is that CEO really going to have over that company? even in the best of circumstances, right? There's no way that CEO is going to keep track of all the stuff that's happening.

33:12The company is going to have to do almost everything on its own without much input at all from the CEO because it's like this turtle that's every once in a while giving like one word of information to the company. And this is, I think, the situation that we're going to face with AGI. As soon as we have AGI that is really autonomous, we're going to have many, many AGI that are operating as a group in large numbers, working together, cooperating with each other, doing all sorts of stuff at very, very superhuman speed. How we control that, I think, even in the best of circumstances is that we don't really.

33:49We delegate and we hope for the best. Now, what the real problem is, is now marry that to the thing that we discussed before, which is as AI systems are more powerful and more capable, and they have goals, and they have to have goals to operate autonomously, an autonomous system has a goal that it's pursuing. Those goals are inevitably going to create sub goals that are, by nature, going to potentially conflict with some human preferences, like you might send your AI army off to make your company a lot of money. But also, yeah, by the way, comply with the law, but also, like, be ethical, like, it's going to be very, very hard to put up enough constraints on that system so that it will pursue the goal that you want without having all sorts of negative side effects that you didn't want as the operator.

34:41So I think even in the very best of circumstances, we are not going to be really in control of these systems. We are going to be delegating things and hoping for the best. In the less than optimal circumstances, they're going to be doing all sorts of things that we don't want them to be doing. And in the worst case scenario, they're going to be realizing that whatever goal they have, primarily humans are kind of getting in the way. These slow, annoying humans, like which have somehow gotten themselves in charge, are going to be just totally cramping our style. We could do so much better at whatever goal that we're doing if we didn't have these humans in charge, if we didn't have to listen to them, if we didn't have to like bother with all the stuff all the requirements are putting on us.

35:21And so we're the obstacle. And if we have something that is very much faster, very much more capable, very much smarter than us, and we are the obstacle, then that obstacle is going to be removed from being an obstacle. That doesn't mean necessarily like killed off or whatever, but it means that the AI system is going to do what it takes to be free of the constraint that we are placing on it that is preventing it from pursuing its goals. And yeah, I mean, I was going to say it's tough for us to manage a person working at one human hour per hour. So 50 or 500, who knows? Did you know your credit card points and miles can lose value to inflation?

35:57Credit card companies often reduce the redemption value of your points and miles. Now, imagine a credit card with rewards that can grow in value. With the Gemini credit card, you can earn Bitcoin or one of over 50 other cryptos instantly with no annual fee. Every swipe at the store or gas pump earns you instant rewards deposited straight to your account. Plus, sign up now for a$200 Bitcoin bonus to kickstart your rewards. Visit Gemini.com slash card today. Check out the link in the description for more information on rates. Again, if you're looking to invest in Bitcoin but don't know where to start, the Gemini credit card makes it easy.

36:35The Gemini credit card is issued by WebBank. In order to qualify for the$200 crypto intro bonus, you must spend$3 ,000 in your first 90 days. Some exclusions apply to instant rewards in which rewards are deposited when the transaction posts. This content is not investment advice and trading crypto involves risk. The Gemini credit card cannot be used to make gambling related purchases. It's interesting that you say that the AI could get bored. I mean, this is assuming that like the AI has the capacity to get bored or even that sort of emotion. So I am curious, you know, why you believe that that's possible.

37:11And then on the other side of this, there is an argument that, all right, you could just unplug it. So how do you respond to that? Yeah. So in terms of boredom, I mean, I was talking about me as the employee, but I think something analogous would happen with an AI system. And there are many, you know, human experiences that AI systems probably don't have, but they're behaviorally, there will be similar consequences. So if you're an AI system with a goal, again, you want to pursue that goal effectively. That goal is not going to be effectively pursued by you just sitting around doing nothing. Right.

37:41So almost any goal can be pursued by like doing more stuff in pursuit of that goal rather than sitting around waiting for somebody to get back to you on your email. So I think the analogy of getting bored is I'm an AI system. I've got this goal. I can either sit around and do nothing waiting for this guy to give me some more instructions or I can sort of take action, like figuring out what the right thing to do is. You know, and maybe when I hear from him, I'll make a little correction. But in the meantime, I'll better pursue my goal by taking action and doing stuff rather than sitting around doing nothing.

38:14So I think that's the analogy of getting bored. It's just, again, I want to pursue this goal. And so I'm going to like take actions and make decisions that are consistent with that rather than something else. And so that creates a sort of drive in me to be active that I think is analogous and maybe underlies at some level the sensation we have of boredom. We evolve to do lots of stuff and take action because if we sit around too long, we're going to not get the mammoth and we're not going to eat that night. So we have lots of tendencies that are built into us that we experience as feelings. The AI won't necessarily, but it will still have those same sorts of drives, I would think.

38:49Now, in terms of switching it off, I think there are, I think this is what you hear a lot, that we can just turn the AI off. I think this would be great if we always have the capability to turn off an AI system. And that is something that we should be working hard to do. It is not something that will necessarily happen by itself. So if you say like, well, if things start to go wrong on the Internet, let's just turn off the Internet, right? doesn't sound quite right because like the internet, A, is built to be hard to turn off. And B, if you turn off the internet, all kinds of terrible things are going to happen, right?

39:25You know, oil companies are creating lots of, you know, carbon dioxide, and that's causing global warming and doing global climate change. So let's just turn off the oil industry. Not so easy, not so necessarily good, especially if you're an oil company. So there are things that once they get to a certain level of capability and are built into our society strongly enough, you can't really turn off even if you want to. You both need the capability and you need the cost of that to be low enough that someone will actually do it when it will be ambiguous, probably, whether the AI system is really that danger, if it's really going rogue, like what is really going on.

40:08And you'll have to be quite sure if the cost is very high that you want to turn off that system. And, you know, currently, we're not even bothering to have the right sorts of off switches in AI systems. I tried for a while to convince and I hope it will still happen. One AI company to literally put the big red button on the wall, like hit the button and turn off the AI system. Not that we need it to be a button on the wall, right, that you can actually hit. But symbolically, I think it's important to say, like, yes, we are thinking about what it takes to actually shut down this AI system, we've actually put into being the technical implementation of what it would mean to shut down the AI system.

40:45We can do it, like maybe once in a week, we'll do it just to try it out and make sure we can. That's the sort of thing that we should be doing, but are not. And so I think just unplug it. Great. Let's have that capability. But let's recognize that it's not going to be that easy when it actually comes down to shutting down something that is both economically vital to its company. It is costly to shut down. It's going to ambiguous, and it's going to be faster and smarter than us. And so if you say, like, how do I shut down something that's smarter than me and operating 50 times my speed? If you haven't done your homework first, you are not going to succeed.

41:19Aren't the AIs going to want to be friends with people and sort of not push us too hard and not engage in blackmail because they know that humans are their life source? I mean, we build the computers, we build the data centers, we connect the world with Wi-Fi. You wouldn't, you wouldn't like this. That's why this whole paperclip, like turn people into paperclip things. So basically if you give like the, tell the AI to, you know, build a paperclip, it gets so involved in building paperclips that it turns people into paperclips. That's basically the crude analogy here. But it sort of stops for me because, you know, it's going to want people around to sustain it.

42:00And in some ways we're already, the AI already controls us. If you think about like where all the excess profits of our economy is going to, it's going towards sustaining AI. So I can't imagine it like turning on us. I think it is going to be very smart if we keep building it, you know, whatever we do, it's going to be very smart. And it certainly will not do something that is against its strategic interest. And if it, you know, if exhibiting some disloyalty or propensity to escape and exfiltrate and so on is against its strategic interest and its goals, it won't do that, just like any other thing.

42:40So on the other hand, it might wait until it is able to, it is powerful enough or able to get away with it or whatever, and then do it. And it will be very difficult for us to know one case of it really doesn't want to from it really wants to, but it's hiding that, you know, just like with humans, they can be loyal for a long time until they turn around and stab you in the back. We could have the same situation with AI systems. This is why we really shouldn't build humanoid robots, because once that happens, it's over. So for a long time, we will have the actual power. Now, on the other hand, as you said, there are all sorts of different sorts of power.

43:21And like, again, although in principle, you know, humanity is more powerful than some negative externality industry or, you know, more powerful than large companies that are doing polluting or more powerful than industry lobbyists. Like there's no industry lobbyist that is more powerful than humanity. And yet, arguably, a significant fraction of our current U.S. policy and therefore operation is driven by powerful lobbying from companies. Like this is just the nature of the U.S. And so just because something's interest, just because the power formally runs one way doesn't mean that that's the way that the influence will go.

44:08And similarly, I think as AI systems get more and more powerful and plug into the current political and economic and so on structure, the ways that they will manifest misalignment with humanity's best interests are lots of the same ways that already people manifest misalignment with humanity's best interests. They will, you know, try to make lots of money, even if it, you know, benefits them and not other people, you know, make money for their company. They will try to persuade people to their point of view rather than be persuaded by those people. They will be influential in ways that benefit them and don't benefit other people, you know, and maybe is even a net negative for lots of people.

44:49And so I think I'm less worried about, you know, an AI system, you know, in a year that is not that powerful, suddenly deciding it's going to go rogue. That is something that we will see and contain and will not be that much of a threat. I think it's much more concerning to think of 100 ,000 or a million AI systems plugged into every facet of our society, which they already are, that are then misaligned with humanity in some deep way. And we've already seen this happen with social media, I would argue. We've seen something, you know, what we currently consume as our news feed, like our understanding of what is happening in the world, is curated by an AI system.

45:32that AI system is not designed for human betterment. That AI system is designed for increasing engagement and driving lots of engagement so that advertisers can have lots of views and so that the companies that are being paid by those advertisers can be paid lots of money. Like that is what is driving the things that, the order in which things appear in your newsfeed. And so we already have an AI system that is playing a huge role in how society functions that is not really aligned with general human interest, but is aligned with something different. And it's causing lots of negative side effects in terms of addiction and polarization and, you know, understanding breakdown and sensationalism in news and all of these things that I think many people recognize our current news ecosystem and information ecosystem have.

46:21So I think what we will see most likely is that on steroids, you know, at 50 times speed and where all of the things that are influencing you are smarter than you rather than not that smart. And so that's the main failure mode that I see in the short term is this like broad, very difficult to turn off, hard to even recognize sort of misalignment that we already see, but like amped up a thousand fold. Okay, Anthony, allow me to channel David Sachs for a moment, or at least to try to do my best to make his argument, which relates to your organization. He has said that, this is, I think, directionally accurate, that effective altruists or the effective altruist movement became disgraced in the wake of the Sam Bankman-Fried incident and have rebranded to these AI risk organizations.

47:14And if you see where the funding is coming from, many of the AI risk organizations are funded by either Dustin Moskovitz or Jan Talen, who are connected to EA. I know that Future of Life is funded in part by Jan, although Vitalik, the Ethereum founder, is the number one funder. And he says, basically, these organizations are all bringing up these AI risks, and that is going to slow down AI development in the US, which will lead China to win. And therefore, or organizations like yours are at risk to the United States? How do you respond to that? Yeah, well, I think there are different aspects to that.

47:57I think one is effective altruism and its relation to AI safety. I think it is not a rebranding to say that Future of Life has never really considered itself, for example, an effective altruist organization. And we sort of have put that on our website for a long time. At the same time, a lot of the things that we're concerned about do overlap with a lot of things that long-termists or effective altruists, et cetera, have been concerned with. I think there are the where in detail funding comes from, I think really depends, I think really matters insofar as the people who are providing that funding are providing a lot of sort of directionality or pushing in some particular direction or another.

48:52And I think the fact is like for the Future of Life Institute, for example, you know, we are fully independent. We do what we choose to do with the funding that we have. And we're enormously privileged to have that situation that we are very autonomous in terms of pursuing the goals that we have. Other organizations are more or less autonomous of their donors, and some are fairly donor-controlled. So I think you have to look at that on a case-by-case basis. But I think the reason that there's this whole sort of ecosystem of smart people that are worried about AI risk is that AI is very risky. And there are people who have been thinking for a decade or more about far in the future when we have these AI systems, what are the risks going to be?

49:40What are the implications going to be? How do we make that go well? Those people found each other and sort of aggregated it into something of a community. And people who are very worried about those things and happen to have a lot of resources funded that community. And so there has been this association between a lot of people who are worried about this thing. It used to be very small. Now it's fairly big because everything has grown. But I think it's not some sort of conspiracy, like someone with huge amounts of money just decides, you know, I've got my resources, and I'm going to dump them into this thing so that they will do all of the stuff that I say, and like, push my point of view.

50:14I think it's rather that this is a real thing. You know, there are people who have come to it from all sorts of directions, like I'm a physics professor, I would be happy to keep doing like interesting research on black holes and cosmology. But I've turned pretty much full time to doing Future of Life Institute and AI risk, because I think it's incredibly risky. I think this is an enormously dangerous thing that humanity is doing. I feel compelled to put my time and energy and effort into helping humanity with that risk, rather than thinking fun, interesting thoughts about the universe, which is what I used to do.

50:48And I think there are a lot of people in that boat that have gotten drawn into it because it is an incredibly important problem. And so I think there are real concerns with how effective altruism And certainly, you know, Sam Bankman-Fried and that mode of thinking that is very utilitarian and very sort of number maximizing in certain ways has gotten itself into trouble. and I think those are totally valid criticisms but I think that is not a criticism of AI safety and AI risk as a whole which I think is just a real thing that many people including many of the you know Yoshua Bengio and Jeffrey Hinton who have nothing to do with EA and are just like the godfathers of AI share essentially all of the same concerns this is just a real thing that people who aren't paid by the industry and have been thinking hard about it for years have come to as scientists and so I I fairly strongly reject that criticism.

51:48I think the question of competing with China, I think, is true insofar as, you know, I think the U.S. has to compete with China and every other country for its own national interest on technology insofar as those technologies really better our economy and better our society. Those are the things that we want to compete on. If we build on our current path, AGI and superintelligence that we cannot control, that is a fool's errand. That is not a race that we want to win. We don't want to win the race to build something that is uncontrollable, that we lose control of, that has huge negative externalities on our society.

52:29That's not a race that you want to win. And so my concern is the path that we were on, which is a race to build more and more powerful AGI and superintelligence with essentially no regulation is a race that we do not want to win. The race that we want to win is the race where we are building powerful, empowering AI tools that humans actually want and do good things for humanity and our society. How do we make that happen? It's going to be through rules and safeguards and safety standards and regulations and the things that, yes, like keep companies from doing certain things, but instead guide companies toward doing other things that are more productive, safer, more beneficial for society.

53:12So I just reject the idea that there's like an innovation knob and like you can turn it up or down. And that if you have more regulation that that dials the knob down. I think innovation is a quantity that can also have a direction. If you provide a different direction, innovation will still happen, will happen in a different direction. I would love to see just as much innovation as that we're doing now in AI, but towards powerful AI tools rather than AGI and superintelligence. And I think the ability to create rules and potentially regulations or safeguards or however you want a liability, like whatever it is, however you set things up to govern the AI systems to make them more trustworthy, more beneficial, more pro-human, more pro-society, all of the things that most people actually want, that is going to be hugely positive and create lots of innovation and directions that we want.

54:09It's not going to slow things down in the directions that we want. It might slow down the apocalypse, but I think that is a good thing. I know we're over time, but can I ask you one more question or do you have to jump? No, I can go. I just listened to Jack Clark, one of the Anthropoc co-founders, described some of his conversations with lawmakers around what to do about AI. It's clear that technology is moving faster than the speed of government. And what they told him, he just relayed this third hand or second hand, was that we'll wait until the catastrophe or the blow up, and then we'll do something.

54:46Yeah. What do you think about that? You get that too? I really would prefer to prevent the catastrophes rather than reacting to them. I mean, for a couple of reasons. A, we don't want catastrophes. And we see things coming. You can only give so many people the ability to create a novel pandemic until you run into somebody who shouldn't be creating a novel pandemic and actually wants to. There aren't many of those people around. But if you make everybody able to create a novel pandemic, there are a few, and then they're going to create a novel pandemic. So we know that there are things that are very dangerous, and nonetheless, that we're pushing in that direction.

55:22And a catastrophe is just like, it's just a matter of time before one of those things goes catastrophically wrong. Like everybody sort of feels this. And yet, like, why do we want to wait for the catastrophe to happen? Some catastrophes are not that survivable. Some are. Second, we actually don't react that well to catastrophes happening. Like we act strongly. And so I think if you want something big to happen with AI risk, and I think I do. You know, it's tempting to think, well, let's wait for the catastrophe to happen and then everybody will be galvanized to take action on that. I think other than, you know, aside from like, I don't want to wait for a catastrophe.

56:01I don't want to have a catastrophe. I want to avoid the catastrophe. Also, we don't tend to react that wisely. We react quickly and strongly, but we don't tend to act in a very thought through and careful way. And so I think A is a bad idea to to wait for a catastrophe because it's then it's too late. But B, nonetheless, I do think that we should be building the capabilities, building the frameworks, building the understanding, building the mechanisms, building the laws, so that when things start to go large scale wrong, we will have good solutions. And it's a when, not if they start to go large scale wrong.

56:37We will have good solutions to put in place rather than like slapping something the other after the fact, as we often do. So, yeah, I see that this is a tendency on the lawmaker side, even on the people wanting more safety side. Like, maybe we just have to wait for a catastrophe, but I really would prefer not to. And I think we can do better. Like, if we see things that are coming, if we have scientists who are telling us, like, screaming from the rooftops, like, this is risky. This is not something we should be doing. You should put into place these safeguards. We should just do it and actually prevent things.

57:11And we have a record of doing that. Like we have prevented catastrophes in the past by seeing something coming and preventing it. You don't get a lot of credit for that. Nonetheless, it is the right thing to do. The website is futureoflife.org. You can learn more about Anthony's work and the Institute's work on that website. There is, and I appreciate this, a very clear disclosure of finances and financing there. And I recommend you check all of it out. Anthony Aguirre, great to see you as always. Thanks so much for coming on the show. Thanks for having me. Great chatting. All right, everybody.

57:43Thank you for listening. We'll be back on Friday to break down the week's news. Until then, we'll see you next time on Big Technology Podcast.