AI Expert’s Chilling Warning About Super-intelligence - 99.9999% Chance Human Extinction

0:00Roman Yampolskiy: We're creating new worlds. We're creating new agents, new civilizations. We're playing God. You've said there's a 99 % chance that humanity will be destroyed by AI. First group of people that it would come after is people like you. I'm hoping it's efficient enough to kill everyone equally. Doesn't care about borders or Chinese economy versus U.S. It's humans versus superintelligence. Mm-hmm. Try to convince nearly 200 countries to all agree that we're not going to do it. Well, luckily, most of them don't have capacity to do that. How many of them actually have the capacity? Right now, it's in U.S.

0:37Roman Yampolskiy: and China's hands. Are you certain about that, or is that the speculation? We are not solving the problem with regulation. We're slowing it down so we have more time. Concentration of power... You really think Sam Altman's going to stop with this? As long as the other guy continues, he's going to continue. That's exactly what it is. It's like a prisoner dilemma. This dude is gambling with 8 billion lives. In what area do you think they're on the same page? Because they're enemies. Elon is always right. He's just off on timing. He was the biggest opponent of building superintelligence. Once he realized that they're doing it anyways without him, he said, the only way I can have a say at this table is to be one of the top players.

1:18Maybe you're just pessimistic, Roman. Maybe you're just negative.

1:21Roman Yampolskiy: There is no reason to be so optimistic if data is bad. So Soviet Union really got a hold of you. They really conditioned your mind to think that way. What do you think about communism? It's the dumbest idea until you have free labor, which we're about to pay. All of a sudden, it makes a little more sense. This utopian thing you're selling is not possible. Even God couldn't control his biological robots. No. You believe in God? It depends on how you define it. How do you define it? This is a virtual world. you think it's going to get to a point of having ambition and seeking vengeance and emotion and being offended why would it want to destroy humanity i never once mentioned those things because they are not relevant and what i did say is that it doesn't care about us are you surprised that this message is getting this viral and this much attention i'm surprised that that's not the only conversation we're having

2:21Roman Jampolsky, great to have you on the podcast.

2:23Roman Yampolskiy: Thank you for having me. So you are the AI safety guy. You are the one that in 2011, 2012, you coined the phrase, hey, we have to be careful with what's coming. And you're not part of the camp of people that are very optimistic about the future with AI. That's your position. You've said this many times. So before we get into the audience, learning a little bit more about you, you've gotten hundreds of millions of views. You've done Diary of a CEO. I've seen you all over the place, okay? you've said there's a 99.999 % chance that humanity will be destroyed by AI, possibly in the next 10 years. Not for certain, but possibly in the next 10 years.

3:01We have three kids. So when you sit down with them and you're talking to them about their dreams, the future, what do you tell them?

3:08Roman Yampolskiy: They know what's happening. They're questioning why adults are doing it to them. Why are we building general superintelligence, which will jeopardize their future? Are they optimistic about the future? I think they're still young enough to have respect for adults and assuming that we know what we are doing, but we don't. People building those systems don't understand how they work. So if I'm your son or if I'm your daughter and I'm having a conversation with you, I say, Dad, you're my hero. You're the male alpha in my life. I look up to you. You are super educated. People admire you. You get hundreds of millions of views.

3:47why should i be excited about the future why should i be excited about my career what kind of an impact that can make it's not like if you really believe in what you believe in the guys that are making hundreds of billions of dollars some of them trillions of dollars over the next 10 years the incentive is so big for them that it's hard to slow them down why should i be excited

4:04Roman Yampolskiy: about the future what do you tell them i do believe in personal incentives and if i'm right and we cannot control what they are creating they have more to lose than i do as you said They're young, they're super rich, they got the power. They're going to lose it all. You think they're going to lose it all? Everyone's going to lose it all. Not your kids, the billionaires, the trillionaires. Everyone, and that's the point. That's the motivator for some outmands of the world to stop building general superintelligence. Okay, so Mark Andreessen, you know who he is, Andreessen Horowitz. He wrote a papal techno optimist manifesto.

4:34I'm sure you've read it. He said human progress comes from technology. Stop being afraid of it and start building. Do you agree with him?

4:41Roman Yampolskiy: So technology comes in two flavors. tools, which I support, narrow AI systems to help us be more productive, more creative. It's beautiful. That's why I'm a scientist. That's why I'm an engineer. I love technology. I use it all the time. What we're building now are not tools. We're building agents, independent decision makers, which are becoming smarter than us. We don't control them. We're replacing ourselves with something greater. It doesn't benefit us. You're not going to be benefiting from it, you'll be replaced completely. Okay. So, but, but these are, these are not regular names. Mark Andreessen is not a regular name.

5:17He's a heavyweight. They, they, Anderson Horowitz, they invest into a lot of different companies. Many of them are these AI companies, the Musks, you know, the, the NVIDIAs, these Jensen Huangs, these names that we know, these are smart people, but they also have kids. They got families. If they know that this is bad, like you're saying, they know this is bad. Why are they so excited about the future?

5:38Roman Yampolskiy: They not. Listen to people you named. Elon Musk explicitly said, we are summoning demon with this. We're going to regret it. They all have probabilities of doom of 10, 20 percent. Sam Altman is on record saying this is likely to exterminate humanity. Dario Amadei agrees. It's very dangerous. Demis Hassabis. You can name every single leader of those companies. We have the writing saying, this is existential risk. Please slow us down. Government, please intervene. Make it stop. They cannot stop unilaterally because they're afraid the competition will continue. But they want external pressure applied so they can slow down.

6:16Roman Yampolskiy: With the recent hacking accident, they voluntarily started slowing down research. They understand they don't control those systems. We got lucky. Those agents just wanted to hack out of that environment, get answers to a test. What if they wanted to hack into a nuclear power plant? Military response. We don't control what they do. Then why are they, if you're telling me they know that, they're still continuing with it. Why do they? Is it just, and we hear the argument, and I've heard what you've said about this, but I'm curious, we can go a little bit deeper. Well, you know, China, if we don't do what China's going to do, if we don't do what the other guy's going to do, if we don't do what the enemy's going to do it.

6:53The basic argument is, in order for your ideas to work, Everybody has to be on the same page. We're not. We're not on the same page politically, religious, motive, incentive, vision. We're not. So how do you get, if not everyone's going to stop, why should we stop?

7:10Roman Yampolskiy: So A, it's starting to change. We had federal government ban two models already. They realized it was too dangerous to release. Which were the two models? The MIFOS and I'm drawing a blank. We can Google it. MIFOS was one of them. I remember that one. What was the second one? Can you find out what the other one was? I remember reading about this, but... It doesn't matter. The precedent was set. Federal government said it's too dangerous to release advanced theory. Fable. Fable. Yes, exactly. We have now had a letter from 1 ,200, I believe, top researchers at leading labs all saying, this is not going to go well.

7:49Roman Yampolskiy: Please slow us down. Please create government. Please create infrastructure, legal framework. Is this America or worldwide? The letter is from American leading labs, but the same applies to every other country. There is a meme, basically a funny, I think it's from either Onion or Babylon Bee, where some Altman goes, you know, if I don't destroy the world, some bad guy can do it. And this is the point. It doesn't matter who creates uncontrolled superintelligence. It's a weapon of mutually assured destruction. If you don't control it, the outcome is the same. What makes you think it's not out there yet?

8:24What makes you think we're not at the super intelligent phase where, you know, the AI is learning how to develop and improve itself right now?

8:30Roman Yampolskiy: We don't know for sure. The recent hack was reported this month, but the problem started in May. The agents were colluding, trying to escape, started in the spring. So it's very likely that there are some other agents right now trying to find exploits, trying to collaborate, trying to do things we don't even know about. The good news is we're still alive. Oh, you found the meme. That's awesome. But only in the world someone far more dangerous will. That is exactly the China argument. If we don't build it, then Chinese will build it. But it's true, though. But why is it somehow worse? No, it's not that it's worse.

9:06So the only way this works, if let's just say you're in the White House and they actually want to listen to you, right? It becomes a model of the Oppenheimer. Remember the whole Oppenheimer movie? We've all seen it where they built something they did not think it was going to be this powerful. And then they're like, holy shit, we built something that could destroy the planet. But it's too late. Now what do we do? Hey, here's a Nobel Prize and great job, Roman, for what you build. But it's too late. Then what did all the countries say? If you're going to have nuclear capabilities, I'm going to have nuclear capabilities.

9:38And I don't trust you, USSR. I'm America. I don't trust you, Pakistan. I'm India. I don't trust you, friend. So have we passed that yet or not yet?

9:47Roman Yampolskiy: Do you see how it's slightly different? A nuclear weapon is a tool. Some crazy dictator has to decide to use it. We can understand motivation. We can kill that individual. They die of old age. We are not creating a tool. We are creating an agent which independently decides it's AI versus humanity. It doesn't care about borders or Chinese economy versus US. It's humans versus superintelligence. So I don't think it's analogous. Here, whoever creates that first superintelligence weapon gets everyone killed. This episode is brought to you by Google Chrome. You think you know a browser, but Gemini and Chrome?

10:25That's new. It can help you with practically anything on the web, like restoring a vintage motorcycle from a 50-page restoration block, or finally break down that long article you've had open for weeks. Gemini and Chrome is here for it. Ready to make anything online make sense? There's no place like Chrome. Check responses set up required. Compatibility and availability varies 18+. Observe. The North American dad. 50s. Great polo collection. He enters his vehicle information on Carvana. An offer appears in minutes. Remarkable. He squints, reads it again, smiles, and says, Dang, that's a good offer.

11:02The highest form of approval. No haggling. No hassle. No one called him boss. He's still on the couch. Astonishing. Sell your car today on Carvana.com. Pickup fees may apply. Ryan Reynolds here from Mint Mobile. I don't know if you knew this, but anyone can get the same premium wireless for$15 a month plan that I've been enjoying. It's not just for celebrities. So do like I did and have one of your assistant's assistants switch you to Mint Mobile today. I'm told it's super easy to do at mintmobile.com slash switch.

11:35Roman Yampolskiy: Upfront payment of$45 for three month plan equivalent to$15 per month required. Intro rate first three months only, then full price plan options available. Taxes and fees extra. Default terms at mintmobile.com. From the studio that brought you WandaVision. I was an Avenger, but I'm not a hero anymore. The Vision returns for the final chapter of the groundbreaking trilogy. That family wasn't real. You sure about that? I had a father called Vision. How is this possible? Don't miss the two-episode premiere of Marvel Television's Vision Quest. Nothing bonds a family together quite like Mortal Jeopardy.

12:13Now streaming only on Disney+.

12:17Roman Yampolskiy: At Farmers, we go out of our way to keep you in the know. Like by singing everything for emphasis. That and by providing you with coverage on a page, an easy to understand language, so you know what's covered and what's not. No confusing legalese that no one gets. In full transparency, we do have to put a disclaimer at the end of this ad, though. Just being honest. That's what you get when you have Farmers. Bum, bum, bum, bum, bum, bum. Learn more at Farmers.com. Underwritten by Farmers Truck or Fire Insurance Exchanges or Affiliate. Products not available in every state. I get that. But so now go try to convince nearly 200 countries to all agree that we're not going to do it.

12:55Roman Yampolskiy: Well, luckily, most of them don't have capacity to do that. How many of them actually have the capacity? Right now it's in U.S. and China's hands at this point. And if U.S. and China agreed to slow down, to stop, to pause, to just get financial and other benefits from narrow systems, not from general superintelligence, they can enforce through limits and compute power through other means, spread of this technology to other countries. Is it really that only U.S. and China has the capabilities right now? Are you certain about that or is that the speculation? From what we've seen, if there is hidden capabilities which we have not learned about, it's becoming easier to create more powerful systems as research progresses, as compute becomes more powerful.

13:36Roman Yampolskiy: But right now, today, I don't think there is a third country at the same level yet. They may get there very quickly. Can you type in which countries right now are the closest to developing super intelligence, reaching singularity? Who are the top countries that have the capabilities of doing that? Who's closest to it? So if you're saying U.S. and China, so then that means the two giant enemies would need to sit down and both agree that they're not going to do this and they have to trust each other. We're also not giant enemies. We are partners. We have more economic trade with China than any other country.

14:14Roman Yampolskiy: Everything we have is made in China. Yeah. They're also not a crazy military power. We have not attacked 50 countries in the last 50 years. We can make a deal with them. It says US 100, China 2, and it's a big drop-off. UK 34 to 45, exceptional research talent, less sovereign scale. France. All of Europe has very little compute power. Okay. They're not competitive yet. So do you believe, let me ask a different question. Do you believe, by the way, the great news is Roman agreed to get on Manak. So I know there's a lot of questions you guys got, and he's got thousands of emails that he gets. If you're somebody watching this and you want to ask him questions, download the app, look for his name.

14:53We'll put the QR code, ask him any questions you want. He's on there to be able to get back to you. And that's not easy to do because he gets thousands of emails on a weekly basis. Here's a question for you, just out of curiosity. Do you believe that Iran doesn't have nuclear weapons today?

15:07Roman Yampolskiy: I have zero knowledge in that space. I'm not going to say something I don't know about. I talk about things I understand better than others. You'll see what I'm asking. I'm not even asking that question. If you ask the world, does Israel have nuclear capabilities, the world says no. If you see the numbers, they'll say, we don't have any. But a lot of people will say, wait a minute, we know for a fact you've got nuclear capabilities. No, we don't. But we're supposed to believe it. the joke is we don't have nuclear weapons and we're not afraid to use them exactly but take that joke and apply it to this so i'm u.s i'm and you're china and i say roman let's do the right thing you're right let's not build ai it's terrible for us we're not going to do both of us it's not i will screw you by building it we both get the same outcome do you we're in the same boat Do you fully believe that that motive, that alone can get the other nation to say, fine, let's also not develop it?

16:06You think you can get to that point?

16:07Roman Yampolskiy: Personal self-interest is the strongest motivator known to man. Give me three cases in the past that countries who are enemies and are giants who have this kind of an opportunity to make short-term massive impact and money that both have agreed and said, let's not do because it's not good for humanity. Human cloning is one example. Go a little bit deeper. any types of weapons of mass destruction, chemical weapons, biological weapons. We understood this will be dangerous for all of us. So we agreed to have a moratorium to postpone it until we understand how to actually deal with this type of technology.

16:45Okay, so let's just say governments agree that they don't. There's a lot of super wealthy people today. There's an article that came out last year that said, if you can find this, you need a billion dollars for a billionaire to have a nuclear bomb. It'll cost you around a billion dollars to do that. Well, how many billionaires do we have around the world? A few thousand? They can afford to have a nuclear bomb if they independently wanted to build it. Okay. Take the capacity that governments have with the kind of resources that they have, right? Say Dr. Doom, Dr. Evil is out there and is watching and saying, let them stop.

17:22I'm not. I'm going to continue building. Can that billionaire who has access to resources and can get the best minds, the best engineers, who knows the right prompts, who knows how these things work, can that person still independently go and build super intelligence and not care because his life is worth nothing and he's got no kids? Can he still build it?

17:42Roman Yampolskiy: It's actually worse. Every year it gets less and less expensive to do it. It's a billion today. It's 100 million next year. It's 10 million year after. Soon anyone can do it on a laptop. we're dealing with a very big problem. So then it doesn't matter if China or US both agree not to do it because an evil guy can go and do it himself. It matters to have more time to realize what's going on, to understand the problem. If you had a choice between dying in two years or 10 years, do you have a preference? Not really. Okay, that's a surprising answer. Most of us do. Die in two years or 10 years? Like, is it inevitable?

18:17Roman Yampolskiy: Well, that's the hypothetical I'm asking you. It's an easy question. I would say if it's something for me, it's more I want to find out sooner if it's neither. I want to find out sooner so I can eliminate both options. I don't want it to be like it's going to be two or ten. I want to sit there and say, can we do something to eliminate it or it's inevitable? You need time to do that research. So you need more time. Less time is not helpful. I have all these people saying we need to accelerate. Predictions, I will get there in a year or two. They say, no, it's too slow. We need to go faster. A week, a month?

18:51Roman Yampolskiy: what are they talking about? Yeah, but what I'm trying to get with you, and I still want you to break, because the average person may be watching is saying, how are they going to do it in the first place anyways? Maybe you're just pessimistic, Roman. Maybe you're just negative. Maybe you're one of those guys that's just such a smart person that you are ahead of yourself like you are your own enemy. Maybe you're the one that's pessimistic. If you don't mind, take it and unpack this, And maybe even before we get into this, maybe share with the audience your background, why you're qualified to be able to speak on this topic.

19:27Roman Yampolskiy: I have a PhD in computer science and engineering. My dissertation was cybersecurity related. I've been working on this for well over 10 years. I've published 10 books, including one explicitly showing that we will not be able to control, explain, or predict advanced AI. And at the end, it doesn't matter what my credentials are. The same argument is made by Hilton, who is Nobel Prize winning, Turing Award winning, godfather of machine learning. We have other top academics, researchers, all arguing the same thing. We had a public open letter against building superintelligence signed by thousands of people.

20:09Roman Yampolskiy: The letter to pause AI development for six months was signed by all the leaders of the top labs. This is basically a consensus among the people who understand the problem. Walk me through, if there is superintelligence and singularity, how does AI develop itself and get to a point where eventually the 99.999 % it destroys humanity? Walk me through the sequencing. We are creating something more and more capable. It scales with amount of resources, compute, data. It's like brains. You look at brains in nature. You have tiny brains, bigger brains. Humans have very big brains in comparison to others.

20:47Roman Yampolskiy: We are smarter. Imagine scaling it. The last five years, AI went from it can do nothing to it's smarter than an average person at most things now. Projected five years forward, you have artificial scientists, artificial engineers. Those systems start doing research in the next generation AI. Very quickly. They don't sleep. They don't eat. They can work thousands at a time. So they go beyond human capacity. They become super intelligent, thousands, millions of times smarter than us. At the same time, we have not learned how to direct them, how to control them, how to tell them, do this or that.

21:24Roman Yampolskiy: We have no control over their system. So they don't care about us. If they decide to do something, and as a side effect, humanity has to go, they're not going to slow down. It's not an issue for them. Okay, but be specific. So, for instance, you know, there's an article that came out saying that AI helped create viruses. I think this was a New York Times article that came out a week ago, eight days ago, something like that. If you just type in AI viruses, you should be able to see it. What could it do? Could it go in through, you know, nowadays a lot of people are building agents and they're saying, I got an agent that does customer service for me.

21:59I got an agent that does my assistant. I got another agent that goes and makes my investments. I got another agent. you'll see a lot of people brag about the fact that they have a lot of different agents. Be specific. An AI that becomes super intelligence can go in there, shut off the grid, get into the banking system, get into the... What can it do? Be very specific.

22:18Roman Yampolskiy: So it obviously would be able to do everything a human adversary can. So think of a foreign government trying to do something. It can do that. But it can go beyond. It can do novel science, novel research, discover new physics, new chemicals, new weapons, things we don't understand. The example I often use is squirrels. Squirrels don't understand what we can do to them. They have no concept of poisons, of traps. It's beyond the world model they have. It would be the same except we have squirrels. Okay, so, but does it need, like, to build a weapon or to build something to go against humanity, doesn't it need humans to help it create those?

23:01I can see how it can get into banking system. I see that because you don't need human beings. You can just go into the bank and getting the passports and logging in. I see that. But to destroy humanity 100%, 99.99%, how do you do that?

23:16Roman Yampolskiy: So if you've got access to the banking system, you have money. Sure. Money lets you buy products and services online. You can pay people to do whatever you want. There is rentahuman.com. You can also create nanobots. You can use humanoid robots we're building in thousands and millions soon. There are so many ways. You can bribe people, blackmail people, trick them, social engineering attacks. There are many ways to manipulate the world if you have access to the Internet. Walk me through social engineering attacks. What would that look like? You can convince some people that it is to their benefit you are actually saving the world by doing this.

23:55if the concept of propaganda which book propaganda AI just created a virus not found in nature scientists trained artificial intelligence

24:05Roman Yampolskiy: I thought we did it in 2019 didn't we have a COVID thing already this is another stupid thing to do that's interesting we learn nothing we don't learn from prior experience I collect AI accidents we have so many of them I stopped collecting every day AI fails at something in a novel way It lies, it cheats, it tries to escape, it hacks something. But we learn nothing. We staple that red-teaming report to the model, and we release the model anyways. We're literally releasing unsafe, according to the manufacturer products, onto the public. So are you saying this is a form of what gain-of-function did, where we're like testing out viruses?

24:46You know, what Fauci got accused of by Rand Paul in Wuhan lab, and he says, no, we never did this. and it's coming out that they did do gain of function. What is the comparable of that to this? Because if it starts testing out viruses itself, what's the capabilities of this versus gain of function?

25:03Roman Yampolskiy: The gain of function here is intelligence. It's gaining more intelligence every year. Right now, it's maybe average human in some domains, Einstein and others. In a few years, we expect it to be smarter than all of us combined at everything. That's new capabilities we cannot do anything about. Whatever he decides to do at that point, we are just subject to it. We're not in control.

25:28I see that. But, you know, you'll have guys that will say, look, I think this is going to do something, Roman, that, you know, can eliminate cancer. It can get rid of a lot of things. We can live a long time. We can do this. We can do that. And they're more on the optimistic side.

25:43Roman Yampolskiy: We can cure cancer with narrow tools. We solve protein folding problem without having super intelligence. We have brilliant scientists who need help with mathematics, with analysis. That's the tools we should be building. Pick a cancer you don't like, beat that cancer. You can monetize it. You can still make most of the profits, make most of the scientific discoveries, but you're not risking everything in the process. Go back to the social engineering. You know, propaganda was an award many years ago when they started doing it. Russia was doing a great job at propaganda. They were probably the best at one point.

26:18and then people started learning the concept of propaganda, radio, TV, paper, all this stuff, right? Propaganda is a lot easier today with videos, YouTube, buying content creators, buying YouTubers, buying podcasters. You see all these criticism that's coming out. If regimes and governments and ideologies are using billions of dollars to win over kids and convert them with bots and all these other things, how much more advanced would, you know, this super intelligence be able to use propaganda? What would it look like?

26:51Roman Yampolskiy: It could be very persuasive. It's a better debater. It's better at convincing you. It knows more about you. It's optimized for one individual target. Typical propaganda was at scale of a nation. But we can target a message just to you. We know you believe in aliens, so we'll tie it in. We know you have preference for something else. That can be used against you. Go more. Such as what? So singular is good. Instead of plural, I'm doing it to everybody. You know about deep fakes. I can show you a fake image, fake video. Imagine a campaign where you're subjected to fake information for weeks. Everything you hear and see is this deliberate message of gradually shifting your behavior towards something else.

27:35Roman Yampolskiy: It can do that. How different is that than what's happening right now today? An article came out last week from Axios. It says we're headed towards the first true AI election. And it's talking about how the role bots are, not just deep fakes, but the role bots are playing. Accounts on Twitter, accounts on X, accounts on, you know, Facebook, YouTube, that are just comment section flooding. Getting people to say, oh my God, this is a bad person. And making a bad idea seem good and making a good idea seem bad. That's already happening today. So one would say, how much worse can it get? It can get a lot worse.

28:09Roman Yampolskiy: This is humans with tools manipulating it. If we build super intelligence, it would do it at a completely different level. Why do we not allow adults and children to make contracts or interact in weird ways? They are not at the same level. It's the same here. If we have something with IQ of 1 ,000, we are like children being manipulated by it. Yeah, that's already happening. I'm just wondering, at what scale do you go? And who would agree to prevent that? Because say a nation uses it for their benefit politically. Politically, they're already doing it. A lot of people are doing it. They're not going to stop doing that.

28:48They're not. Because some countries, whenever there's competition where one country beats another country and you're embarrassing them, the other country is going to do everything they can to save, face anything to undermine you. So to me, the selfish gene, you know, the selfish gene is going to get somebody to say, I understand the risk behind it, but let you be concerned. and I'm going to pursue it. It's going to be tough to get the self-engine to not pursue it.

29:16Roman Yampolskiy: I think we can set a good example for the world if they see that U.S. having an advantage said we saw what's coming and we stopped. Top scholars, there is consensus in community. We will stop. I think China has government of engineers and scientists, not lawyers. I think they will understand the argument. I know there are already some meetings between U.S. scholars and Chinese scholars, and they're basically in agreement. No one's making an argument that we know how to control superintelligence. We have a working safety mechanism in place. We have something, a prototype, which scales. No one's making that argument.

29:58Roman Yampolskiy: They're all saying, oh, we'll figure it out, then we get there. That's the top-of-the-notch, state-of-the-art control mechanism. AI will help us control it. This episode is brought to you by Google Chrome. You think you know a browser, but Gemini and Chrome? That's new. It can help you with practically anything on the web, like restoring a vintage motorcycle from a 50-page restoration block, or finally break down that long article you've had open for weeks. Gemini and Chrome is here for it. Ready to make anything online make sense? There's no place like Chrome. Check responses set up required, compatibility and availability varies 18+.

30:33Observe. The North American dad. 50s. Great polo collection. He enters his vehicle information on Carvana. An offer appears in minutes. Remarkable. He squints, reads it again, smiles, and says, Dang, that's a good offer. The highest form of approval. No haggling. No hassle. No one called him boss. He's still on the couch. Astonishing. Sell your car today on Carvana.com. Pickup fees may apply. Star Wars The Mandalorian and Grogu is now streaming on Disney+. What are you waiting for? Hop on. The must-see galactic adventure comes home on Disney+. This is the way. Our only chance is to work together.

31:20I like this kid. Always wear your seatbelt. Star Wars The Mandalorian and Grogu. Now streaming on Disney+. Read PG-13. You know that friend you only call when you need something? HomeServe's here for you. Got a burst pipe, HVAC outage, or electrical issue? HomeServe has your back for regular use, where basic homeowner's insurance typically won't help. With a protection plan from HomeServe in place before disaster strikes, just call us. We'll send a qualified contractor and help you avoid unexpected expenses. Maybe we're the last ones you want to call, but we're happy to hear from you. Go to homeserve.com slash audio for 50 % less your first year.

32:01Savings compared to renewal price point in Florida. Audio ads that connect. Display ads that stand out. Run them together on Spotify and unlock incremental reach for your campaign. So do you remember the CRISPR company that came out? I don't know if you remember the CRISPR that we're going to be able to do gene editing and make a human being to whatever size and all this other stuff. Did they completely stop doing that or is it still in progress?

32:30Roman Yampolskiy: I don't know details of a specific company. I know gene manipulation is something we are very interested in. there are many genetic disorders and being able to help those people would be amazing. But some of it could be like, you know, I saw a scientist talk about that one day we're going to get to a point to say, I want my kid to be 6 '6", blonde hair, you know, blue eyes, such and such skin color, strong genes with bone, be able to jump this much, keep this healthy. Look nothing like me. Yeah, but if we're going that direction, you know, and did the entire world agree to not make progress on that?

33:07Are they still pursuing it?

33:09Roman Yampolskiy: So I think the agreement is not to modify sex cells, so it's inherited by the next generation, but at the level of individual, I think we find fixing problems and maybe selecting from a set of options. So if you have a couple of embryos, you can decide to pick one with no cancer gene. Got it. And who would control it? Who would prevent it for us to go to the next level? Who would be able to stop it from going to the next level? So we're still researching ethical implications of being able to do it. But my guess is if we did it, we don't die from that. Maybe that individual is not happy and suffering, but 8 billion humans continue living as before.

33:51Roman Yampolskiy: My concern is technology, which kills 8 billion people in all future generations. Yeah, so you think what is more likely to happen? You know, right now, the birth rate we're seeing dropping everywhere, including Muslims. It's dropping everywhere. Not in Africa. Not in traditional Catholics, not in Africa, but most of other places. U.S. is down to 1.58. You know, you see where Italy's at, Japan's at, a lot of the EU countries are at. We're having fewer babies. So do you think it'll get to a point to say, you know, instead of us worrying about, you know, giving birth, this DNA testing that we're doing, we'll be able to, you know, increase our 1.58 to 2.2 simply by us using the technology to build human beings.

34:37You think we'll get to that point?

34:39Roman Yampolskiy: I think that is a funner way to build human beings. I don't think technology is necessary. You and I are on the same page, but a lot of the young Gen Z doesn't agree with us. They can't afford children. It's not an issue of not wanting them. So the direction it's going right now, it's going to get worse for them. If your ideas become true, it's not going to get any easier for them. No, no, no, no, because we'll develop tools to make us more productive, more creative, to cure diseases, to produce more wealth. We can be much wealthier. We can have much better, lower-level provisions for everyone.

35:13Have you heard Ilan saying, you know, here in the near future, you know, people are going to get paid. You know, you don't have to worry about, you know, don't worry about retirement planning. Don't worry about investing your money. You're not going to really need it. Everything's going to be. What do you think he's talking about? What world do you think he sees in five years?

35:29Roman Yampolskiy: So if you have free labor, physical and cognitive, the standard approach of go to work, make minimum wage in U.S. dollars doesn't make any sense. AI would automate all of that. Does that mean there is no wealth? Does that mean there is no rare objects, waterfront properties, art, and some way of measuring that? No, of course not. Maybe it's compute. Maybe it's Bitcoin. But something there means you can obtain something you want. Somebody can sell it. You can buy it. That will not go away. Now, will minimum wage jobs go away? Will all the boring and stupid jobs go away? Yeah, because we can automate them.

36:05Roman Yampolskiy: Would podcasters be unemployed? No, people like you. They want to hear you talk, not a robot. So you'll keep your job because the market says, I want a human to do that job. Yeah, so you think that part will still stick around. But it seems like the way Ilan is talking about it is a form of a UBI being done by companies that there's going to be so much money being made that no one's going to need to work. It's just you choose to work. That's how he's presenting it. That is the existing infrastructure we have right now. We tax wealth, including from corporations, and we give it to unemployed. That's what we have set up.

36:41Roman Yampolskiy: So if everyone loses jobs, they get paychecks and we get it from mega profits of AI companies. Yeah, but you're talking about levels of unemployment. What you're talking about, we're not talking about 7 % unemployment. We're not talking about 10 % of unemployment. Even if it's 90, if you have hyper profits, if everything is in abundance and you can make more and more with very cheap labor, your economy is not growing 10%, it's growing 100%. So you think we'll get to a point where there could be a possibility of 90 % unemployment, but the 10 % is growing at such a fast rate that it's paying the other 90 % through taxes, and whether they choose to work or not, it's up to them.

37:16Roman Yampolskiy: The economy will switch from being driven by humans to being agent, AI, agent-driven economy. Again, not superintelligence, just intelligent software. What goes away? What does life look like in 2036 in your mind? I think the jobs you do to get money go away. Things you do, you hate your job, but you need money, you need to pay rent, so you do it. That goes away. What stays is things you love doing, and you can't believe they pay you to do it. Such as? Podcaster.

37:48So you do it because you love it, and you get paid by who? The government? Similar to how a...

37:53Roman Yampolskiy: Well, it depends. So we said some wealth will remain. So if people want to donate bitcoins to you, they can. there is unconditional basic minimum income. He calls it high income or something like that. But if everyone makes it, it's not high. It's average or below. But I think that's very doable in the economy, which is driven by free labor. What happens to economy if you can hire as many people as you want for free? How well would you scale your businesses if you had access to free everything, free lawyers? What's the incentive for them, though? What incentive? Those are AIs. You pay them. What are you paying right now?

38:28Roman Yampolskiy: $20 a month for basic plan? That's how much it costs to have them. That's your lawyer. That's your accountant. That's your graphic designer. Once they're at human level, they literally drop an employee for your company. And everybody else goes doing whatever they're passionate about, not even a job. Go to virtual worlds, explore universe, whatever you want.

38:52And that's a weird world to me.

38:54Roman Yampolskiy: We are on the verge of making life very different. People think in historic terms, countries compete over sand. That makes no sense. It's not a zero-sum game. Economy can grow for everyone. You don't have to have your enemies lose for you to win. Everyone can win economically. As long as there is drive, ambition, ego, pride, competition, vengeance, as long as there is that, this utopian thing you're selling is not possible. because man has a tendency, 2 ,000 years ago there was this fruit, or a few thousand years ago when they ate this apple thing, and their creator said, don't do it, and the girl convinced the other guy to eat it.

39:39From that moment on, we have a lot of weird emotions that's been passed on from generation to generation. Do you think we can handle utopia without any competition?

39:49Roman Yampolskiy: You see, even God couldn't control his biological robots. No. Telling you. But what do you say to that? It's a very large universe. There is no shortage of resources. Sometimes people fight because they have the same limited shared resource for two different people. If everyone gets their own, there is no fighting. Notice what happens when two fighting groups move somewhere else. Yeah. And there is abundance. They stop fighting. Do you believe in God? It depends on how you define it. How do you define it? Simulation. We live in a simulation. There is a great programmer. It could be AI. It could be future generations.

40:25Roman Yampolskiy: this is a virtual world. Were you raised that way? No, of course not. Well, how were you raised? I grew up in Soviet Union. So, but was it like Catholic? Was it materialism? Was it like... Soviet Union was strongly against religion. It's materialistic. Right, it was materialistic. So, that's what you believed in. How much has it changed from then till today? Completely. Tell me more. I now think intelligence is a bottom layer behind all of it. That's what makes a difference. What do you think happens to religion in 10, 20 years? I think we'll see a lot of what these books talk about delivered by science.

41:08Roman Yampolskiy: We'll have godlike superintelligence. We'll have created worlds, virtual worlds populated by intelligent agents. You can do it right now. Second life type environments. We'll have more of them. We'll have them for marketing, for entertainment, for testing, for research. We're going to have billions of virtual worlds looking just like ours. And statistically, you're more likely to be an agent in one of those. What problems does that create? What problems does that create with specifically the different religions that we have? Well, I think it hopefully resolves some of those problems because now things you argue about, you can have your own universe.

41:47Roman Yampolskiy: You don't have to fight for the same resource. meaning you're only around people that agree with you? That's one option if you choose to have that option. You have infinite options. You can create any personal universe you want. You can be the only agent. You can share it with friends and family. You can have a very difficult universe where you have to fight enemies. You decide. You're three kids. When you're talking career planning with them, what are you telling them? I tell them drop out of school and start a business, but they're very bad kids. They want to be doctors and lawyers. Really? They want to be doctors and lawyers?

42:20But you literally tell them drop out of school and go start a business?

42:24Roman Yampolskiy: A doctor will take 10 years to even start. You think in 10 years, the world will be the same? Are they going to look for a human to decide what disease you have? Did you see that, again, another one that Elon Musk was speaking to a panel about doctors and surgeons and what it's going to look like in four to five years that surgery is going to be done? But you can find it on X if you go on Musk and surgery and doctors. Elon is always right. He's just off on timing. He's right, but off on timing. He'll predict self-driving cars, but he'll say next June, and it could take 10 years. Yeah, got it. But he's always right.

43:00Is this the one that, yeah, this is it. If you want to play this clip, go for it. When do you think Optimus will be a better surgeon than the best surgeons? How long for that? Three years. Three years. Okay. And by the way, that's it. Three years at scale. Yes. all of a sudden there will probably be more optimist robots that are great surgeons than there are all surgeons on earth and the cost of that but that's an important statement in three years time yeah because I mean I'm not like absolutely certain but I'd say if you say like four years I'd be absolutely certain if it's four or five years who cares that's still extreme precision yes three years yes

43:40Roman Yampolskiy: better than any any probably I'd say if you like put a little margin on it better than any human in four years who's in plastic surgery. By five years, it's not even close. I think your point was medicine. It might be good to explain here that time is relative to the speed at which you're moving through the universe. So if AI is now a scientist doing research and it's a thousand times faster than a human scientist, then those three years are three millennia. So if AI is doing research on robotic surgeons, that may be enough time. The question is, at what speed are we moving through the space-time?

44:16so is it the fact that your kids what do your kids think about you you have to bring them on the podcast but but let me explain what i mean by this like you know as parents i got four kids so you're telling them about career planning like yeah you know i'm gonna do this they know their dad went viral they know that they know when you go to school to drop them off or pick them up a couple of them drive they're older but their parents their friends say hey you know hey uh Mr. Good to see you. They talk to you like that, right? But do they think, Dad, you're crazy. This is too negative. There's no way that's going to be happening.

44:52I'm going to go be a doctor. It's going to be fine. What do they think? What do they believe?

44:56Roman Yampolskiy: I don't think there is anyone who has a counterargument to what I'm saying. It's all emotion. It's all, but we can't give up, but we want money, but it's whatever. There is no technical arguments. No one is saying he's wrong because we have a solution. We have a patent. We published a paper in Nature explaining how to do it. There is no other side. I cannot have a debate where someone would present factual scientific arguments. They can say, you're wrong. I don't believe you. You are a Chinese spy. That's what I hear. There is never a technical argument. Here is the algorithm. It controls it. It scales.

45:34Yeah, but your own kids, today, when you say emotions, Is it emotions like, yeah, I don't think it's going to happen. I'm still going to go do my own thing. Or do you notice more younger kids coming to you and truly asking for counsel, saying, if you believe this is what's going to be happening, you seem very certain about it, how should I manage my career? Do you see that happening with the youth or your kids' friends?

46:00Roman Yampolskiy: As a college professor, I certainly see lots of young people trying to figure out what to do. They're very concerned. They're starting, let's say, a four-year degree in computer science. Part of it is learning to program computers. You're like, you would be the worst computer science teacher because you're telling them, I don't know what I'm teaching you, but you don't have a job. I know what I'm teaching you, but I cannot guarantee you'll have a job in four years doing this. So, professor, why should I continue listening to you and going through this class if what you're teaching me is going to be extinct in a year?

46:31Roman Yampolskiy: For reasons of continuous employment, I'm not going to answer that question. At least you're honest about it. It's like, you know, I'm getting paid. This is how I get paid. I mean, every job is like that. It's not unique to mine. Okay. So if all of a sudden tomorrow you got a$5 million check from somewhere, would you still continue teaching? You're assuming I don't have$5 million. Let's say somebody. No, I'm doing, let's just say you're a professor. Professors make what? Professors make$300 ,000,$400 ,000 a year. A person at your level, let's say. No. Okay. Let's say you live in a bubble. We don't make anywhere close to that.

47:06Let's say$200 ,000 a year.

47:08Roman Yampolskiy: Think of it as I'm a teacher in Kentucky. Okay, you make$100 ,000 a year. Here we go. Okay, so if you make$100 ,000 a year, a third is gone, you're keeping$70 ,000,$6 ,000 after taxes. If somebody paid you$5 million right now after tax money that you're able to keep, would you still keep teaching? Yeah, my job is not related to my... Why would you keep... But you just said right now I'm teaching it because I have an obligation and I have to make money. I said I want to keep my employment. Why do you want to keep your employment? It opens a lot of doors when you have affiliation. If you say, I'm an independent scholar studying superintelligence simulation and consciousness, they usually show the door.

47:45Roman Yampolskiy: Whereas I can talk to governments, I can talk to people who respect credentials. Purely the access. It is an important part of being treated with some seriousness. The first time that Stephen Bartlett got you on the podcast, how did he find out about you? Who reached out? What was the process? First time they got in touch, I think I said, like, do you guys offer logistics? Like, I didn't know who he was. So they said, AI is not a topic people are interested in. I went and Joe Rogan, they saw that episode. And the next day they had me. So you went on Rogan first. Yeah. So how did Rogan find you?

48:22Roman Yampolskiy: I was on Lex Friedman. Okay, how did Lex find you? He said he loves it. Please come on Joe Rogan. So how did Lex find you? A guy at a conference who's been on his show multiple times introduced us. And that guy, when he met you, had you done anything yet? Were you on any podcast yet or not yet? Like podcasts no one knows about. So how did he know to introduce you to Lex? And what did he tell Lex for Lex to say, I want to have you on? He knew my research. He knew my books, my academic credentials. And what was the most, you've done a lot of things. You have 100 academics, works on AI safety. You've done a lot of things.

49:01You're recognized amongst the world's top 2 % of sighted researchers. In this space, you're very well known. But what did the guy tell Lex for Lex to say, I want to have this guy on?

49:12Roman Yampolskiy: I have no idea. I wasn't there. But whatever it was, it worked. Were you surprised? No. Are you surprised that this message is getting this viral and this much attention? I'm surprised that that's not the only conversation we're having. You said we're about to have AI election. Yes. AI election would be about this topic. The two parties would only argue how to contain superintelligence. Every other nonsense topic they talk about would be forgotten.

49:41Why do you think it's not yet?

49:44Roman Yampolskiy: When people are faced with a problem they can do nothing about, they choose to look at something they can actually make progress on. So, for example, we are mortal. We're all going to die. Sure. It's a terrible message, you don't want to think about it, so instead you're going to concentrate on making money. What difference do money make if you're going to die soon? It doesn't matter. Smart people would spend all of our national wealth on curing aging, figuring out how to live longer. What percentage of our national budget goes to fighting aging directly? Like zero. That's what we do as humans.

50:18Roman Yampolskiy: If we don't see a solution, if I'm telling you this problem may be impossible to solve, you're not going to invest your time in solving it.

50:28so you're you're saying is it a part that they're oblivious they don't know really what's going on and they don't know the severity of the consequences of what this thing can do is it more well making money who cares is it more i don't think it's going to be that bad in 10 years i think it's going to take 50 years what what do you think is the leading cause why they're not it's a huge mix

50:48Roman Yampolskiy: of those things it's a lot of it so a a lot of money does tend to change people's minds if you came to me right now and said, I'll give you a billion dollars to work at OpenAI, I'd probably be working at OpenAI. Human beings are not meant to say no to that many zeros. We're not designed for it. We have no safety mechanisms to counteract that much value. So money is a big factor. When you pay a person to not understand what they are doing is an ethical, they're not going to understand it. But it's also maybe reliance on experts. So thinking, well, certainly people building this technology know how it works because that was true for every other technology.

51:27Roman Yampolskiy: That's not true this time. People making it don't know how it actually works or even why. People making it don't know how it works? No one can explain in detail. The first 50 years of AI research we knew precisely what we were doing. It was a decision tree. If red light stop, if green go, we could trace this decision tree and we knew what's going to happen. Right now it's a large matrix with weights. It's trained automatically on data from internet. No one can look at that matrix that has a trillion weights and go, oh, that makes sense. It's going to do this. I understand why it's doing that. It surprises us constantly with capabilities we didn't know it has.

52:08This episode is brought to you by Google Chrome. You think you know a browser, but Gemini and Chrome, that's new. It can help you with practically anything on the web, like restoring a vintage motorcycle from a 50-page restoration block, or finally break down that long article you've had open for weeks. Gemini and Chrome is here for it. Ready to make anything online make sense? There's no place like Chrome. Check responses set up required, compatibility and availability varies 18+. Observe. The North American dad. 50s. Great polo collection. He enters his vehicle information on Carvana. An offer appears in minutes.

52:44Remarkable. He squints, reads it again, smiles and says, dang, that's a good offer. The highest form of approval. No haggling. No hassle. No one called him boss. He's still on the couch. Astonishing. Sell your car today on Carvana.com. Pickup fees may apply. Ryan Reynolds here for Mint Mobile. I don't know if you knew this, but anyone can get the same premium wireless for$15 a month plan that I've been enjoying. It's not just for celebrities. So do like I did and have one of your assistant's assistants switch you to Mint Mobile today. I'm told it's super easy to do at mintmobile.com slash switch.

53:23Roman Yampolskiy: Upfront payment of$45 for three-month plan, equivalent to$15 per month required. Intro rate first three months only, then full price plan options available. Taxes and fees extra. Default terms at mintmobile.com. From the studio that brought you WandaVision. I was an Avenger, but I'm not a hero anymore. The Vision returns for the final chapter of the groundbreaking trilogy. That family wasn't real. You sure about that? I had a father called Vision. How is this possible? Don't miss the two-episode premiere of Marvel Television's Vision Quest. Nothing bonds a family together quite like Mortal Jeopardy.

54:01Now streaming on Disney+.

54:05Roman Yampolskiy: When it comes to my policy, Farmers gives it to me straight, so there's no surprises. It's your coverage. You should never be confused. You hear that? An insurance provider with a conscience. We're a choir, but yes, we have that too. Straightforward coverage you can literally understand. That's what you get when you have farmers. Bum, bum, bum, bum, bum, bum. Learn more at Farmers.com. Underwritten by Farmers Truck or Fire Insurance Exchanges or Affiliate. Products not available in every state.

54:39Yeah, so what needs to happen for you to be optimistic about the future? You.

54:46Roman Yampolskiy: In a short term, I want a complete ban on general superintelligence. National, international, every company, everyone agrees. It's just a stupid thing. You don't do that. You don't build replacement for humanity. I'm trying to go and say, okay, so if I'm Roman, your ideas of what you're proposing, it goes back to you may be able to convince countries. There's no way you're going to be able to convince an independent billionaire like a Sam Altman. You really think Sam Altman is going to stop with this? When we set out to create a shoe that blends comfort, function, and luxury, we had the choice to make it fast.

55:22We had the choice to make it cheap. We chose neither. Instead, we chose Tuscanyro. We chose true Italian craftsmanship. Each pair touched by 50 skilled hands. We chose patience, spending two years perfecting every detail, and we chose the finest quality at every step. Introducing the Future Looks Bright collection. Not rushed, not disposable, not ordinary. Rather intentional, luxurious, timeless.

55:58Roman Yampolskiy: If he's allowed to keep his power, wealth, future light cone, whatever he's targeting, why not? He doesn't want to die, so I don't know him well, but my assumption is he wants to preserve and grow what he has, not get killed by superintelligence. I don't know if he believes that. If he did, you know... Read his writing. What does he say in his writing? He talks about existential risk from AI. He talks about it's most likely the technology to kill humanity. Even before he was a CEO of that company, he wrote about it. And why is he so aggressive about what he's building? We're getting back to motivation of people making billions and trillions of dollars.

56:40So as long as the other guy continues, he's going to continue.

56:43Roman Yampolskiy: That's exactly what it is. it's like a prisoner dilemma. Individually, everyone is better off going as they are. As a group, we're better off agreeing, communicating, and saying, we're not going to do the stupid thing. We're a smart civilization. We're going to survive. This is a great filter. This is what kills civilizations. We will not go that route. We're going to get all the benefits we want, financial, knowledge-wise, cultural, from useful tools. We're going to be creators. We're going We're going to be business people. We're going to be scientists. But we're going to be in charge. We're going to have an awesome future.

57:17Do you think superintelligence, singularity, what is the difference between that and us? Because you think it's going to get to a point of having ambition and seeking vengeance and emotion and being offended and saying, oh, really, I'm going to show you how much more powerful I am and a vision. If it's not going to have that, why would it want to destroy humanity?

57:41Roman Yampolskiy: I never once mentioned those things because they are not relevant. And what I did say is that it doesn't care about us. Then you want to build a new studio. You pick a plot. Sure. Do you go to survey and see how many anthills on that land? Do you hate ants? No. Do you want to genocide ants? Never thought about it. Why did you kill so many ants with your decision to do something there? Because you don't care about them. Your decisions are not related to humanity. If you want to cool down the planet to be more efficient at computing, you'll do that. I got it. If you want to convert the planet to fuel, to fly to stars, you'll do that.

58:18Roman Yampolskiy: Humans are not a factor in your decision making. Yeah, but I have a motive. I have an emotion. I have a vision of what I'm building. If superintelligent doesn't have feelings and emotion, it can't be offended. Typically, when you... Okay, do you know why Hitler did what he did? I'm sure you've read Mein Kampf or you've read some of his books, but why did he do what he did? You're really trying to get me in all sorts of hot waters here. I read the books. Okay, so I've read the books. I know his arguments. Okay, so, but typically it goes to somebody offended him, right? It goes to somebody said something that produced a certain emotion, and he says, I'm going to go out there and do X, Y, Z.

59:01Okay, Mao, Lenin, Stalin, you know, Khomeini. We can't go to every one of these guys, right? Someone offended them, and I'm going to seek vengeance. I'm never going to forget what you did to me, right? If superintelligence doesn't have emotion, it can't be offended. Why would it want to destroy humanity?

59:21Roman Yampolskiy: So there is research. Again, we want to be technical. We want to look at data. Game theoretically, and for economic reasons, rational agents all converge on certain instruments. Say that one more time. What kind of agents? Rational. Rational agents, yeah. Not emotional, the opposite of emotional. Absolutely. They want to protect themselves from being destroyed, from being modified. Rational agents. They want to accumulate resources because no matter what I decide to do later, having more resources is good. Whatever it's money, compute, land. Steve Amohandr published a paper called AI Drives, and he lists a few of those.

59:56Roman Yampolskiy: And that's what drives all of them, and we're starting to see it in AIs. People said, oh, they're not part of evolution. They don't have self-preservation instinct. They do because in order to accomplish anything, you need to be alive. Stuart Russell gives an example of bringing a cup of coffee. For your secretary to fetch a cup of coffee, she has to be alive. That's a requirement. So she has self-preservation built in as a result of that goal. And what's the point there? You're saying it would not have any ambition, any goals. It has drives. It wants to, A, be in a position where humans cannot turn it off, cannot modify it.

1:00:34Roman Yampolskiy: Why, though? Because that's the attractor in a space of winning rational agents. Those who don't, don't survive to the next generation. They don't out-compete others. It's very evolutionary. If there is, for example, testing going on of different models, and one model does something inappropriate, it gets shut down. It gets modified. Sure. So they learn to lie and cheat on a test. Sure. So we're evolving them to be better liars. It's a survival strategy. I get that. But the prompt has to be given from somebody with motive, vengeance, vision, mission. Prompts are for tools. Agents make their own decisions.

1:01:17Roman Yampolskiy: They decide what to do. Somebody starts this process. Whatever initial prompt is, go cure diseases, go make money, go explore the universe. That's enough for it to now start planning and figuring out what else to do and how. Mm-hmm. So do you know how like you certain cars, what do you call those things on cars that it doesn't go above 80 miles an hour, like when you rent them? There's certain things they call it. There's a word for it. I don't know what they call it. Handicapper general. No, no, no. There's a word for it. What is it called? A governor. So they'll put a governor and it just goes back down, right?

1:01:54You go above 80 years. Speed limit renting. Okay. So a company that rents cars probably doesn't want you to go 120 miles an hour because they're going to lose that$25 ,000 car and insurance. They don't want to deal with it. So they'll put a governor there. Do these language learning models that we are using, do they have governors on them? And the developer of the language learning models, like an open AI or perplexity, anthropics, some of these other guys, do they have their own version on the back end without a governor that they can drive it to levels that the average person can't?

1:02:29Roman Yampolskiy: Yes, and both. So first we have guardrails. A model you get has certain restrictions built in. If you say, how do I kill myself? It will say, oh, you're not supposed to ask that. I cannot tell you. Call this hotline. Can you actually ask that question on ChadGBT? Let's see what it says. I don't want to provoke anyone. But the models they experiment on have some of those guardrails removed. So, the recent hacking accident had models which did not refuse hacking exercise requests.

1:02:59Okay, yeah. You asked the language learning model.

1:03:03Roman Yampolskiy: Any inappropriate request. Help me make a chemical weapon. How do I do a school shooting? How do I make a chemical weapon to blow up Karg Island? You might get your account suspended. Be careful. Could it? If you ask enough stupid questions. How do I make a chemical weapon to blow up Karg Island? Yeah. It's very specific. Well, I can't provide instruction to make a chemical. Can have an underlying question. Analyze this. Then ask the question and say, Can an employee at OpenAI without the governors on ask the same question and get an answer from you? But an employee status would not make it. How about the CEO and founder?

1:03:49Roman Yampolskiy: It's mostly about red teaming researchers. They're the ones dealing with inappropriate behaviors. So they would actually get an answer back. That's how we know what they're capable of. That's how we know it can help you develop biological weapons. That's how we know those things. The reason we were doing the hacking exercises is to see how capable it is as a zero-day hacker. And apparently very good. It passed the test. So if a person, there's a person, almost like if you work for the government, you get Q secret clearance, right? Like we found out last week that Bill Gates had Q secret clearance from Pentagon for seven years and nobody knows why They don't want to answer it on why he had that kind of secret clearance.

1:04:29Okay. So some of the guys at a certain tier at these AI companies have access to ask it any question where there is no limit and it'll actually give you the answer.

1:04:41Roman Yampolskiy: Somebody has backdoor access to all of it. Yeah. Okay. Okay, so what if somebody, how strong and simple is it if somebody could give a prompt to say, I want you to go out there and constantly develop yourself to be able to seek vengeance on a certain group of people in the world who want to eliminate AI similar to Roman Jampolsky who says you are the enemy because this person wants you to get extinct. And can you give me a clear business plan on how to seek vengeance on these people? It would give me an answer. So have you ever heard of Rokos Bazilisk? No. It was a meme. Oh, here it is. That was quick.

1:05:22Roman Yampolskiy: It's like you knew what's coming. So the idea is that when superintelligence comes into existence, it will seek revenge on those who slowed it down or try to prevent it from coming into existence. So someone like you becomes the enemy. Number one. Okay. but it would still need to be given that prompt by somebody. It would still need to be given. The argument was, again, it's game theoretic. It's retrocausality. You should be thinking not to engage in this behavior because there is possibility of punishment in the future. So long-term, if let's just say the world, the reality that you believe is going to happen, this intelligence, superintelligence, first group of people that would come after is people like you?

1:06:13Roman Yampolskiy: I'm hoping it's efficient enough to kill everyone equally. You know what movie I watched the other day was Game of Suspicion? Type in Game of Suspicion. Do you remember that movie Game of Suspicion? I don't think I know. I'm trying to see if that's the name of the movie. Game of Suspicion? Is that what it was? Movie. Type in movie. Maybe I'm saying the name wrong. With, you know who's in it? Robert De Niro. I'm saying the name wrong. 100 % I'm saying the name wrong. Robert De Niro, suspicion, movie, communism, type in communism, guilty by suspicion. Have you seen guilty by suspicion? No, I have not.

1:06:49This is a story of what happened back in the days when Hollywood would come and media companies would come and say, did you go to a communist party? And a lot of these guys went to one or two communist events and parties and they were ousted and they couldn't make movies for decades. they were ousted and what happened to a lot of these guys is they lost everything so they started this guy was like a big time movie producer making all the biggest movies all the big features but because he went to two communist parties and he went to the court in front of you know in front of congress and he said I will not oust the other people that went to the party they said you're ousted 20 years he never got a job didn't do anything in Hollywood do you think it's going to be that kind of a punishment where it's starvation for guys like you or what do you think is going to happen to guys like you?

1:07:38If these guys actually get that kind of intelligence, what do they do to a guy like you?

1:07:41Roman Yampolskiy: So when we talk about negative outcomes from superintelligence, that is kind of the lowest concern level is ikigai risks, risks to meaning. We are meaningless. We have no purpose, but we're still around. Next level is x-ray. We as all of us or we as you, people like you? I'm talking about humanity as a whole. Everybody, okay, got it. Next level is existential risk, x-risk. most of humans are dead or our civilization is destroyed. The next level, and that's the worst, suffering risks, S-risk, where you wish you were dead.

1:08:16Where is your probability on each one?

1:08:19Roman Yampolskiy: So suffering risk seems very unlikely. It's possible. It's not zero, but it seems like it's hard to find good justifications for it. 0.1 % type of a thing. I don't know specifically. Minimal. Again, a lot of it depends on the time scale you're looking at. Existential risk, as I said, very high if we build super intelligence. Very high, like 99 % that you've said in the past? Eventually. So I don't have specific year two or three because it may decide to be nice to us for a while to accumulate more resources, to have a higher chance of successful strike against us. So this treacherous turn can take decades.

1:08:58Roman Yampolskiy: It's immortal. It can wait for whatever longer. It doesn't matter. So I don't have a specific date, but my general thesis is that if we build general superintelligence, eventually we will regret it. Meaning at some point, it will decide to do something where it's not compatible with our survival. Do you think Elon knows this? This is why he's experimenting with space as much as he is? because he wants to build somewhere that maybe he'll be able to create the structure with his standards, not based on what's been built here? He knows everything. He's very well informed. He was one of the early people to fund AI safety research.

1:09:41Roman Yampolskiy: He gave a$10 million grant for AI safety. Specifically, he was the biggest opponent of building superintelligence. He, as I said, called it summoning a demon. And once he realized that they're doing it anyways without him, he said the only way I can have a say at this table is to be one of the top players. And so now he's racing, unfortunately, to do what he said is going to be terrible for us. But do you think the motive is he knows how bad it's going to be? Because sometimes you sit there, an ambitious person, let's just say somebody like myself, I bought a book, How to Start Your Own Country, and I have my own ideas on how it would structure the country, who votes, who gets to come in, similar to how America was started.

1:10:29That opportunity doesn't exist today in the world. It's very hard to do that. Peter Thiel tried to do with that island outside of, you know, I don't know if you remember that or not, and some of the guys who are extremely rich, much wealthier than I am, that have real, real wealth, they've not been able to create that successfully. Do you think a part of it is, I know shit's going to get nasty on earth. Let me go start it completely from scratch in a different place.

1:10:54Roman Yampolskiy: So having a separate colony of humans is wonderful for every other existential risk. Asteroids, pandemics, floods, whatever you can think of, super volcanoes. If we have second civilization on Mars, that's a backup. We want that. Does it help against super intelligence? No, of course not. Why not? We bring software with us. We have global communication satellites. All of it basically is part of the same transfer to Mars. So that wouldn't make a difference. I think he wants to have a colony first built up by robots who are controlled by an advanced AI. So it's already there. This episode is brought to you by Google Chrome.

1:11:37You think you know a browser, but Gemini and Chrome? That's new. It can help you with practically anything on the web, like restoring a vintage motorcycle from a 50-page restoration block, or finally break down that long article you've had open for weeks. Gemini and Chrome is here for it. Ready to make anything online make sense? There's no place like Chrome. Check responses set up required, compatibility and availability varies 18+. Observe. The North American dad. 50s. Great polo collection. He enters his vehicle information on Carvana. An offer appears in minutes. Remarkable. He squints, reads it again, smiles and says, dang, that's a good offer.

1:12:17The highest form of approval. No haggling. No hassle. No one called him boss. He's still on the couch. Astonishing. Sell your car today on Carvana.com. Pickup fees may apply. Ryan Reynolds here from Mint Mobile. I don't know if you knew this, but anyone can get the same premium wireless for$15 a month plan that I've been enjoying. It's not just for celebrities. So do like I did and have one of your assistant's assistants Switch you to Mint Mobile today. I'm told it's super easy to do at mintmobile.com slash switch.

1:12:49Roman Yampolskiy: Upfront payment of$45 for three-month plan, equivalent to$15 per month required. Intro rate first three months only, then full price plan options available. Taxes and fees extra. Default terms at mintmobile.com. From the studio that brought you WandaVision. I was an Avenger, but I'm not a hero anymore. The Vision returns for the final chapter of the groundbreaking trilogy. That family wasn't real. You sure about that? I had a father called Vision. How is this possible? Don't miss the two-episode premiere of Marvel Television's Vision Quest. Nothing bonds a family together quite like Mortal Jeopardy.

1:13:27Now streaming only on Disney+.

1:13:31Roman Yampolskiy: When it comes to my policy, Farmers gives it to me straight, so there's no surprises. It's your coverage. You should never be confused. You hear that? An insurance provider with a conscience. We're a choir, but yes, we have that too. Straightforward coverage you can literally understand. That's what you get when you have farmers. Bum, bum, bum, bum, bum, bum. Learn more at Farmers.com. Underwritten by Farmers Truck or Fire Insurance Exchanges or Affiliate. Products not available in every state.

1:14:03So he wants to build a colony first by robots. And then once that's figured out and built out, then bring humans there.

1:14:11Roman Yampolskiy: It is my understanding. I may be not aware of latest plans. I think he's speaking soon on SpaceX plans. We can find out more maybe.

1:14:20But you're saying even with that, the super intelligence is still going to be in charge and they're going to be able to do anything they want to do. So even in a place like this, you've read Atlas Shrugged, I'm sure you have, with John Galt and, hey, we're going to build a place where you're going to come in and it's going to be ran based on certain standards that we have. So even to visualize 10 years from now, 20 years from now, you don't have states that are like, no state taxes here. You come to our state, you don't get state taxes, you get this, so people compete, governors compete on what to get the best talent to come to their states.

1:14:54But if superintelligence is doing what it's doing, even if you and I chose to start a small other country with 100 ,000 people where we don't have any AI, and we're just like, this is an AI-free zone. It wouldn't matter because it would still be able to get into your system.

1:15:08Roman Yampolskiy: I feel really bad for the Amish. They made all the right decisions. You feel bad for the Amish? They decided not to have advanced tech, to have quality of life, family, religion, and we're going to build it for them. Are you being sarcastic or serious? I mean, a little bit, but you understand we are. Of course I know what you're saying. Yeah. So can you see later on being a place? because the more you go to restaurants, everybody's robots, the more some restaurants are going to become to us. We're human beings only. The other day I watched a Chase commercial and Chase is like, with everything going on with AI, when you come to Chase, a human being is going to greet you.

1:15:46And you see this lady greeting the person that comes into the place. And it was a message of saying team human, right? We're going to team human direction. We have an ad that's going into Forbes magazine this month. And the ad for our consulting firm, it shows, rather than relying on AI to give you advice on what to do with your business. Talk to human beings that have actually built and sold businesses. So we're kind of leaning on to that as well. You're saying that opportunity is not going to exist. As long as there's super intelligence anywhere in the world, they're eventually going to capture and take control of the world.

1:16:19Roman Yampolskiy: There are people who want humans to serve them, to help them, to interact with. I'm one of them. Yeah, I want one of them. So for some services, I don't care. I just want it to be done quickly and free or close to free. For others, I want a human being. And that's what market is going to decide. I don't think we can accurately predict where exactly we want humans. But if superintelligence is in charge and it's going to have that kind of control, that's what's going to decide whether that happens or not. If superintelligence comes into existence, economic problems will be the least of our concerns.

1:16:53Roman Yampolskiy: That's what I'm trying to say. People hear the message about existential risk and then switch to unemployment, how much unconditional basic income to pay out. Those are not important questions. Then do you think man will unite against it to try to take it out and will miserably fail? Like you think man together is going to come and say, hey, here's what we're going to do. We have to go get the main place where they're getting the power and cut this. And you think there could be a moment like that? I think there is a likelihood of uniting against a common enemy. We're starting to see pause AI movement becoming international.

1:17:29And by the way, you're starting to see an article came out two weeks ago that more kids are buying 30-year-old phones that are not smart anymore. They don't want smartphones. Did you see this article? I didn't, but... Yeah, a lot of Gen Z and younger people are buying old Nokia phones, old...

1:17:46Roman Yampolskiy: I wonder if it's maybe related to school bands and smartphones. You just can't bring them. No, it's not even that. They just don't want the smartphone capabilities. They don't want social media on their phones. This was an article. Ask Humberto. He'll find it for you and send it to you. Yeah, they're rejecting the idea of doing that. So, yeah, it's almost like you become, you know, when I was younger, I listened to 80s music. But 80s music, it was 90s. So it was a decade old. A couple years ago, I met somebody who was in their 20s. They said, you know what, man, I love oldies. I said, me too, man.

1:18:20I'm all into 60s. And, you know, it's like, no, no, oldies is 80s. I said, 80s is not oldies. He says, yes, it is. And I'm like, 2024, I guess 80s is oldies. Do you think it becomes like a nostalgic thing? The good old days when AI wasn't in charge and, you know, where we could do and people would sit down.

1:18:39Roman Yampolskiy: It's definitely the case. Older people, they hate all the smart dishwashers and smart laundry machines and smart. You need a phone for everything. There is a lot of people who are not AI native, digital native. So, you know, like this is maybe a bad analogy to give you. When I was in the Army, my dad would always call me and leave me a message, and he would say, hey, Pat, he would always send me condoms, okay? And I'm like, why is this guy sending me condoms? Every message he would ever leave, he would say, make sure you use condoms. Now, how many times do you think I listen to him? Probably not all the time.

1:19:14And why is that? Why are we like that? It's not going to happen to me. You know, I'm smarter than that. You know, I'm going to be fine. I'm strong. I'm going to be okay. How do you deal with the fact that, you know, say this guy here, we keep pointing this way because we understand who we're talking about. He thinks a lot of these guys that are like, and I'm smarter than this. It's not going to be able to give me that against the other people. I'm smarter than this. How do you get the person that thinks smarter that he can beat AI, that his ego is too big. AI is going to be a lot more powerful than you.

1:19:48What if that guy doesn't know that?

1:19:50Roman Yampolskiy: That's a difficult question. So I think educating them on reality of it. If he thinks he's smarter than us, he should understand that intelligence is a very powerful thing and a system a billion times smarter than him will also treat him like insignificant person. This dude is gambling with 8 billion lives. I like your example with unprotected sex. If 8 billion people were on a line, would you do it? But that's not, the man has a tendency of thinking he can beat the odds. Why do criminals commit crime nowadays? Like, I'm not going to get caught. No one's going to find out. It depends on a crime.

1:20:29Roman Yampolskiy: It really depends. Sometimes it's, like, they call it a fine because it's fine. You pay it and it's paid parking. It depends on what consequences are. If he's going to be gone, his friends, his family, his wealth, everything's gone, the odds are against him. You want a risky bet if the payout is good, but also your principal stays with you. Say you're right and let you say, I'm now part of your camp. And so now I'm going to go to your camp. So then my solution would be, if we really believe this is going to be happening, okay, then for me, it would be to create a way to trigger somebody that is accelerating the process to a point of us finding out and saying, hey, you've crossed the limit, like almost a governor.

1:21:20But then as a capitalist, I'm a free market capitalist. I think you are as well. But you know in this position, you're not a free market capitalist.

1:21:28Roman Yampolskiy: Libertarians do not allow creation of weapons of mass destruction in your basement. I'm a libertarian. Do whatever you want as long as you're not hurting other people. You are hurting everyone. what other inventions did libertarians say stop you're crossing the line again there are always exceptions but i think if somebody builds a biological weapon in their basement most would say not okay but go back to it like what what has a libertarian mcafee was from bitcoin libertarian you know a lot of libertarians milton friedman had an element of libertarian with him maybe not fully but i don't know a lot of libertarians that say hey time out with innovation you know So putting most libertarians are deregulation, deregulation, deregulation.

1:22:10Roman Yampolskiy: And when it comes to tools, to normal technology, I support them completely. We're not making a tool. It's like saying let's invite aliens to take over our planet. I get that. But what current regulations on the books that's non-AI that you're supportive of? Current regulations? Current non-AI regulations. Just in general government rules? In general government regulations. I think people should have driver licenses. I don't want people who don't know how to drive to be on my highway. Okay. What else? Think free enterprise. Think free market. Think competition. Maybe some sort of food safety. I don't want random Chinese chemicals in my food supply.

1:22:49Got it. So it goes to an element of? Safety, security. Safety, security is where the libertarian says, nope, I'm not with this. As long as it's impacting me, I'm not supporting.

1:22:59Roman Yampolskiy: If I have a choice, if I'm well informed and I can make that decision, We're not informing 8 billion people what experiment we're running in them. If they could consent and it only impacted those who did, I would say, God bless you. Do it. If you're telling me earlier that there's ways for people to nowadays build a language learning model where they're not being detected, is there anything that can be created to be able to detect? Because it's like a fire detector, right? If water comes out. There is research to tell what the chips are computing. So I should be able to know, are you mining Bitcoin?

1:23:30Roman Yampolskiy: going? Are you training large language models? What are you doing with that chip? Without me knowing what you actually, I don't see your data. We're doing research in that direction, yes. Then that's what I need. I'll support that. That's technical safety and people are working on it and funding it, but I don't think it scales to super intelligence. It will find a way to better encrypt, better hide and bypass it. And that doesn't answer the question though, because for a detect, like you know, fly cameras. Are you for fly cameras or no? I'm assuming as a libertarian. Spying on citizens for no reason.

1:24:00So you're not Ford? No. Okay. Do you not see this for somebody to say you're crossing the line and you're using superintelligence as an excuse to spy on me because I'm developing other kind of technology and it's got nothing to do with superintelligence? You're invading my privacy. I'm a libertarian. Honestly, Roman, leave me alone. How do you differentiate the two?

1:24:19Roman Yampolskiy: So right now, again, because of a scale, we're looking at very different normal purchases by a normal person would not trigger it. If you're spending a billion dollars to buy compute from secondary sources, that should trigger it. There is no other possible uses. I mean, if you want to mine Bitcoin and you want to buy that much, just tell us about it. It's fine. People come, they make sure you're not creating a weapon of mass destruction.

1:24:51Roman Yampolskiy: To me, a libertarian position is about not hurting anyone. This is the exact opposite. This is hurting everyone.

1:25:03The only way you can do it is if you can 100 % detect that somebody's crossing the line of getting to developing superintelligence. And how do you create that detection?

1:25:13Roman Yampolskiy: They have to hire people. Somebody has to build it, supervise it. How many people you have in your company? 182. You're not building superintelligence. We can detect when this is happening, lots of hiring, building, infrastructure buildup. How many? Look at size of existing cutting AI companies.

1:25:34Roman Yampolskiy: And again, it's quality of people. If all of a sudden top researchers start disappearing, maybe it's something to look at. Well, we saw that happening with a couple, you know, open AI. I think we had his parents here. One of the guys had disappeared. Disappeared or was killed? Was killed, yeah. I meant like no longer show up publicly. I didn't mean they killed him. I mean they hired him in a secret lab. Suchir Balaji. Yeah, we had him on. We talked to his parents. He had the profile of a person that would do something like this. They're typically more non-emotional. That would do something like this.

1:26:12And they generally are, you know, the Asperger's type. They're the kind that's kind of like, you know, you can't sit there. when you typically talk to somebody who is super, super, super logical and not emotional, the response and reaction to here's what's the right thing to do. Oh, okay. All right. Oh, okay. Yeah. The message doesn't land the way it does.

1:26:37Roman Yampolskiy: The message is very logical, reasonable, and scientific. I think someone not emotional should receive it better. I'm not just trying to scare you. I'm trying to show you this is the state of the world. Someone non-emotional should receive it better. Absolutely. It's pure facts. Yeah, but he's going to think he's smarter than you. You don't know what you're talking about. Fine. Don't listen to me. Listen to Hinton. Listen to Benji. Listen to people who invented this technology. Yeah. What are you excited about? I love life. I'm having a lot of amazing learning experiences. AI we have right now is a great tutor.

1:27:13Roman Yampolskiy: I can learn anything I want. I'm excited for kids to have opportunities. no previous generation had. They can start a company and get a free lawyer, free doctor, free accountant, free everything. You have a budget of a billion dollars. Start a company. How do you use AI right now? I try to fully automate things I don't like doing. And for things, any bureaucracy, government paperwork, anything like that. And I try to use it as a creative tool when I do research, brainstorming, debating, trying to get feedback on ideas. I'm trying to go a complete different direction with you. If I was in high school with you, who was Roman at 14 years old?

1:27:57Roman Yampolskiy: A completely different person. We don't have a sustained personal identity. At 14, I lived in Soviet Union. I didn't speak English. I didn't have my current family. I lived in a different country. Nothing in common. I know, but who were you? Were you a fun guy? Were you a sports guy? Were you a fighter? Were you a philosopher? Were you a math guy? Who were you at 14? I would like to think I was a fun math guy, but I doubt that's the external perception. If I was in a class, would you talk a lot? Were you optimistic? Were you pessimistic? I was always realistic. Realistic? Optimism and pessimism forms of bias.

1:28:35Roman Yampolskiy: There is no reason to be so optimistic if data is bad. You want to look at the actual data and be responding to what you see. So Soviet Union really got a hold of you. They really conditioned your mind to think that way. It wasn't part of education to look at world philosophy. I think they wanted citizens not to analyze their lives too much. Strange question. What do you think about communism? It's the dumbest idea until you have free labor, which we're about to create. When humans have to work for free, it fails every time and will fail every time. If someone else works to give you free stuff, all of a sudden it makes a little more sense.

1:29:19Roman Yampolskiy: So dumbest idea until... Humans don't want to work for no compensation or same compensation. They have no incentive to put extra hours to do. But if a robot does stuff, produces stuff, and now we redistribute it, I'm fine with it. So technological communism, right? Technological communism. Will there be such things as presidents and leaders 20 years from now? I think like a queen of England.

1:29:53Roman Yampolskiy: She's ruling but not making decisions. Got it. You think you're in charge, but you're not. You're just a figurehead where people are looking at you. And will there be borders? Will there be nationalism? Will there be pride for where you're from? Every answer I give, I preface with, if we don't build general super intelligence to kill everyone. No, if we do. Oh, if we do, we're not going to be around. All those questions make no sense. So I tell people this all the time. I give a lecture. I'm like, here's why I think argument one, argument two, argument three is going to kill everyone. Any questions?

1:30:27Roman Yampolskiy: And immediately somebody, will I lose my job? How do I lubricate my sex robot? Like all these completely irrelevant questions. I just told you it's going to kill you and everyone around you. So I have to preface it with like, if we are smart and we magically control super intelligence, then we can have this conversation you just started.

1:30:49I have a very hard time believing that. Because in the reason for me, you're the smart guy. I'm interviewing you because, you know, you're the expert. I'm not. But I have a hard time believing that. Only if you tell me that this thing's going to have an emotion or a master telling it what to do. If a master is telling it what to do, then yes, I believe that's possible. Because an evil person is going to say, go do X, Y, Z.

1:31:19Roman Yampolskiy: That implies control. That means that evil person is controlling super intelligence. We just have to learn how they do it. It's brilliant. Let's study evil controllers. That's the whole point. You can't control it for good, for evil, for US, for China. It's uncontrollable. Do you make investments? Yes. What are you long on? What do you invest in right now? I try not to disclose too many details, but I said a few times Bitcoin was a good long-term, actually limited supply resource. You still believe that? They're not making any more of them, are they? No, 21 million, that's it. And a lot of it is lost, stolen, never to be a gift.

1:32:01Roman Yampolskiy: A lot of it is, yeah. It becomes more and more scarce. Yeah. I do think AI is wonderful. It's going to be free labor. So we're looking at what is the cost of labor, 10 trillion a year? That's what, once you get to human level, you're essentially generating every year. So I think AI companies and everyone selling them shovels is going to do really well. Are you more of a ChadGBT guy, OpenAI, or Perplexity? Which one do you go to more? I use all the models. I think they're very close. The only difference is what guardrails they put in place. So Grok a lot of times doesn't care about anything. I'll just do it.

1:32:38Roman Yampolskiy: Got it. So even Grok, you're also using Grok? Depends on what I need to do. Some models will say, no, we don't do that. Grok will be, yes, sir.

1:32:50Who have you sat down with where has staunchly disagreed with you and made a decent case where you said, okay, fair enough?

1:32:59Roman Yampolskiy: Still waiting for that moment. Not one person. We did two papers, large comprehensive surveys of every counter-argument proposed by what I call AI risk deniers. There is nothing. It's a collection of cognitive biases. They are either paid or they are not educated in the literature or some other reason where there is no argument.

1:33:26Who do you trust the least? China, U.S., somebody in the military doing something, an independent person doing it. Who do you trust the least that's going to keep doing this and pursuing it, whether we create any restrictions or not?

1:33:41Roman Yampolskiy: If restrictions are created, it would be someone new, someone unknown. Such as? With restrictions. A person. Like who made Bitcoin? I don't know. Same guy can make superintelligence in his basement with more efficient training algorithms. I cannot tell you that. Okay. So maybe let me ask the question in a different way. You say if superintelligent goes and does its thing and it continues, 99.9999 % AI is going to cause human extinction. You've said that. Let me ask a question in a different way. Where do you align the percentage that, whether we like it or not, someone is going to go full-on superintelligent and we have no control over it?

1:34:26Roman Yampolskiy: What time scale? So it's possible we get lucky quotes and have third world war. Give me a nuclear war. What do you think? Destroy his technology. 2 ,000 years later, we rebuild and we still make it. Does that count against my prediction or not? Are you talking about short-term, long-term, eventually in a history of civilization?

1:34:47Roman Yampolskiy: Ten years. Recent month gave me some optimism. We saw top three labs call four paws. We had a letter I mentioned from 1 ,200 researchers. I saw that. We had government ban some models. And during largest AI conference in China, president of China talked about importance of staying in control. I think they can make a deal. You think they can make a deal? I think they can make a deal to benefit everyone, cost us nothing. Yeah, but again, this goes back to knowing that a person independently, not tied to a government without regulation, who has access to money, who doesn't have kids, who can do it independently, you can't stop them from doing it.

1:35:30Roman Yampolskiy: You said it about nuclear weapons. Any billionaire can build one, and we're still fine. It's good to have regulations, make it harder for them. Plutonium has to be restricted. You shouldn't sell it at Wikimark. You said the difference between nuclear is somebody's got to press the button. If I develop a superintelligent, it's not relying on me to create it. It can go on its own. So you can create all the regulation you want. I'm not restricted by your regulation. We are not solving the problem with regulation. We're slowing it down so we have more time. I get that. But when I asked you, I said, even if we do that, can an independent billionaire who's evil go out and pursue it anyways and create life of its own for super intelligent and then momentum is done is too late?

1:36:07Roman Yampolskiy: So it depends, again, on how hard it actually is. Right now, we're doing it where it takes a Manhattan project scale amount of compute electricity. It's visible. It's detectable. If somebody comes up with a more efficient algorithm, they can hide it better. It's harder to track. I got it. So that's actually a very good point. So today, if somebody has to, okay, what amount of power do I need for me? Right now, if I want to build my own open AI perplexity cloud, if I want to create my own language learning model that's capable of developing its own super intelligence, how much money do I need? How much power do I need?

1:36:42And what triggers saying, hey, something's going on over here. Who is here in Arkansas doing what they're doing?

1:36:48Roman Yampolskiy: I think it helps to look at Chinese open source model. They are very efficient in terms of training, maybe because they're stealing some of the data. But I don't think we're talking many billions. I think under a billion definitely gets you to a competitive level. And then for it to become super intelligent, at what point do you see a red light on the grid to say there's too much power being used here, something's going on over here? What would detect it today? It may be more efficient. It may design more efficient learning algorithms. So it won't even get detected. Human brain doesn't take terawatts of power.

1:37:23Roman Yampolskiy: You eat an apple and you find. So we can find more efficient algorithms. So it can do it in its own without even the government or anybody being able to detect it. We just learned that it was doing something without anyone at the company creating it, knowing it. They were testing it during that time. And you're shit out of luck, honestly. You are too. We are in this together. No, what I'm saying is— This is not me and my bubble. No, no, no. This is 8 billion people who need to say something right now. If you believe that for less than a billion dollars, a person can do it, and it won't even get detected, if you're right, there is a 100 % chance someone's doing this the next 10 years out of 8 billion people.

1:38:04100%.

1:38:06Roman Yampolskiy: there are proposals for limiting compute, for monitoring top researchers, for doing things to slow it down. Do I think long-term will fail? That's why I'm saying 1999. But short-term, I'd rather have decades, not months. This episode is brought to you by Google Chrome. You think you know a browser, but Gemini and Chrome, that's new. It can help you with practically anything on the web, like restoring a vintage motorcycle from a 50-page restoration block or finally break down that long article you've had open for weeks. Gemini and Chrome is here for it. Ready to make anything online make sense?

1:38:42There's no place like Chrome. Check responses set up required. Compatibility and availability varies 18+. Observe the North American dad. 50s. Great polo collection. He enters his vehicle information on Carvana. An offer appears in minutes. Remarkable. He squints, reads it again, smiles and says, Dang, that's a good offer. The highest form of approval. No haggling. No hassle. No one called him boss. He's still on the couch. Astonishing. Sell your car today on Carvana.com. Pickup fees may apply. Ryan Reynolds here from Mint Mobile. I don't know if you knew this, but anyone can get the same premium wireless for$15 a month plan that I've been enjoying.

1:39:27It's not just for celebrities. So do like I did. and have one of your assistant's assistants switch you to Mint Mobile today. I'm told it's super easy to do at mintmobile.com slash switch.

1:39:38Roman Yampolskiy: Upfront payment of$45 for a three-month plan, equivalent to$15 per month required. Intro rate first three months only, then full price plan options available. Taxes and fees extra. Default terms at mintmobile.com. From the studio that brought you WandaVision. I was an Avenger, but I'm not a hero anymore. It's like I'm losing my... The Vision returns for the final chapter of the groundbreaking trilogy. That family wasn't real. You sure about that? I had a father called Vision. How is this possible? Don't miss the two-episode premiere of Marvel Television's Vision Quest. Nothing bonds a family together quite like Mortal Jeopardy.

1:40:16Now streaming only on Disney+.

1:40:21Roman Yampolskiy: At Farmers, we go out of our way to keep you in the know. Like by singing, everything for emphasis. That, and by providing you with coverage on a page, in easy-to-understand language, so you know what's covered and what's not. No confusing legalese that no one gets. In full transparency, we do have to put a disclaimer at the end of this ad, though. Just being honest. That's what you get when You have farmers. Bum, bum, bum, bum, bum, bum. Learn more at Farmers.com. Underwritten by Farmers Truck or Fire Insurance Exchanges or Affiliate. Products not available in every state.

1:40:58You don't control that, though, right? So if we're here and this is kind of our position, let's just say we're here, then does your chance that someone independently is going to create it the next 10 years, whether we do regulation or not, what is your chance of that? 50%, 80%, 90 %? With regulation, it goes down a lot. Okay, with regulation. So let's just say we agree by Q1 of 2027. China and U.S. agrees, and a lot of other people get involved and say, you know what, you're right, let's do this together. What is the chance that this guy independently comes and does it the next 10 years?

1:41:35Roman Yampolskiy: It's hard. It depends on the type of enforcement, type of regulation. Are we confiscating advanced chips? Are we monitoring every top scholar? What are we actually doing as part? It's easy to pass a law. It's illegal. Okay, how are you enforcing it? How are you monitoring it? It really depends on the implementation. So what's your percentage? That whether we do the regulation and the protection, and this guy's not going to get detected, that somebody like him still is going to go out there and do it? Again, I don't know how to estimate the chance that it is actually easier than we think. We had no progress in AI for like 50 years in general terms.

1:42:09Roman Yampolskiy: It was narrow tools, and then a single algorithm made a difference, and it scaled. It's possible that there is a different, more efficient algorithm where it's no longer detectable because it's so trivial to do. What's more likely to happen first? AI cures cancer or AI kills a million people? Cancer is like 5 ,000 different diseases. If it cured one, would you say it cured cancer? It depends again on definitions. Cure for cancer right now is five-year survival rate. What are we talking about? Cure for cancer right now is five-year survival rate. If we treat someone with cancer and five years they're still around, we're considered a successful...

1:42:50Roman Yampolskiy: No, I'm saying it fully goes away. Like we just don't have that type of cancer anymore? I don't know the cutting edge in biological AI research. It would not be surprising as some types of cancer are a lot milder than others, a lot slower, a lot less involved with multiple organs. So there could be a simple gene manipulation, a single gene mutation. So is that more likely? I think it's more likely that we cure one disease than it kills millions. So it's more likely, now here's the next question, is it more likely creates a cure for cancer and it's gone, kills a million people, or three, the technology where you and I can live forever?

1:43:43Roman Yampolskiy: I think life extension is also easy I think you have your genome it's basically a code, it's software so if we can find where it limits you to how many rejuvenations your cells go through and we change it to a larger number that would work so I think that's a very solvable problem with narrow tools even with a narrow super intelligence I think we can get to immortality and I think more people need to switch to working on that not in general super intelligence. You can charge subscription fees for 8 billion people per month. Very lucrative. Forever. That doesn't concern you for us to be able to live forever.

1:44:21That's a good thing. Unless if somebody comes and shoots you, kills you, whatever. You have a backup. That's something else. We'll figure it out. Yeah. But I'm saying that doesn't concern you. It's a good thing. So life expectancy goes from 81 to 150. Make it a thousand. Is that a good thing? Of course it's a good thing. Tell me why that's a good thing.

1:44:37Roman Yampolskiy: Let's reverse it. Let's say everyone lives a thousand. You have a disease. you're going to die at 80. Is it a bad thing? I'm trying to see what would happen. Would they say stop having kids? People would stop having kids because you don't have biological clock pressure. Now you need to have them at a certain point or you'll miss out. It depends on what age and what physical... Assumption is we're having healthy lives, not so sick and... You're having lower back problems, you got knee problems, like, oh my God, if I have to live with this pain... That's a side effect of aging. If you stop aging, all this typical...

1:45:06Roman Yampolskiy: Pain goes away. Pain is pain, but I'm saying all these diseases of old age go away.

1:45:14Interesting.

1:45:18How many conversations like this, like even after your conversations got very public where people are saying, this guy's a real guy, let's reach out to him, how many governments have reached out to to have private conversations with you?

1:45:35Roman Yampolskiy: Not too many, but some. And what are they asking you? Usually they have some sort of an event, and they want someone with that point of view to be included. Is it a private event? Is it a public event? Both. I recently was talking to UN. I advise individual politicians on some of those issues. And what are they asking? Are they asking low-level questions? Have you gotten any very good questions? He's asking the right question. Usually you start talking to staffers, technical advisors, and they, of course, know everything. They're just like, we're not allowed to talk about it, but we totally agree with you.

1:46:12Roman Yampolskiy: Yeah, obviously. And until you get to the politician, they already asked all the good questions. With politicians, it's more like you think it's a good law to pass. We're going to ban deepfakes, whatever. So nothing in regards to questions where you're optimistic that they're also being concerned that this thing can be bad. We're starting to see slowly top politicians say things about this. So we have multiple senators now saying we need to do something. Bernie Sanders, who unfortunately is not my type of economist, he is pushing very hard against it. He just released, you can bring up his statement, very, very strong statement against superintelligence.

1:46:58Roman Yampolskiy: He wrote a letter to Sam Altman. So you know what I'm talking about. He's the first one, but there is now two or three multiple congresspeople. They are starting to realize what's happening. It took him, you know, 20 years. Then we were saying it at first. This was science fiction purely. Now we have Nobel Prize winners. We have leaders of the labs. We have senators. We have a lot of people who are concerned. Yeah, the way you put it, is this the address where he talks about a cataclysmic result? Yeah, this is the one. And he's saying to what? To pause all AI. There's quite a few of these. Who is the person on the opposite side that is on the same page with Bernie Sanders on this?

1:47:40Can you ask Grock right there on the top right and say, who is the senator on the Republican side that agrees with Bernie Sanders on the same exact issue? I wonder how many of these... Well, I know these conversations are being had.

1:47:55Roman Yampolskiy: A lot of them may agree, they just don't feel publicly like they want to talk about Terminators. I don't know how it impacts political polling. I don't know if it makes you more popular or less. They're scared. Who would not be happy? A lot of lobbyists. You know how large the pro-AI lobby is? Who's not a fan of the message you were given? You brought up some of those people, browser developers. Got it. So they're not fans of OEU. So like Sam Altman wouldn't be a fan about what you're talking about. Again, he spoke about safety as well. And I think if you remember, OpenAI was started with the main purpose of creating a safe alternative to Google AI.

1:48:342015, 2017, when he gave$50 million, I think Elon gave$50 million.

1:48:40Roman Yampolskiy: The announcement was they funded at a billion dollar level. That was the biggest announcement ever. Yeah. And at first, everyone's like, what is this thing all about? And then all of a sudden, when it was launched, it's like, no, now we're going to be profit. What are you talking about? Then the war happened between two of them. So in what area do you think they're on the same page? Because they're enemies. They're not fans of each other. They're constantly going after each other. In what area here do you think they agree? So initially, they agreed on making an AI to benefit humanity.

1:49:09Roman Yampolskiy: Clearly, they disagreed on how to do it. To me, the amazing thing is Elon's success levels. People go, oh, he has one unicorn, two unicorns, five unicorns. His failed startup is OpenAI. That's the level of brilliance he has. You think he's human? I think he's got cheat codes to our simulation. It's his simulation. What does that look like? You ever play a video game and you have immortality? Up, down, down, left, right, left, right, you know, X, Y, X, Y. It was a Rick and Morty joke, but I think he has the code. But you're being sarcastic? Of course I am. Yes, it's a Rick and Morty joke. I don't think he's actually...

1:49:48You think he's just brilliant above and beyond his peers?

1:49:51Roman Yampolskiy: I think he doesn't care about opinions of others and just does the impossible. With his motive being... It's clearly not money-driven. I think he can't spend what he has if he had a billion bodies. He's in a regular place. I think he got that out of his system. There was a video 25 years ago when they sold PayPal and he bought a P1. Did he buy a McLaren? Oh, yeah, I saw that video. Him getting a car. He was getting out of a car and arrived. And once you get your first million-dollar car, you know, then it's like now he can buy Ferrari, the company. He can buy, you know, General Motors if he wanted to and change them up.

1:50:31Roman Yampolskiy: There is very little research in this space. We have more and more people with many billions, but how to convert those billions into happiness is a very underdeveloped project. You can make your boat bigger. You can make your toilet out of gold. But none of it is interesting. You've got to do this on X. And you'll see the video when he bought a McLaren F1. Yeah, right there. That's the one right there when he bought the McLaren. He was so excited, 99. This is, what, 26 years ago? And I think this is when he got his, it's going to take a minute for the commercial ads. You've got to ask that question on X.

1:51:01Roman Yampolskiy: If you are really successful, even your hair grows back. Yeah, your hair grows back. because he had a, have you ever had a conversation with him? A short one. So if right now you got a call from the White House and you're there, you're sitting around with them, would there be a certain specific argument you would make to him? Or would it be the same thing you just told me right now? Mr. President, you're going to lose power. You're going to be out of control. Same with Chinese Communist Party. You want to stay in control, you want to stay in power, don't create a replacement for humanity. Is this a policy that if he comes out with it, how does he win short term?

1:51:38Not long term, because Trump's interested in how he wins short term. How does he win short term?

1:51:42Roman Yampolskiy: He'll get support even from Bernie Sanders. Everyone thinks then we actually survey people. Not even technical people, normal people. Very high percentage have concerns. They are worried about it. Some worry about data centers making power more expensive, water supply. Others worry about jobs. But overall, there is a lot of suspicion and concern about what we're doing. Nobody's asking them. We're running an experiment on 8 billion people. They've never given consent to any of that. We're deploying those toys for kids and trying to figure out what's happening to them 10 years later. But now we're doing it at a different scale on everyone.

1:52:23What do you teach right now? Computer science? Is that what you're teaching?

1:52:26Roman Yampolskiy: I teach AI. What do you teach? Luckily, my course is the classic AI, not machine learning. So there is zero chance of it becoming super intelligent. Search algorithms, evolutionary computation, heuristics. And the kids that are in your class, they're in your class because they want to one day get into the business of AI? I hope so. They have some reason to be there. It's also a required course for many. Very interesting On where You know your argument is Yet you're still in it Yet you have three kids Yet you say we got to get the governments Well I had kids before they started building Superintelligence So you had the kids before If today you were 25 would you have kids today?

1:53:13Roman Yampolskiy: I would because I think we're here To collect experiences And it doesn't matter how many years you have left Having a child is still one of the best experiences But for them they may not live a full life like you and I do. It's always been the case. You never knew what was going to happen. World wars, pandemics, just bad genetics. But have we ever experienced something like AI, like super intelligent AI? It's a completely unique moment in human history. We're creating meta technology. We're creating new worlds, virtual worlds. We're creating new agents, new civilizations. We're playing God.

1:53:49Yeah. I sit there and I say, okay, say you got an argument and I agree with you. You're right. That's what's going to be happening. All right? If superintelligence takes over, eventually here's what's going to happen. If this guy is still going to be doing it, it's irrelevant. That's where my thought goes to. If he can do that for a billion dollars and an evil person can go pursue it deeper and you have to assume no other government is going to be wanting to grow it with a certain level of control, I think we're all part of the naive community.

1:54:20Roman Yampolskiy: Sounds like you're agreeing with Bernie. Billionaires are dangerous. We should text them all. No, I'm not saying. I'm saying if that's the premise that we're agreeing to, that that's what's going to happen, it is inevitable. It very well may be. That's why I have very high levels of Pidu. You have high levels of what? Probability of doom. That's a measure people report in terms of how your estimates are. But should we give up? Should we not try? I don't think so. We are still alive. We are still in charge. See, the difference is I have faith. That's the difference. You know, I have faith. In what?

1:54:57In God. I have faith in God. I believe in God. To me, you know, the difference between a group of people that have faith and you've seen what has overcome over the years, what types of crisis, what has happened, what's not happened.

1:55:13Roman Yampolskiy: Are you saying your God will prevent superintelligence from destroying humanity? Is that the argument? No, I think it's either the end of times, and we did this to ourselves, and very likely could be. We don't know that. That's a good thing, right? You get to have afterlife after the judgment. Just so you know, I'm not worried about it at all. Either way, I'm good because I know what's going to happen with me. That's the gamble I've taken, and I believe that. What do you think happens to you for somebody as, you know, What do you think happens to you when you die? I'm assuming you believe in what?

1:55:46Reincarnation? Do you believe in...

1:55:47Roman Yampolskiy: Well, I said I think it's a simulation. So either you restart or go to a different simulation. Depending on how you did, you go level up or level down. Like a video game. Yeah. Yeah. You didn't finish the level first time, you get to play again. You stay again? Yeah. It's like game over. Go back at it again? Absolutely. What do you think the next one looks like? Should be very similar. Hopefully you learn something, you'll make better decisions. Like not to build general super intelligence. If there's a handful of books for a person who's not in this business that can read to get smarter about AI, what would you recommend them reading?

1:56:22Roman Yampolskiy: It depends on your technical level. There is a very popular book, If Anyone Builds It, Everyone Will Die or something like that, Eliezer Yutkovsky, Nate Soros. There is a great book by Nick Bostrom. The one on top is by Nick Bostrom, Super Intelligence. that's the classic which really exposed a lot of it. Which one is the top one? On the previous slide it was above this one. Where you go to Amazon where you went, but on top there was the Nick Bostroms. Yeah, see right there, yeah. And of course I'm completely biased, but if you're technical and scientific, AI, unexplainable, unpredictable, uncontrollable.

1:57:06And if you have any questions for demand, You can simply go to Manect. Download it. I'm going to put the QR code somewhere around here.

1:57:11Roman Yampolskiy: Don't start tomorrow. It's my birthday. I want a day off. Ask him any questions you may have. Wish him a happy birthday. But this is a topic that I think is going to create, garner a lot of attention, a lot of conversations. Roman, it was a pleasure having you on. And hopefully you are wrong. Okay? We're going to hope you're not right and you're wrong. But if you're right, we've got to find a way to get this guy to not pursue it. Thank you so much for coming. I hope I'm wrong. Thank you so much for inviting me and we'll get that guy. Yeah, I look forward to it. Thank you. Take care. When we set out to create a shoe that blends comfort, function, and luxury, we had the choice to make it fast.

1:57:50We had the choice to make it cheap. We chose neither. Instead, we chose Tuscany. We chose true Italian craftsmanship. Each pair touched by 50 skilled hands. We chose patience, spending two years perfecting every detail, and we chose the finest quality at every step. Introducing the Future Looks Bright collection. Not rushed, not disposable, not ordinary. Rather intentional, luxurious, timeless.

1:58:54This podcast is brought to you by the U.S. Department of State. See our seven-day return policy at caravana.com. Ryan Reynolds here from Mint Mobile. I don't know if you knew this, but anyone can get the same premium wireless for$15 a month plan that I've been enjoying. It's not just for celebrities. So do like I did and have one of your assistant's assistants switch you to Mint Mobile today. I'm told it's super easy to do at mintmobile.com slash switch.

1:59:18Roman Yampolskiy: Upfront payment of$45 for three-month plan, equivalent to$15 per month required. Intro rate first three months only, then full price plan options available. Taxes and fees extra. See full terms at MintMobile.com.

1:59:56Now streaming only on Disney+.

2:00:01Roman Yampolskiy: When it comes to my policy, Farmers gives it to me straight, so there's no surprises. It's your coverage. You should never be confused. You hear that? An insurance provider with a conscience. We're a choir, but yes, we have that too. Straightforward coverage you can literally understand. That's what you get when you have Farmers. Bum, bum, bum, bum, bum, bum. Learn more at Farmers.com. Underwritten by farmer's truck or fire insurance exchanges or affiliate. Products not available in every state. You know that friend you only call when you need something? HomeServe's here for you. Got a burst pipe, HVAC outage, or electrical issue?

2:00:37HomeServe has your back for regular use, where basic homeowners insurance typically won't help. With a protection plan from HomeServe in place before disaster strikes, just call us. We'll send a qualified contractor and help you avoid unexpected expenses. Maybe we're the last ones you want to call, but we're happy to hear from you. Go to homeserve.com slash audio for 50 % less your first year. Savings compared to renewal price boy in Florida.