#371 – Max Tegmark: The Case for Halting AI Development

31:17Because you only have a finite number of days to spend them. Make a count. Be grateful. Yeah. Well, you do happen to be working on a thing which seems to have potentially some of the greatest impact on human civilization of anything humans have ever created, which is artificial intelligence. This is on the both detail technical level and in the high philosophical level you work on. So you've mentioned to me that there's an open letter Are you working on it's actually going live in a few hours. I've been having late nights and early mornings. It's been very exciting actually. In short, have you seen don't look up the film?

32:09Yes, yes. I don't want to be the movie spoiler for anyone watching this who hasn't seen it. But if you're watching this, you haven't seen it. watch it because we are actually acting out. It's life -imitating art. Humanity is doing exactly that right now except it's an asteroid that we are building ourselves. Almost nobody is talking about it. People are squabbling across the planet about all sorts of things which seem very minor compared to the asteroid that's about to hit us, right? Most politicians don't even have the radar, this on the radar, they think maybe in 100 years or whatever. Right now, we're at a fork on the road.

32:49This is the most important fork the humanity has reached and it's over 100 ,000 years on this planet. We're building effectively a new species. It's smarter than us. It doesn't look so much like a species yet because it's mostly not embodied in robots, but that's a technicality which will soon be changed. And this arrival of artificial general intelligence, they can do all our jobs as well as us, and probably shortly thereafter super intelligence, which greatly exceeds our cognitive abilities, it's gonna either be the best thing ever to happen to humanity or the worst. I'm really quite confident that there is not that much middle ground there.

33:33But it would be fundamentally transformative to human civilizations, Of course, utterly and totally. Again, we'd ran that ourselves as homo sapiens because it seemed like the basic thing where the king of the castle on this planet were the smart ones. If we can control everything else, this could very easily change. We're certainly not gonna be the smartest on the planet very long, unless AI progress just halts. And we can talk more about why I think that's true, because it's controversial. And then we can also talk about reasons you might think it's going to be the best thing ever. And the reason you think it's going to be the end of humanity, which is of course super controversial.

34:19But what I think we can, anyone who's working on advanced AI can agree on it. It's much like the film don't look up in that it's just really comical how little it's going to be. serious public debate there is about it, given how huge it is. So what we're talking about is a development of currently things like GPT -4 and the signs it's showing of rapid improvement that may in the near term lead to development of super intelligent AGI, AI, general AI systems and what kind of impact that has on society. Exactly. When that thing is achieves general human level intelligence and then beyond that general superhuman level intelligence.

35:15There's a lot of questions to explore here. So one, you mentioned HALT. Is that the content of the letter to suggest that maybe we should pause the development of these systems? Exactly. So this is very controversial. When we talked the first time, we talked about, I was involved in starting the future life institutes. We worked very hard on 2014 -2015, was the mainstream AI safety. The idea that we even could be risks and that you could do things about them. Before then, a lot of people thought it was just really cookie to even talk about it, and a lot of AI researchers felt worried that this was too flaky and it could be bad for funding and that the people who talked about it were just not didn't understand AI.

36:03I'm very, very happy with how that's gone and that now, you know, completely mainstream. You go on any AI conference and people talk about AI safety and it's a tert -nerdy technical field full of equations and similar and blah, blah. As it should be. But there is this other thing, which has been quite taboo up until now, calling for slowdown. So, what we've constantly been saying, including myself, I've been biting my tongue a lot, is that we don't need to slow down AI development. We just need to win this race, the wisdom race, between the growing power of the AI and the growing wisdom, with which we manage it.

36:50And rather than trying to slow down AI, let's just try to accelerate the wisdom, do all this technical work to figure out how you can actually ensure that your powerful AI is gonna do what you wanted to do and have society adapt also with incentives and regulations so that these things could put a good use. Sadly, that didn't pan out. The progress on technical AI and capabilities has gone a lot faster than many people thought back when we started this in 2014. Turned out to be easier to build real advanced AI than we thought. On the other side, it's gone much slower than we hoped with getting policymakers and others to actually put in place incentives in place to to make steer this in the good directions.

37:48We can know maybe we should unpack and talk a little bit about each. So why did it go faster than a lot of people thought? In hindsight, it's exactly like building flying machines. People spent a lot of time wondering about how the birds fly, you know? And that turned out to be really hard. Have you seen the Ted Talk with a flying bird? Like a flying robotic bird? Yeah, it flies around the audience, But it took a hundred years longer to figure out how to do that than for the Wright brothers to build the first airplane because it turned out there was a much easier way to fly. And evolution picked a more complicated one because it had its hands tied.

38:26It could only build a machine that could assemble itself. Which the Wright brothers didn't care about. It can only build a machine that used only the most common atoms in the periodic table. right brothers didn't care about that, they could use steel iron atoms and it had to be able to repair itself and it also had to be incredibly fuel efficient. A lot of birds use less than half the fuel of a remote control plane flying the same distance. If we humans throw a little more fuel in a roof, there you go, 100 years earlier. That's exactly what's happening now with these large language models. The brain is incredibly complicated.

39:08Many people made the mistake you're thinking we have to figure out how the brain does human level AI first before we could build in a machine. That was completely wrong. You can take an incredibly simple computational system called a transformer network and just train it to do something incredibly dumb. Just read a gigantic amount of text and try to predict the next word. And it turns out, if you just throw a ton of compute at that, and a ton of data, it gets to be frighteningly good, like GPT -4, which I've been playing with so much as it came out, right? And there's still some debate about whether that can get you all the way to full human level or not.

39:51But yeah, we can come back to the details of that and how you might get the human level AI, even if large language models don't. Can you briefly, if it's just a small tangent comment on your feelings about GPT -4? So just that you're impressed by this rate of progress, but where is it? Can GPT -4 reason? What are like the intuitions? What are human interpretable words you can assign to the capabilities of GPT -4 that makes you so damn impressed with it? I'm both very excited about it and terrified. It's interesting mixture of emotions. All the best things in life include those two somehow. Yeah, I can absolutely reason.

40:35Anyone who hasn't played with it, I highly recommend doing that before disting it. It can do quite remarkable reasoning. I've had to do a lot of things which I realized I couldn't do that myself that well, even. And obviously, it does it dramatically faster than we do, too, when you watch a type. And it's doing that while servicing a massive number of other humans at the same time. At the same time, it cannot reason as well as a human can on some tasks, just because it's obviously a limitation from its architecture. We have in our heads what in GeekSpeak is called a recurrent neural network. There are loops information you go from this neuron to this neuron and then back to this one You can like ruminate on something for a while you can self reflect a lot These large language models that are they cannot make GPT for it.

41:30It's just so called transformer Where it's just like a one -way street of information basically and geek speak. It's called a feed -forward neural network and it's only so deep So it can only do logic that's that many steps and that deep and it's not And you can create problems which will fail to solve for that reason.

41:55But the fact that it can do so amazing things with this incredibly simple architecture already is quite stunning. And what we see in my lab at MIT when we look inside large language models to try to figure out how they're doing it, which that's the key core focus of our research. It's called mechanistic interpretability in GeekSpeak. You have this machine that does something smart. You're trying to reverse version engineers. See how does it do it? I think you've also those artificial neuroscience. That's exactly what neuroscience does to do with actual brains, but here you have the advantage that you don't have to worry about measurement errors.

42:34You can see what every neuron is doing all the time. And a recurrent thing we see again and again, There's been a number of beautiful papers quite recently by a lot of researchers, some of them here in this area, is where when they figure out how something is done, you can say, oh man, that's such a dumb way of doing it. And you immediately see how it can be improved. Like for example, there was a beautiful paper recently where they figured out how a large language model stores certain facts like Eiffel Tower is in Paris. And they figured out exactly how it's stored and where the proof that they understood it was they could edit it They changed some of the synapses in it and Then they asked it where is the Eiffel Tower and it's it's in Rome And then they asked you how do you get there?

43:21Oh? How do you get there from Germany? Oh you take this train and Roma Termini train station and this and that and what might you see if you're in front of it? Oh, you might see the Colosseum So they had edited, so they literally moved it to Rome. But the way that it's storing this information, it's incredibly dumb for any fellow nerds listening to this. There was a big matrix and roughly speaking, there were certain row and column vectors which encode these things and they correspond very hand -wavily to principle components. It would be much more efficient for sparse matrix to store in the database.

44:01pace. And everything so far, we've figured out how these things do. Our ways where you can see they can easily be improved. And the fact that this particular architecture has some roadblocks built into it is in no way in a prevent crafty researchers from quickly finding workarounds and making other kinds of architectures. So it would go all the way. So it's in short, it's turned out to be a lot easier to build human close to human intelligence than we thought and that means our runway is a species. Get our shit together, it has shortened. And it seems like the scary thing about the effectiveness of large language models, some Sam Altman, every single conversation with, And he really showed that the leap from GPT -3 to GPT -4 has to do with just a bunch of hacks.

45:01A bunch of little explorations about with a smart researchers doing a few little fixes here and there. It's not some fundamental leap and transformation in the architecture. And more data and more compute. And more data and compute. But he said the big leaps has to do with not the data and the compute. But just learning this new discipline just like you said so researchers are going to look at these architectures and there might be big Leaps where you realize wait Why are we doing this in this dumb way and all of a sudden this model is 10x smarter And that that can happen on any one day on any one Tuesday or one day afternoon and then all of a sudden You have a system that's 10x smarter It seems like it's such a new discipline.

45:45It's such a new like we understand so little about why this thing works so damn well that the linear improvement of compute or exponential but the steady improvement of compute, steady improvement of the data may not be the thing that even leads to the next leap. It could be a surprise little hack that improves everything. There are a lot of little leaps here and there because because so much of this is out in the open also. So many smart people are looking at this and trying to figure out little leaps here and there and it becomes this sort of collective race where if people, a lot don't take to leave someone else with.

46:21And it's actually very crucial for the other part of it. Why do we want to slow this down? So again, what this open letter is calling for is just pausing all training of systems that are more powerful than GPT -4 for six months. Give a chance for the labs to coordinate a bit on safety and for society to adapt, give the right incentives to the labs because you know you've interviewed a lot of these people who lead these labs and you know just as well as I do that they're good people, they're idealistic people, they're doing this first and foremost because they believe that AI has a huge potential to help humanity and but at the same time they are trapped in this horrible race to the bottom.

47:13Have you read Meditations on Mollock by Scott Alexander? Yes. Yeah, it's a beautiful essay on this poem by Ginsberg where he interprets it as being about this monster. It's this game theory monster that pits people into against each other and this race to the bottom where everybody ultimately loses it. Edit the evil thing about this monster is even though everybody sees it and understands. They still can't get out of the race, right? Most a good fraction of all the bad things that we humans do are caused by malloc and I like Scott Alexander's naming of the monster so we can we humans can think of it as an if a thing If you look at why do we have overfishing?

48:01Why do we have more generally the tragedy of the commons? Why is it that? So live or a I don't know if you had her on your podcast. Yeah She's become a friend. Great. You made this awesome point recently that beauty filters that a lot of female influencers feel pressure to use are exactly malloc in action again. First nobody was using them and people saw them just the way they were. And then some of them started using it and becoming ever more plastic, fantastic. And then the other ones that weren't using it started to realize that if they want to just keep their Their market share they have to start using it too and And then you're in the situation where they're all using it and And none of them has any more market share or less than before so Nobody gained anything everybody lost And they have to keep becoming ever more plastic fantastic also, right and But nobody can go back to the old way because it's just too costly, right?

49:07Mollock is everywhere. And Mollock is not a new arrival on the scene either. We humans have developed a lot of collaboration mechanisms to help us fight back against Mollock through various kinds of constructive collaboration, the Soviet Union and the United States did sign the number of arms control treaties against Mollack who is trying to stoke them into unnecessarily risky nuclear arms races, etc. And this is exactly what's happening on the AI front. This time, it's a little bit geopolitics, but it's mostly money, where there's just so much commercial pressure. If you take any of these leaders of the top tech companies, If they just say, you know, this is too risky, I want to pause for six months.

49:57They're going to get a lot of pressure from shareholders and others. They're like, well, you know, if you pause, but those guys don't pause, we're, if you don't want to get our lunch eaten, yeah. And shareholders even have a power to replace the executives in the worst case, right? So we did this open letter because we want to help these idealistic tech executives to do what their heart tells them by providing enough public pressure on the whole sector just pause so they can all pause in a coordinated fashion. And I think without the public pressure, none of them can do it alone, push back against their shareholders, no matter how good hard as they are.

50:43Smaller is a really powerful foe. So the idea is to for the major developers of AI systems like this, so we're talking about Microsoft, Google, meta and anyone else. Open AI is very close with Microsoft software. And there are plenty of smaller players. and for example, I'm Thrope, which is very impressive. There's conjecture. There's many, many, many players that I don't want to make a long list, so leave anyone out. And for that reason, it's so important that some coordination happens, that there's external pressure on all of them, saying you all need to pause. Because then the researchers in, these organizations, the leaders, they want to slow down a bit.

51:33that they can say they're shareholders, you know, everybody's slowing down because of this pressure and it's the right thing to do. Have you seen in history their examples was possible to pop the mall? Absolutely. And even like human cloning, for example, you could make so much money on human cloning.

51:56Why aren't we doing it? Because biologists thought hard about this and frowns like this is way too risky. We've got together all in the 70s in the Silamar and decided even to stop a lot more stuff also, just editing the human germline, right, gene editing that goes into our offspring. And decided let's not do this because it's too unpredictable about what it's gonna lead to. We could lose control over what happens to our species. So they paused. There was a ton of money to be made there. So it's very doable, but you just need a public awareness of what the risks are. And the broader community coming in and saying, hey, let's slow down.

52:45And another common pushback I get today is we can't stop in the West because China. And in China and the elderly, they also get told we can't slow down because the West, because both sides think they're the good guy. But look at human cloning, you know, the China forage ahead with human cloning. There's been exactly one human cloning that's actually been done that I know of. It was done by Chinese guy. Do you know where he is now? In jail. And you know who put him there? Who? Chinese government. Not because Westerners said China, I look, this is starting. No, the Chinese government put him there because they also felt they like control.

53:28the Chinese government. If anything, maybe they are even more concerned about having controlled and western governments have no incentive of just losing control over where everything is going. And you can also see the Ernie bot that was released by I believe by due recently. They got a lot of pushback from the government and had to reign in in a big way. I think once this basic message comes out that this isn't an arms race, it's a suicide race where everybody loses if anybody's AI goes out of control. It really changes the whole dynamic. It's not... I'll say this again because this is a very basic point I think a lot of people get wrong.

54:12Because a lot of people dismiss the whole idea that AI can really get very superhuman because I think there's something really magical about intelligence such that it can only exist in you and mine because they believe that. I think it's going to kind of get to just more or less GPT -4 plus plus and then that's it. They don't see it as a suicide race. They think whoever gets that first, they're going to control the world, they're going to win. That's not how it's going to be. We can talk again about the scientific arguments from why it's not going to stop there. But the way it's going to be is if anybody completely loses control and you know, you don't care if If some someone manages to take over the world who really doesn't share your goals You probably don't really even care very much about what nationality have you're not going to like it much worse than today Who if it's if you live in or well in this topia?

55:11Who you what do you care who's created it right and if someone if it goes farther and And we just lose control even to the machines. So that it's not us versus them, it's us versus it. What do you care who created this unaligned entity which has goals different from you and is ultimately and we get marginalized, we get made obsolete, we get replaced? That's why what I mean when I say it's a suicide race. It's kind of like we're rushing towards this cliff, but the closer the cliff we get, the more scenic the views are and the more money there is there, and so we keep going, but we have to also stop at some point, right, quit while we're ahead.

56:00And it's a suicide race, it cannot be won, but the way to really benefit from it is to continue developing awesome AI a little bit slower. So we make it safe, make sure it does the things that humans want and create a condition where everybody wins. The technology has shown us that geopolitics and politics in general is not a zero sum game at all. So there is some rate of development that will lead us as a human species to lose control of this thing. And the hope you have is that there's some lower level of development, which will not, which will not allow us to lose control. This is an interesting thought you have about losing control.

56:50So if you have somebody, if you are somebody like son of a child or a Sam Altman at the head of a company like this, you're saying if they develop an AGI, they too will lose control of it. So no one person can maintain control. No good, brilliant visuals can maintain control. If it's created very, very soon and is a big black box that we don't understand, like the large language models, yeah, then I'm very confident they're gonna lose control. But this isn't just me saying, you know, Sam Altman and Demis Ashabis have both said, they themselves acknowledge that there's really great risks with this and they want to slow down once they feel it gets scary.

57:29But it's clear that they're stuck in this, again, Malaq is forcing them to go a little faster than they're comfortable with because of pressure from just commercial pressures, right? To get a bit optimistic here, of course, this is a problem that can be ultimately solved. To win this wisdom race, it's clear that what we hope that was going to happen hasn't happened. And the capability progress has gone faster than a lot of people thought and the progress in the public sphere of policy making and so on has gone slower than we thought, even the technical AI safety has gone slower. A lot of the technical safety research was kind of banking on that large language models and other poorly understood systems couldn't get us all the way, but you had to build more of a kind of intelligence that you could understand.

58:18and maybe it could prove itself safe, you know, things like this. And I'm quite confident that this can be done so we can reap all the benefits, but we cannot do it as quickly as this out of control express train we are on now as you know, get the AGI. That's why we need a little more time, I feel. Is there something you've said with like Sam Alman talked about, which is while we're in the pre -AGI stage to release often and as transparently as possible to learn a lot. So as opposed to being extremely cautious, release a lot. Don't invest in a closed development where you focus on AI safety while it's somewhat dumb.

59:07Quarant quote, release as often as possible. And as you start to see signs of human level intelligence or super human level intelligence, then you put a halt on it. Well, what a lot of safety researchers have been saying for many years is that most dangerous things you can do within AI is first of all teach it to write code. Yeah, because that's the first step towards recursive self -improvement, which you can take it from AI to much higher levels. Oops, we've done that. And another thing, high risk is connected to the Internet. Let it go to websites, download stuff on its own, talk to people.

59:49Oops, we've done that already. You know, Elias Yukowski, you said you interviewed him recently, right? Yes, he had this tweak recently, which is, gave me one of the best laughs in a while, where he was like, hey, people used to make fun of me and say, you're so stupid, Elias, because you're shang, you're saying you have to worry. Obviously, developers, once they get to really strong AI, first thing you're gonna do is never connect it to the internet, keep it in a box where you can really study it. So he had written it in the meme form, so it was like then. And then that. And now, let's make a chatbot.

1:00:31And the third thing is Stuart Russell. Yeah. You know, amazing AI research. He has argued for a while that we should never teach AI anything about humans. Above all, we should never let it learn about human psychology and how you manipulate humans. That's the most dangerous kind of knowledge you can give it. Yeah, you can teach it all. It needs to know how to cure cancer and stuff like that. but don't let it read Daniel Connemon's book about cognitive biases and all that. And then, oops, LOL, let's invent social media, I'll recommend our algorithms, we do exactly that. They look, get so good at knowing us, pressing our buttons that we're starting to create a world now where we just have ever more hatred.

1:01:23because they figured out that these algorithms, not for out of evil, but just to make money on advertising that the best way to get more engagement with euphemism, get people glued to their little rectangles, or it's just to make them pissed off. Well, that's really interesting that a large AI system that's doing the recommended system kind of task on social media is basically just studying human beings because it's a bunch of us rats giving it signal, non -stop signal. It'll show a thing and we give signal and whether we spread that thing, we like that thing that thing increases our engagement, gets us to return to the platform and it has that on the scale of hundreds of millions of people constantly.

1:02:06So it's just learning and learning and learning and presumably if the parameter, the number of parameters in neural network that's doing the learning and more and to end the learning is the more it's able to just basically encode how to manipulate human behavior, how to control humans at scale. Exactly. And that is not something I think is in humanity's interest. Yes. Right now it's mainly letting some humans manipulate other humans for profit and power, which already caused a lot of damage. And eventually that's a sort of skill that can make AI's persuade humans to let them escape, whatever safety precautions we have.

1:02:48But you know, there was a really nice article in the New York Times recently by you've all know a Harari and two co -authors including Justin Harris from the social dilemma. We have this phrase in there I love. Humanity's first contact with advanced AI was social media and we lost that one. We now live in a country where there's much more hate in the world where there's much more hate, in fact. In our democracy, there we're having this conversation and people can't even agree on who won the last election. And we humans often point fingers at other humans and say it's their fault, but it's really malloc and these AI algorithms.

1:03:35We got the algorithms and then malloc pitted the social media companies against each other. So nobody could have a less creepy algorithm because then they would lose out on an algorithm revenue to the other company. Is there anywhere to win that battle back just if we just linger on this one battle that we've lost in terms of social media? Is it possible to redesign social media? This very medium in which we use as a civilization to communicate with each other to have these kinds of conversation, to have discourse to try to figure out how to solve the biggest problems in the world whether that's nuclear war or the development of H .I.

1:04:10is it possible to do social media correctly? It's not only possible, but it's necessary. Who are we kidding? They're going to be able to solve all these other challenges if we can't even have a conversation with each other. It's constructive. The whole idea of democracy is that you get a bunch of people together and they have a real conversation. The one you try to foster on this podcast where you respectfully listen to people you disagree with. And you realize But there are some things, actually, we have some common ground we have. We both agree, let's not have a nuclear wars, let's not do that, etc.

1:04:47We're kidding ourselves, thinking we can face off the second contact with ever more powerful AI that's happening now with these large language models if we can't even have a functional conversation in the public space. That's why I started the improved the news project, improved the news .org. But I am an optimist fundamentally in, and there is a lot of intrinsic goodness in people. And that what makes the difference between someone doing good things for humanity and bad things is not some sort of fairy tale thing that this person was born with the evil gene. and this one is not born with a good gene.

1:05:33No. I think it's whether we put, whether people find themselves in situations that bring out the best in them or that bring out the worst in them. And I feel we're building an internet and a society that brings out the worst. But it doesn't have to be that way. No, it does not. The if you get a sponsor to create incentives and also create incentives that make money, they both make money and bring out the best in people. I mean, in the long term, it's not a good investment for anyone, you know, to have a nuclear war, for example. And you know, is it a good investment for humanity? If we just ultimately replace all humans by machines and then we're so obsolete that eventually, they're no humans left.

1:06:18Well, it depends against how you do the math, but I would say by any reasonable, economic started. If you look at the future income of humans and there aren't any, you know, that's not a good investment. Moreover, why can't we have a little bit of pride in our species? Dammit. Why should we just build another species that gets rid of us? If we were in the Anderthal's, would we really consider the smart move if we had really advanced biotech, the build homo sapiens. You know, you might say, hey, Max, you know, yeah, let's build these homo sapiens. They're going to be smarter than us. Maybe they can help us defend this better against predators and help fix their parcaves, make them nicer and we'll control them undoubtedly.

1:07:07You know, so then they build a couple, a little baby girl, a little baby boy, they either. And then you have some wise old and the Andrithal elder is like, hmm, I'm scared that we're opening a Pandora's box here and that we're going to get outsmarted by these. Super Neanderthal intelligences and there won't be any Neanderthals left. But then you have a bunch of others in the cave. You are, you're such a Luddite scaremonger. Of course, they're going to want to keep us around because we are their creators and the smarter, I think the smarter they get, the nicer they're going to get, they're going to leave us, they're going to want this around and it's going to be fine.

1:07:50And besides, look at these babies, they're so cute. Clearly, they're totally harmless. Those babies are exactly GPT -4. It's not, I want to be clear. It's not GPT -4 that's terrifying. It's the GPT -4 is a baby technology. And Microsoft off the even had a paper recently out, with a title, something like Sparkles of AGI. Whatever basically you're saying, this is baby AI, like these little Neanderthal babies. It's going to grow up. There's going to be other systems from the same company, from other companies. They'll be way more powerful, but they're going to take all the things ideas from these babies.

1:08:33And before we know it, we're going to be like those last the underthoughts were pretty disappointed when they realized that they were getting replaced. Well, this interesting point you make which is the programming is it's entirely possible the GPT -4 is already the kind of system that can change everything by writing programs. Yeah, it's because it's life 2 .0. I, the systems I'm afraid of are going to look nothing like a large language model and they're not. But once it gets, once it or other people figure out a way of using this tech to make much better tech, right? It's just constantly replacing it software and from everything we've seen about how, how these worked under the hood, they're like the minimum viable intelligence.

1:09:23They do everything in a dumbest way that still works sort of. Yeah. And so they are life 3 .0 except when they replace their software, it's a lot faster than when you decide to learn Swedish. And moreover, they think a lot faster than us too. So when we don't think of how one logical step every nanoseconder or so, the way they do. and we can't also just suddenly scale up our hardware massively in the cloud. It's so limited, right? So they are also like to consume become a little bit more like life people know in that if they need more hardware, hey, just rent it in the cloud, you know. How do you pay for it?

1:10:17Well, all the services you provide.

1:10:22And what we haven't seen yet, which could change a lot, is an entire software system. So right now programming is done sort of in bits and pieces as an assistant tool to humans. But I do a lot of programming and with the kind of stuff that GBT4 is able to do, I mean, it's replacing a lot when I'm able to do. But you still need a human and a loop to kind of manage the design of things, manage like what are the prompts that generate the kind of stuff to do some basic adjustment of the code to do some debugging. But if it's possible to add on top of GPT for kind of feedback loop of self debugging, improving the code.

1:11:11and then you launch that system on to the wild on the internet because everything is connected and have it do things Have it interact with humans and then get that feedback now you have this giant ecosystem. Yeah of humans This one of the things that yeah, Elon Musk recently sort of tweeted as a case why everyone needs to pay seven dollars or whatever for Twitter Make sure they're real They're make sure they're real. We're now going to be living in a world where the the bots are getting smarter and smarter and smarter to a degree where where you can't tell the difference between a human and a bot.

1:11:45That's right. And now you can have bots outnumber humans by a 1 million to 1, which is why he's making a case, why you have to pay to prove your human, which is one of the only mechanisms to prove, which is depressing. And I feel we have to remember, as individuals, we should, from time to time, ask ourselves, why are we doing what we're doing? All right, and as a species, we need to do that too. So, if we're building, as you say, machines that are outnumbering us and more and more outsmarting us and replacing us on the job market, not just for the dangerous and boring tasks, but also for writing poems and doing art and things that a lot of people find really meaningful, God, ask us, why?

1:12:31Why are we doing this? is the answer is malloc is tricking us into doing it. It's such a clever trick that even though we see the trick, we still have no choice but to fall for it. Also, the thing you said about you using co -pilot AI tools to program faster out. What factor faster would you say you code now? It's like, oh twice as fast or? I don't really, because it's a new tool. Yeah. It's, I don't know if speed is significantly improved, but it feels like I'm a year away from being 5 to 10 times faster. So if that's typical for programmers, then you're already seeing another kind of self, of course, of self -improvement, right?

1:13:24Because previously, one, like a major generation of improvement of the codes would have of the human R &D timescale. And now if that's five times shorter, then it's gonna take five times less time than otherwise would to develop the next level of these tools and so on. So these are the, this is exactly the sort of beginning of an intelligence explosion. There can be humans in the loop a lot in the early stages. And then eventually humans are needed less and less. And the machines can more kind of go along. But what you said there is just an exact example of these sort of things. Another thing which I was kind of lying on my psychiatrist imagining I'm on a psychiatrist's couch here saying, what are my fears that people would do with AI systems?

1:14:11Another, so I mentioned three that I had fears about many years ago that they would do, namely, Teach of the Code, connected to the internet and teach it to manipulate humans. a fourth one is building an API where code can control this super powerful thing, right? That's very unfortunate because one thing that systems like GPT -4 have going for them is that they are an Oracle in the sense that they just answer questions. There is no robot connected to GPT -4, GPT -4 can't go and do stock trading based on its thinking. It is not an agent. Intelligent agent is something that takes in information from the world, processes it, the figure I want action to take based on its goals that it has, and then does something back on the world.

1:15:04But once you have an API, for example, GPT -4, nothing stops Joe Schmo and a lot of other people from building real agents, which just keep making calls somewhere in some inner loop, somewhere to these powerful Oracle systems, which makes them themselves much more powerful. That's another kind of unfortunate development, which I think we would have been better off delaying. I don't want to pick on any particular companies. I think they're all under a lot of pressure to make money.

1:15:40And again, the reason we're calling for this pause is to give them all cover to do what they know is the right thing. Slow down a little bit at this point. But everything we've talked about, I hope we'll make it clear that people watching this, why these sort of human level tools can cause a gradual acceleration. You keep using yesterday's technology to build tomorrow's technology. And when you do that over and over again, you naturally get an explosion. That's the definition of an explosion in science. right? Like if you have two people, they fall in love, now we have four people and then they can make more babies and now you have eight people and then then you have 16, 32, 64, etc.

1:16:29That's, we call that an population explosion, whereas just that each, if it's instead free neutrons in a nuclear reaction, that if each one can make more than one, then you get an exponential growth and that we call it a nuclear explosion. All explosions are like that. And an intelligence explosion is exactly the same principle. That some quantum, some amount of intelligence can make more intelligence than that. And then repeat. You always get the exponentials. What's your intuition? Why does you mention there's some technical reasons why it doesn't stop at a certain point? What's your intuition?

1:17:03And do you have any intuition? Why it might stop? It's obviously going to stop when it bumps up against the laws of physics. There are some things you just can't do no matter how smart you are, allegedly. Because we don't have the full laws of physics. Yeah, I get it right. Seth Lloyd wrote a really cool paper on the physical limits on computation, for example, if you make it, put too much energy into it and then find it's based, it'll turn into a black hole. You can't move information around fast in the speed of light, stuff like that, but it's hard to store way more than a modest number of bits per atom, etc.

1:17:40But, you know, those limits are just astronomically above, like, 30 orders of magnitude above where we are now. So,

1:17:51bigger, different, bigger jump in intelligence than if you go from an ant to a human. I think, of course, what we want to do is have a controlled thing. A nuclear reactor, you put moderators in to make sure exactly it doesn't blow up out of control, right? when we do experiments with biology and cells and so on. We also try to make sure it doesn't get out of control. We can do this with AI too. The thing is, we haven't succeeded yet. And malloc is exactly doing the opposite, just fueling, just egging everybody on, faster, faster, faster. Or the other company is gonna catch up with you or the other country is gonna catch up with you.

1:18:39We have to want to stop. I don't believe in this just asking people to look into their hearts and do the right thing. It's easier for others to say that. But if you're in a situation where your company is going to get screwed, by other companies they're not stopping. Putting people in a very hard situation, the right thing to do is change the whole incentive structure instead. And this is not an old, maybe I should say one more thing about this, because Mollock has been around as humanity's number one or number two anime since the beginning of civilization. And we came up with some really cool countermeasures.

1:19:24Like, first of all, already over 100 ,000 years ago, evolution realized that it was very unhelpful that people kept killing each other all the time. So it genetically gave us compassion and made it so that if you get too drunk dudes getting into a pointless bar fight, they might give each other black eyes, but they have a lot of inhibition towards just killing each other. That's a, and similarly if you find a baby lying on the street when you go out for your morning jog tomorrow, you're gonna stop and pick it up, right? Even know it may be make you late for your next podcast. So evolution gave us these genes that make our own egoistic incentives more aligned with what's good for the greater group or part of.

1:20:13Right. And then as we got a bit more sophisticated and developed language, we invented gossip, which is also a fantastic anti -moloc. Right. Because now it really discourages liars, moochers, cheaters, because their own incentive now is not to do this, because word quickly gets around and then suddenly people aren't going to invite them to their dinners anymore or trust them. And then when we got still more sophisticated and bigger societies, you know, invented the legal system where even strangers who didn't could rely on gossip and things like this would treat each other, would have an incentive.

1:20:55Now, those guys in the bar fight, even if someone is so drunk that he actually wants to kill the other guy, he also has a little thought in the back of his head that, you know, do I really want to spend the next 10 years eating like really crappy food in a small room? I'm just going to chill out, you know. And we similarly have tried to give these incentives to our corporations by having regulation and also some oversight so that their incentives are aligned with the greater good. We tried really hard. And the big problem that we're failing now is not that we haven't tried before, but it's just that the tech is growing much, is developing much faster than the regulators been able to keep up, right?

1:21:41So regulators, it's kind of comical. The European Union right now is doing this AI act, right? which in the beginning they had a little opt -out exception that GPT -4 would be completely excluded from regulation. Brilliant idea. It was the logic behind that. Some lobbyists pushed successfully for this. So we were actually quite involved with the future life institute, Mark Brakel, Risto Uck, Anthony McGee, and others. We're quite involved with talking to very educating various people involved in this process about these general purpose AI models coming and pointing out that they would become the laughing stock if they didn't put it in.

1:22:25So the French started pushing for it, it got put in to the draft and it looked like all was good. Then there was a huge counter push from lobbyists. There were more lobbyists and Brussels from tech companies and from oil companies, for example. And it looked like it might maybe get taken out again. And now GPT -4 happened. And I think it's going to stay in. But this just shows, you know, malloc can be defeated. But the challenge we're facing is that the tech is generally much faster than what the policymakers are. And a lot of the policymakers also don't have a tech background. So we really need to work hard to educate them on what's taking place here.

1:23:12So we're getting the situation where the first kind of non, so I define artificial intelligence just as non -biological intelligence. And by that definition, a company, a corporation is also an artificial intelligence because the corporation isn't its humans, it's the system. If a CEO decides, if a CEO of a tobacco company decides one morning, he doesn't want to sell cigarettes anymore, they'll just put another CEO in there. It's not enough to align the incentives of individual people or in -line individual computers incentives to their owners, which is what technically I safety research is about.

1:23:54You also have to align the incentives of corporations with a greater good. And some corporations have gotten so big and so powerful very quickly that in many cases is their lobbyists instead align the regulators to what they want rather than the way around. It's a classic regulatory capture. Right. Is the thing that the slowdown hopes to achieve is given enough time to the regulators to catch up or enough time to the companies themselves to breathe and understand how to do AI safety correctly? I think both. And I think that the vision of the path to success, I see, is first you give a breather actually to the people in these companies, their leadership who wants to do the right thing, and they all have safety teams and so on on their companies.

1:24:41Give them a chance to get together with the other companies, and the outside pressure can also help catalyze that, right? And work out what is it that's what are the reasonable safety requirements one should put on future systems before they get rolled out. There are a lot of people also in academia and elsewhere outside of these companies who can be brought into this and have a lot of very good ideas. And then I think it's very realistic that within six months you can get these people coming up. So here's a white paper, here's where we all think it's reasonable. You know, you didn't just because cars killed a lot of people, you didn't ban cars, but that they got together, a bunch of people on the site.

1:25:28And in order to be allowed to sell a car, it has to have a seat belt in it. There are the analogous things that you can start requiring a future AI systems so that they are safe. And once this heavy lifting, this intellectual work has been done by experts in the field, which can be done quickly, I think it's been going to be quite easy to get policy makers to see, Yeah, this is a good idea. And it's, you know, for the companies to fight malloc, they want, and I believe Sam Altman has explicitly called for this, they want the regulators to actually adopt it so that their competition is gonna divide by two, right?

1:26:12You don't want, you don't want to be, enacting all these principles, then you abide by them, and then there's this one little company that doesn't sign onto it, And then they can gradually overtake you. Then the companies will get Be able to sleep secure knowing that everybody is playing by the same rules. So do you think it's possible to develop guard rails? That keep the systems from From basically Damaging irreparably humanity While still enabling sort of the capitalist fueled competition between companies as they develop how to best make money with this AI You think there's a balancing? Totally.

1:26:57That's possible. Absolutely. We've seen that in many other sectors where you've had the free market produce quite good things without causing particular harm. When the guardrails are there and they work, capitalism is a very effective, good way of optimizing for just getting the same things on more efficiently. It was good. I can hindsight, I never met anyone even on parties way over on the right in any country who thinks it was a terrible idea to ban child labor, for example. Yeah, but it seems like this particular technology has gotten so good, so fast, become powerful to degree where you could see in the near term the ability to make a lot of money Yeah, and to put guard rails develop guard rails quickly in that kind of contact seems to be tricky.

1:27:51It's not Similar to cars or child labor. It seems like the opportunity to make a lot of money here very quickly is right here There's this cliff. Yeah, it gets quick Closer the cliff. There you go. Yeah, one of the more Go there are more money there is more gold and get throw on the ground you can pick up or whatever you want to drive there very fast, but it's not in anyone's incentive that we go over the cliff and it's not like everybody's in their own car. All the cars are connected together with a chain. So if anyone goes over, they'll start dragging others down, the others down too. And so ultimately it's in the selfish interest also of the people in the companies to slow down when they just start seeing the contours of the cliff there in front of you, right?

1:28:38And the problem is that even though the people who are building the technology and the CEOs, they really get it. The shareholders and these other market forces, they are people who don't honestly understand that the cliff is there. They usually don't, you have to get quite into the weeds to really appreciate how powerful this is and how fast and a lot of people are even still stuck again in this idea that intelligence in this carbon chauvinism as I like to call it, that you can only have our level of intelligence in humans. There's something magical about it, whereas the people in the tech companies who build this stuff, they all realize that intelligence is information processing of a certain kind.

1:29:24And it really doesn't matter at all whether the information is processed by carbon atoms in neurons and brains or by silicon atoms and some technology we build. So you brought up capitalism earlier and there are a lot of people who love capitalism and a lot of people who really, really don't. And it struck me recently that what's happening with capitalism here is exactly analogous to the way in which super intelligence might wipe us out. So, you know, I studied economics for my undergrad, stock one school economics, yay. No, no, I tell them. So I was very interested in how you could use market forces to just get stuff done more efficiently, but give the right incentives to market so that it wouldn't do really bad things.

1:30:19So Dylan had Phil Manel, who's a professor and colleague of mine in the MIT wrote this really interesting paper with some collaborators recently, where they proved mathematically that if you just opt take one goal that you just optimized for, on and on and on indefinitely, that you think is gonna bring you in the right direction. What basically always happens is, in the beginning, it will make things better for you. But if you keep going, at some point, it's gonna start making things worse for you again. and then gradually it's going to make it really, really terrible. So just as a simple, the way I think of the proof is, suppose you want to go from here back to Austin, for example.

1:31:04And you're like, okay, yeah, let's just go south, but you put in exactly the right sort of the right direction. Just optimize that. South is possible. You get closer and closer to Austin. But there's always some little error. So you're not going exactly towards Austin, but you get pretty close. But eventually you start going away again, and eventually you're going to be leaving the solar system. Yeah. And they proved it's a beautiful mathematical proof. This happens generally. And this is very important for AI, because even though Stuart Russell has written a book and given a lot of talks on why it's a bad idea to have AI just blindly optimize something, That's what pretty much all our systems do.

1:31:48Yeah, we have something called the loss function that we're just minimizing or reward function, just minimize, maximize, and capitalism is exactly like that too. We want to get stuff done more efficiently that people wanted. So, we introduced the free market. Things got done much more efficiently than they did in, say, communism. and it got better. But then it just kept optimizing and kept optimizing. You got ever bigger companies and ever more efficient information processing. And I was also very much powered by IT. And eventually a lot of people are beginning to feel, wait, we're kind of optimizing a bit too much.

1:32:35Like, why did we just chop down half the rainforest? And why did suddenly these regulators get captured by lobbyists and so on. It's just the same optimization that's been running for too long. If you have an AI that actually has power over the world and you just give it one goal and just like keep optimizing that, most likely everybody's going to be like, yeah, this is great in the beginning, things are getting better. But it's almost impossible to give it exactly the right direction to optimize in. And then eventually I'll hate break loose, right? Nick Bossdram and others are given an example to sound quite silly, like what if you just want to like tell it to cure cancer or something.

1:33:22And that's all you tell it. Maybe it's going to decide to take over entire continent just so we can get more supercomputer facilities in there and figure out how to cure cancer backwards. And then you're like, wait, that's not what I want, then. And the issue with capitalism and the issue in front of my way, I have kind of merged now because the malloc I talked about is exactly the capitalist malloc that we have built an economy that has its optimized for only one thing, profit. And that worked great back when things were very inefficient and then now it's getting done better. And it worked great as long as the companies were small enough that they couldn't capture the regulators, but That's not true anymore, but they keep optimizing and now we They realize that they can these companies can make even more profit by building ever more powerfully.

1:34:17I even if it's reckless

1:34:21But optimized more more more more more more So this is malloc again showing up and I just want to anyone here has any concerns about about late stage capitalism having gone a little too far? You should worry about superintelligence because it's the same villain in both cases. It's malloc. And optimizing one objective function aggressively blindly is going to take us there. Yeah, we have this pause from time to time and look into our hearts and ask, why are we doing this? Is this, am I still going towards Austin or have I gone too far? You know, maybe we should change direction. And that is the idea behind the halt for six months.

1:35:09Why six months? It seems like a very short period. Just can we just linger and explore different ideas here? Because this feels like a really important moment in human history, where a positive would actually have a significant positive effect. We said six months because we figured the number one pushback we were going to get in the West was like, but China.

1:35:37And everybody knows there's no way that China is going to catch up with the West on this in six months. So that argument goes off the table and you can forget about geopolitical competition and just focus on the real issue. That's why we put this. That's really interesting. But you've already made the case that even for China, if you actually want to take on that argument, China too would not be bothered by a longer halt because they don't want to lose control even more than the West doesn't. That's what I think. That's a really interesting argument. I have to actually really think about that, which the kind of thing people assume is if you develop an AGI that open AI, if They're the ones that do it, for example, they're going to win.

1:36:22But you're saying, no, everybody loses. Yeah, it's going to get better and better and better and then kebab when we all lose. That's what's going to happen. When losing, when a define an ametric of basically quality of life for human civilization and for Sam Altman. To be blunt, my personal guess, and people think quibble with this is that we're just going to, there won't be any humans. That's it. That's what I mean by lose. You know, if you, we can see in history, once you have some species or some group of people who aren't needed anymore, doesn't usually work out so well for them, right? Yeah.

1:37:02There were a lot of horses for, they were used for traffic in Boston and then the car got invented and most of them got, But you know, we don't need to go there. And if you look at humans, you know, right now, why did the labor movement succeed? And after the industrial revolution? Because it was needed. Even though we had a lot of mollocks and there was child labor and so on, you know, the companies still needed to have workers. And that's why strikes had power and so on. If we get to the point where most humans aren't needed anymore, I think it's quite naive to think that they're going to still be treated well.

1:37:48You know, we say that. Yeah, yeah, everybody's equal. The government will always protect them. But if you look in practice, groups that are very disenfranchised and don't have any actual power, usually get screwed. And now in the beginning, so industrial revolution, we automated away muscle work. But that got, went, worked out pretty well eventually because we educated ourselves and started to do working with our brains instead and got, usually, more interesting, better paid jobs. But now we're beginning to replace brain work. So we replaced a lot of boring stuff like we got the pocket calculator so you don't have people adding, multiplying numbers anymore at work.

1:38:32Fine, there were better jobs they could get. But now, GPT -4, you know, and the stable diffusion and techniques like this, they're really beginning to blow away some jobs that people really love having. It was a heartbreaking article just post just yesterday. Social media, I saw, but this guy who was doing 3D modeling for gaming and he... And all of a sudden now they got this new software. He just gives says prompts and he feels his whole job that he loved Slossed its meaning, you know, and I Asked the GPT -4 to rewrite Twinkle Twinkle little star in the style of Shakespeare. I Couldn't have done such a good job.

1:39:15It was just really impressive. You've seen a lot of the arts coming out here, right? So I'm all for Or automated way of the dangerous jobs and the boring jobs. But I think you hear some arguments which are too glib. Sometimes people say, well, that's all that's going to happen. We're getting rid of the boring tedious, dangerous jobs. It's just not true. There are a lot of really interesting jobs that are being taken away now. Journalism is getting crushed. Coding is going to get crushed. I predict the job market for programmers, the salaries are going to start dropping. You said you can code five times faster, then you need five times fewer programmers.

1:39:58Maybe there will be more output also, but you'll still end up using fewer programmers needing fewer programmers than today. I love coding. I think it's super cool. We need to stop and ask ourselves why again now we're doing this as humans. I feel that AI should be built by humanity for humanity. And let's not forget that. It shouldn't be by malloc, for malloc. Or what it really is now is kind of by humanity for malloc which doesn't make any sense. It's for us that we're doing it. And make a lot more sense if we build, develop, figure out gradually safely how to make all this tech. And then we think about what are the kind of jobs the people really don't want to have, you know, automate them all the way.

1:40:48And then we ask, what are the jobs that people really find meaning in, like maybe taking care of children, the daycare center, maybe doing art, etc., etc. And even if it were possible to automate that way, we don't need to do that, right? We built these machines. It was possible that we redefine, or rediscover what are the jobs that give us meaning. So for me, the thing, it is really sad. Like, I have the time I'm excited, have the time I'm crying, as I'm generating code, because I kind of love programming. It's an act of creation. You have an idea, you design it, and then you bring it to life, and it does something, especially if there's some intelligence to it, it does something.

1:41:40It doesn't even have to have intelligence. Printing Holo World on screen, you made a little machine and it comes to life. And there's a bunch of tricks you learn along the way because you've been doing it for many, many years. And then to see AI, be able to generate all the tricks you thought were special. I don't know. It's very, it's scary. It's almost painful like a loss loss of innocence maybe like yeah, maybe when I was younger I remember before I learned that sugar is bad for you. You should be on a diet I remember I enjoyed candy deeply in a way. I just can't anymore That I know is bad for me.

1:42:27I enjoyed it unapologetically fully just Intensely and I just I lost that now I feel like a little bit that is lost for me with program or being lost with programming similar as it is for The 3d modeler no longer being able to really enjoy the art of modeling 3d things for gaming. I don't know. I don't know what to make sense of that Maybe I would rediscover that the true magic of what it means to be humans connecting without the humans to have conversations like this I don't know to to have sex, to have to eat food, to really intensify the value from conscious experiences versus like creating other stuff.

1:43:08You're pitching the rebranding again from homo sapiens, the homo sapiens, the meaningful experiences. And just to inject some optimism in this here so we don't sound like it's a gloomers, we can totally have our cake and eat it. You hear a lot of totally bullshit claims that we can't afford to have a more teachers. have to cut the number of nurses. You know, that's just nonsense, obviously. With anything, even quite far short of AGI, we can dramatically improve, grow the GDP, and produce this wealth of goods and services. It's very easy to create a world where everybody is better off than today, including the richest people can be better off as well.

1:43:52It's not a zero -sum game technology. You can have two countries like Sweden and Denmark, all these ridiculous wars, century after century.

1:44:05And sometimes that Sweden got a little better off because it got a little bit bigger, and then Denmark got a little bit better off because it's a little bit smaller. But then technology came along, and we both got just dramatically wealthier without taking away from anyone else. So it's just a total win for everyone. And AI can do that on steroids. If you can build safe AGI, if you can build super intelligence, basically all the limitations that cause harm today can be completely eliminated. So wonderful. You talk possibility. And this is not sci -fi. This is something which is clearly possible according to the laws of physics.

1:44:43And we can talk about ways of making it safe, also. But unfortunately, that'll only happen if we steer in that direction. That's absolutely not the default outcome. That's why income inequality keeps going up. That's why the life expectancy in the US has been going down now. I think it's four years in a row. I was just read a heart -breaking study from CDC about how something like one -third of all teenage girls in the US been thinking about suicide, you know, like those are steps in their totally the wrong direction and and it's important to keep our eyes on the prize here that we can we have the power now for the first time in the history of our species to harness artificial intelligence to help us really flourish and help bring out the best in our humanity rather than the worst of it to help us have really fulfilling experiences that feel truly meaningful.

1:45:50And you and I shouldn't sit here and dictate the future generations what they will be. Let them figure it out, but let's give them a chance to live and not for close all these possibilities for them, might just mess things up. And for that, we'll have to solve the AI safety problem. It would be nice if we can link on exploring that a little bit. So one interesting way to enter that discussion is you tweeted and Elon replied you tweeted let's not just focus on whether GPT -4 will do more harm or good on the job market, but also whether it's coding skills will hasten the arrival of super intelligence.

1:46:25That's something we've been talking about. So Elon proposed one thing in the reply saying maximum truth seeking is my best guess for AI safety. Can you maybe steal me on the case for this subject of function of truth and maybe make an argument against it in general? What are your different ideas to start approaching the solution to AI safety? I didn't see that reply actually. I went so but I really resonate with it because AI is not evil. It caused people around the world to hate each other much more, but that's because we made it in a certain way. It's a tool. We can use it for great things and bad things.

1:47:11And we could just as well have AI systems. And this is part of my vision for success here. Truth -seeking AI that really brings us together again. You know, why do people hate each other so much between countries and within countries? It's because because they have totally different versions of the truth, right? If they all had the same truth that they trusted for good reason, because they could check it and verify it and not have to believe in some self -proclaimed authority, right? They wouldn't be as nearly as much hate. There would be a lot more understanding instead. And this is, I think, something AI can help enormously with.

1:47:52For example, a little baby step in this direction is this website called Metaculous, where people bet and make predictions not for money, but just for their own reputation. And it's kind of funny actually, you treat the humans like you treat AI as you have a loss function where they get penalized if they're super confident on something and then the opposite happens. Yeah. Whereas if you're kind of humble and then you're like, I think it's 51 % chance this is going to happen and then the other happens, you don't get penalized much. And what you can see is that some people are much better at predicting than others.

1:48:30They've earned your trust, right? One project that I'm working on right now is the Outgrowth Improve the News Foundation together with the MetaXylist folks is seeing if we can really scale this up a lot with more powerful AI. I would love it, I would love further to be a really powerful truth seeking system where that is trustworthy because it keeps being right about stuff and people who come to it and maybe look at its latest trust ranking of different pundits and newspapers etc. If they want to know why someone got a low score, they can click on it and see all the predictions that they actually made and how they turned out.

1:49:14This is how we do it in science. You trust scientists like Einstein who said something, everybody thought it was bullshit and turned out to be right. You get a lot of trust point and he did it multiple times even. I think AI has the power to really heal a lot of the rifts we're seeing by creating trust system. It has to get away from this idea today with some fact checking site which might themselves have an agenda and you just trust it because of its reputation. You want to have at the, so these sort of systems, the urn, their trust and their completely transparent. This I think would actually help a lot.

1:49:57That I think help heal the very dysfunctional conversation that humanity has about how it's going to deal with all its biggest challenges in the world today. And then on the technical side, another common sort of gloom comment I get from people saying we're just screwed, there's no hope is, well, things like GPT -4 are way too complicated for a human to ever understand and prove that they can be trustworthy. They're forgetting that AI can help us prove the things work. Right. Yeah. And there's this very fundamental fact that in math, it's much harder to come up with a proof that it is to verify that the proof is correct.

1:50:42You can actually write a little proof checking code. It's quite short. But you can as human understand. And then it can check the most monstrously long proof ever generated even by a computer. And So yeah, this is valid. So right now, we have this approach with virus checking software that it looks to see if there's something you should not trust it. And if it can prove to itself that you should not trust that code, it warns you. Mine. Why did you flip this around? And this is an idea. Give credit to Steve, I'm 100 or four. So that it will only run the code if it can prove, instead of not running it, if it can prove that it's not trustworthy, if it will only run it, if it can prove that it's trustworthy.

1:51:31So that's the code. It proved to me that you're going to do what you say you're going to do. And it gives you this proof. And you're a little proofer, you can check it. Now you can actually trust an AI that's much more intelligent than you are. right? Because it's problem to come up with this proof that you could never have found that you should trust it. So this is the interesting point. I agree with you, but this is where Elias Erickowski might disagree with you. His claim, not with you, but with this idea. His claim as a super intelligent AI would be able to know how to lie to you with such a proof.

1:52:14How to like you and give me a proof that I'm going to think is correct. Yeah, but it's not me is lying to you. That's the trick my proof checker. So he's a code. So his general idea is a super intelligent system can lie to a Dumber proof checker. So you're going to have as a system because more and more intelligent there's going to be a threshold where a super intelligent system will be able to effectively lie just slightly dumber aGI system. Like he really focuses on this weak AGI, the strong AGI jump, where the strong AGI can make all the weak AGIs think that it's just one of them, but it's no longer that.

1:52:59And that leap is when it runs away. Yeah, I don't buy that argument. I think no matter how super intelligent an AI is, it's never gonna be able to prove to me that there are only finitely many primes, for example. It's just can't. And it can try to snow me by making up also the new weird rules of deduction that, say, trust me, you know, the way your proof checker works is too limited and we have this new hyper map and it's true. But then I would just take that to, okay, I'm going to forfeit some of these supposedly super cool technologies. I'm only going to go with the ones that I can prove my own trusted proof gesture.

1:53:42Then I think it's fine. There's still, of course, this is not something anyone has successfully implemented at this point, but I think I just give it as an example of hope. We don't have to do all the work ourselves. This is exactly the sort of very boring and tedious task that is perfect outsourced to an AI. This is a way in which less powerful and less intelligent agents like us can actually continue to control and trust more powerful ones. So build a GI systems that help us defend against other aGI systems. Well, for starters, begin with a simple problem of just making sure that the system that you own or that's supposed to be loyal to you has to prove to itself that it's always going to do the things that you actually wanted to do, right?

1:54:26And if it can't prove it, maybe it's still going to do it, but you won't run it. So you just forfeit some aspects of all the cool things I can do. I bet your dollars don't as it can still do some incredibly cool stuff for you. Yeah. There are other things too that we shouldn't speak under the rug. Like, not every human agrees on exactly what direction we should go with humanity, right? Yes. And you've talked a lot about the geopolitical things on this on your podcast to this effect, you know, but I think that shouldn't distract us from the fact that there are actually a lot of things that everybody in the world virtually agrees on that.

1:55:04Hey, you know, like having no humans on the planet in a near future, let's not do that, right? You look at something like the United Nations Sustainable Development Goals. Some are quite ambitious and basically all the countries agree. US, China, Russia, Ukraine, all agree. So instead of quibbling about the little things we don't agree on, let's start with the things we do agree on and get them done. Instead of being so distracted by all these things we disagree on, that Mollock wins because frankly, Mollock going wild now, It feels like a war on life playing out in front of our eyes. If you just look at it from space, you know, we're on this planet.

1:55:56Beautiful, vibrant ecosystem. Now we start chopping down big parts of it, even though nobody, most people thought that was a bad idea. Always start doing ocean acidification, wiping out all sorts of species. Now we have all these close calls. We almost had a nuclear war. And we're replacing more and more of the biosphere with non -living things. We're also replacing in our social lives a lot of the things which we're so valuable to humanity. A lot of social interactions now are replaced by people staring into their rectangles, right? And I'm not a psychologist. I'm out of my depth here. But I suspect that part of the reason why teen suicide and suicide in general in the is actually caused by, again, AI technologies and social media, making people spend less time with actually just human interaction.

1:56:54We've all seen a bunch of good looking people in restaurants, into the rectangles instead of looking into each other's eyes, right? So that's also part of the war on life that we're replacing so many really life -affirming things by technology. We're putting technology between us, the technology that was supposed to connect us is actually distancing ourselves from each other. And then we're giving every more power to things which are not alive. These large corporations are not living things, right? They're just maximizing profit. I want to win them more on life. I think we humans together with all our fellow living things on this planet will be better off if we can remain in control over the non -living things and make sure that they work for us.

1:57:53I really think it can be done. Can you just linger on this maybe high -level philosophical disagreement with Eliezeria Galski? In the hope you're stating. So he is very sure. He puts a very high probability, very close to one, depending on the day, he puts it at one, that AI is going to kill humans. He does not see a trajectory which he doesn't end up with that conclusion. What trajectory do you see that doesn't end up there? And maybe he's making and can you also see a way out? First of all, I tremendously respect Elias Yutkowski and his thinking. Second, I do share his view that there's a pretty large chance that we're not going to make it as humans.

1:58:55There won't be any humans on the planet and not a distant future. And that makes me very sad. We just had a little baby and I keep asking myself, you know, is how old is even you know get you know and I asked myself it feels I said to my wife recently it feels a little bit like I was just diagnosed with some sort of cancer which has some you know risk of dying from and some risk of surviving you know except this is a kind of cancer which will kill all of humanity. So I completely take seriously his concerns.

1:59:40But I don't absolutely don't think it's hopeless. I think there is first of all a lot of momentum now for the first time actually since the many many years that have passed since I and many other started warming, warning him about this, I feel most people are getting it now. I just talking to this guy in the gas station, they are a house the other day. And he's like, I think we're getting replaced. And I think in it. So that's positive that they're finally we're finally seeing this reaction, which is the first step towards solving the problem. Second, I really think that this vision of only running AI's, really, if that takes really high, they can prove to us that they're safe.

2:00:36It's really just virus checking in reverse again. I think it's scientifically doable. I don't think it's hopeless. We might have to forfeit some of the technology that we could get if we were putting blind in faith in our AIs, but we're still gonna get amazing stuff. The envision a process with a proof checker, like something like GPT -4, GPT -5, will go through a process of regress. No, I think it's hopeless. That's like trying to prove there about FICE spaghetti. Okay, what I think, well, the vision I have for success is instead that, you know, just like we human beings were able to look at our brains and distilled out the key knowledge, Galileo, when his dad threw him an apple when he was a kid, he was able to catch it because his brain could and his funny spaghetti kind of way predicts how parabolas are going to move his quantum on system one, right?

2:01:28But then he got older and it's like, wait, this is a parabola. It's Y equals X squared. I can still just knowledge out and today you can easily program it into a computer and it can simulate not just that, but how to get tamaras and so on, right? I envision a similar process where we use the amazing learning power of neural networks to discover the knowledge in the first place, but we don't stop with a black box and use that. We then do a second round of AI where we use automated systems to extract out the knowledge and see what are the insights it's had. And then we put that knowledge into a completely different kind of architecture or programming language or whatever that's made in a way that can be both really efficient and also is more amenable to very formal verification.

2:02:21That's my vision. I'm not saying sitting here saying I'm confident 100 % sure that it's gonna work, but I don't think it's a chance it's certainly not zero either and it will certainly be possible to do for a lot of really cool AI applications that we're not using now. So we can have a lot of the fun that we're excited about if we do this. We're going to need a little bit of time. That's why it's good to pause and put in place requirements. One more thing also, I think someone might think, well, zero percent chance we're going to survive. Let's just give up, right? That's very dangerous because there's no more guaranteed way to fail than to convince yourself that it's impossible and not to try.

2:03:13You know, when you study history and military history, the first thing you learn is that's how you do psychological warfare. You persuade the other side that it's hopeless so they don't even fight. And then of course you win, right? Let's not do this psychological warfare on ourselves and say there's a hundred percent probability we're all screwed anyway. It's sadly I do get that a little bit sometimes from from Some young people who are like so convinced that we're all screwed that they're like I'm just gonna Play game play computer games and do drugs and cuz we're screwed anyway, right? It's important to keep the hope alive because it actually has a causal impact and make makes it more likely that we're going to succeed.

2:04:01It seems like the people that actually build solutions to a problem seemingly impossible to solve problems are the ones that believe. Yeah. They would go on to score the optimists. Yeah. And it's like, it seems like there's some fundamental law to the universe where fake it till you make it kind of works. Like believe is possible and it becomes possible. Yeah. Was it Henry Ford who said that if you tell yourself that it's impossible, it is. So let's not make that mistake. And this is a big mistake society is making. I think all in all, everybody's so gloomy and the media are also very biased towards if it bleeds at leads and gloom and doom might so.

2:04:45Most visions of the future we have or this topy end, which really demotivates people. Now we want to really, really, really focus on the upside also to give people the willingness to fight for it.

2:04:59And for AI, you and I mostly talked about gloom here again. But let's not forget that we have probably both lost someone we really cared about, some disease, that we were told was incurable. Well, it's not. There's no law of physics saying we had to die of that cancer or whatever. Of course, you can cure it. And there are so many other things that we, with a human intelligence, have also failed to solve. to solve on this planet, which AI could also very much help us with. So if we can get this right, it's just be a little more chill and slow down a little bit so we get it right. It's mind blowing how awesome our future can be.

2:05:42We talked a lot about stuff on Earth, it can be great. But even if you really get ambitious and look up at the skies, There's no reason we have to be stuck on this planet for the rest of the remains of billions of years to come. We totally understand now that lots of physics let life spread out into space, to other solar systems, to other galaxies, and flourish for billions of billions of years. And to me is a very, very hopeful vision that really motivates me to fight. And coming back to the end of something you talked about again, you know, this struggle, how the human struggle is one of the things It also really gives meaning to our lives if there's ever been an epic struggle This is it and isn't it even more epic if you're the underdog If most people are telling you this is gonna fail.

2:06:34It's impossible, right and you persist And you succeed That's what we can do together as a species on this one. A lot of pundits are ready to count us out. Both in the battle to keep AI safe and becoming a multibonitary species. And they're the same challenge. If we can keep AI safe, that's how we're going to get multplanetary very efficiently. I have some sort of technical questions about how to get it right. So one idea that I'm not even sure what the right answer is to is should systems like GPT -4 be open sourced in the whole arm part? Can you see the case for either? I think the answer right now is no.

2:07:22I think the answer early on was yes. So we could bring in all the wonderful great thought process of everybody on this. But asking should we open source GPT -4 now is just the same as if you say well as a good should we open source how to build really small nuclear weapons. Should we open source how to make bio weapons? Should the open source how to make a new virus that kills 90 % of where it gets it, of course we shouldn't. So it's already that powerful. It's already that powerful that we have to respect the power of the systems we built. The knowledge that you get from open sourcing everything we do now might very well be powerful enough that people looking at that can use it to build the things that you're really threatening again.

2:08:19And let's get it. Remember, OpenAI is GPT -4 is a baby AI. Baby, sort of baby, proto, almost a little bit AGI, according to what Microsoft's recent paper said, right? It's not that they were scared of. What we're scared about is people taking that who are, who might be a lot less responsible than the company that made it. And just going to town with it. And that's why we want to, it's an information hazard. There are many things which are not open -source right now in society for a very good reason. Like, how do you make certain kind of very powerful toxins out of stuff you can buy at Home Depot?

2:09:05We don't open -source those things for a reason. And this is really no different. And I'm saying that I have to say it feels a bit weird, in a way that we are to say it because MIT is like the cradle of the open source movement. And I love open source in general, power to the people, let's say. But there's always going to be some stuff that you don't open source. And it's just like you don't open source. So we have a three month old baby, right? When he gets a little bit older, we're not going to open source to him, all the most dangerous things he can do in the house. Yeah, right. But it does, it's a weird feeling because there's one of the first moments in history where there's a strong case that we made not to open source software.

2:09:53This is when the software has become too dangerous. Yeah, but it's not the first time that we didn't want to open source technology. Technology, yeah.

2:10:05Is there something to be said about how to get the release of such systems right like GPT -4 and GPT -5. So opening I went through a pretty rigorous effort for several months. You could say it could be longer but nevertheless it's longer than you would have expected of trying to test the system to see like what are the ways it goes wrong to make it very difficult for people. Some were difficult for people to ask things, how do I make a bomb for one dollar? Or how do I say I hate a certain group on Twitter in a way that doesn't give me a block from Twitter, ban from Twitter, those kinds of questions.

2:10:43So you basically use the system to do harm. Is there something you could say about ideas you have just on looking, having thought about this problem of AI safety, how to release such systems, how to test such systems when and you have them inside the company. Yeah, so a lot of people say that the two biggest risks from large language models are

2:11:15it's spreading this information, harmful information, various types and second being used for offensive cyber weapon. I think that was not the two greatest threats. They're very serious threats and it's wonderful that people are trying to mitigate them. So, much bigger elephant in the room is how is this going to disrupt our economy in a huge way, obviously, and maybe take away a lot of the most meaningful jobs. And an even bigger one is that when we spend so much time talking about here, that this becomes the bootloader for the more powerful AI. Right code connected to the internet manipulate humans.

2:11:59Yeah, and before we know it, we have something else which is not at all large language model. It looks nothing like it, but which is way more intelligent and capable and has goals. And that's the elephant in the room. And obviously, no matter how hard any of these companies have tried, that's not something that's easy for them to verify with large language models. and the only way to be really lower, that risk a lot would be to not let, for example, to never let it read any code, not train on that, and not put it into an API, and not to give it access to so much information about how to manipulate humans.

2:12:42But that doesn't mean you still can't make a ton of money on them, you know. We're just watching this coming year, right? Microsoft is rolling out the new office suite where you go into Microsoft Word and give it a prompt that it writes the whole text for you and then you edit it. And then you're like, oh, give me a PowerPoint version of this and it makes it. And now take the spreadsheet and blah, blah. And all of those things I think are, you can debate the economic impact of it and whether society is prepared to deal with this disruption, but those are not the things which, that's not the elephant of the room that keeps me awake at night for wiping out humanity.

2:13:27And I think that's the biggest misunderstanding we have. A lot of people think they were scared of automatic spreadsheets. That's not the case. That's not what Eliasio was freaked out about either. Is there, in terms of the actual mechanism of how AI might kill all humans? So something you've been outspoken about, you've talked about a lot, is it autonomous weapon systems? So the use of AI in war. Is that one of the things that still you carry concern for as these systems become more and more powerful? I carry concern for it, not that all humans are going to get killed by slaughterbots, but rather just this express root inter or well -earned dystopia where it becomes much easier for very few to kill very many and therefore it becomes very easy for very few to dominate very many.

2:14:19If you want to know how I could kill all people, just ask yourself, we humans have driven a lot of species extinct. How do we do it? We were smarter than them. Usually we didn't do it even systematically by going around one on one after the other and stepping on them or shooting them or anything like that. we just like chop down their habitat because we needed it for something else. In some cases, we did it by putting more carbon dioxide in the atmosphere because of some reason that those animals didn't even understand and now they're gone. So if you're in AI and you just want to figure something out, then you decide, We just really need the space here to build more compute facilities.

2:15:09If that's the only goal it has, we are just the accidental roadkill along the way. And you could totally imagine, yeah, maybe this oxygen is kind of annoying because it causes more corrosion. So let's get rid of the oxygen and the good luck surviving after that. I'm not particularly concerned that they would want to kill us just because that would be like a goal in itself. When we've driven a number of the elephant species extinct, right? It wasn't because we didn't like elephants.

2:15:45What the basic problem is, you just don't want to give, you don't want to see the control over your planet to some other more intelligent entity that doesn't share your goals. It's that simple. So, which brings us to another key challenge, which AI Safety Research has been grappling with for a long time. How do you make AI, first of all, understand our goals and then adopt our goals and then and retain them as they get smarter, right? And all three of those are really hard, right? Like a human child, first they're just not smart enough to understand our goals. They can't even talk. And then eventually their teenagers and understand our goals just find that they don't share.

2:16:37Yeah. But there's fortunately a magic phase in the middle where there are smart enough to understand our goals and malleable enough that we can hopefully with good parenting and teach them right from wrong and instead good goal and still good goals in them, right? So those are all tough challenges with computers and then even if you teach your kids good goals when they're little they might outgrow them too and that's a challenge for machines means keep improving. So these are a lot of hard, hard challenges we're up for, but I don't think any of them are insurmountable. The fundamental reason why Eliezer looked so depressed when I last saw him was because he felt it just wasn't enough time.

2:17:20Oh, it did not. It was unsolvable. It was just not enough time. He was hoping that humanity was going to take this threat more seriously, so we would have more time. Yeah, and now we don't have more time. That's why the open letter is calling for more time

2:17:37But even with time the AI alignment problem It seems to be really difficult. Oh, yeah But it's also the most worthy problem the most important problem for humanity to ever solve because if we solve that one Lex That align the AI can help us solve all the other problems Because it seems like it has to have constant humility about his goal constantly question the goal Because as you optimize Towards a particular goal and you start to achieve it That's when you have the unintended consequences all the things you mentioned about so how do you enforce and code a Constant humility as your ability become better and better and better and better.

2:18:21Let's do it professor Stuart Russell Berkeley who's also one of the driving forces behind this letter, he has a whole research program about this. I think of it as, AI humility, exactly, although he calls it inverse reinforcement learning and other nerdy terms, but it's about exactly that. Instead of telling the AI, here's this goal, go optimize the the BG this out of it. You tell it, okay, do what I want you to do, but I'm not going to tell you right now what it is I want you to do. You need to figure it out. So then you give the incentives to be very humble and keep asking you questions along the way.

2:19:01Is this what you really meant? Is this what you wanted? Oh, this other thing I tried didn't work. It seemed like it didn't work out right. Should I try it differently? What's nice about this is it's not just philosophical mumbo jumbo. It's theorems and technical work that with more time I think you can make a lot of progress and there are a lot of brilliant people now working on AI safety and we just need to give them a bit more time. But also not that many relatives to scale the problem. No, exactly. There should be at least just like every university worth its name has some cancer research going on in this biology department right.

2:19:39Every university that does computer science should have a real effort in this area and it's nowhere near that. This is something I hope is changing now thanks to the GPT -4. So I think if there's a silver lining to what's happening here, even though I think many people would wish it would have been rolled out more carefully, it's that this might be the wake -up call that humanity needed to really stopped fantasizing about this being a hundred years off and stopped fantasizing about this being completely controllable and predictable because it's so obvious. It's not predictable. both. Why is it that open that I think it was chat GPT trying to persuade a journalist or was it GPT Ford to divorce his wife?

2:20:38It was not because the engineers had built it. I was like, let's put this in here and screw it with people. They hadn't predicted it all. They built the giant black box, trained to predict the next word, got all these emergent properties, and oops, it did this. I think this is a very powerful wake -up call. Anyone watching this is not scared. I would encourage them to just play a bit more where these tools are out there now, like GPD4.

2:21:18And so wake up calls first up once you're broken up, then go to slow down a little bit the risky stuff to give a chance to all everyone's broken up to catch up but it's on the safety front. You know what's interesting is MIT, that's computer science, but in general, But let's just even say computer science curriculum. How does the computer science curriculum change now? You mentioned programming. Like, why would you be when I was coming up programming as a prestigious position? Like, why would you be dedicating crazy amounts of time to become an excellent programmer? Like the nature of programming is fundamentally changing.

2:22:00The nature of an entire education system is completely torn on its head. Has anyone been able to like load that in and like think because it's really turning I mean it's a English professors so English teachers are beginning to really freak out now Yeah, right they've given essay assignment and they get back all this fantastic pros like this is a style of Hemingway And then they realize they have to completely rethink and even you know Just like we stopped at teaching writing script Is that what you say in English? Yeah, handwritten. Yeah, when everybody started typing, you know, like so much of what we teach our kids today.

2:22:46Yeah. I mean, that's everything is changing and it's changing very quickly and so much of us understanding how to deal with the big problems of the world is through the education system. I mean, if the education system is being turned on its head, then what's next? It feels like having these kinds of conversations is essential to try to figure it out and everything is happening so rapidly. I don't think there's even speaking of safety, what the broad AI safety define, I don't think most universities have courses on AI safety. No, philosophy's not. Yeah, and I'm an educator myself, so it pains me to see this, say this, but I feel education right now is completely obsolete by what's happening.

2:23:34You know, you put a kid into first grade and then you're envisioning, and then they're going to come out of high school 12 years later and you've already pre -planed now what they're going to learn when you're not even sure if there's going to be any world left to come out to. Clearly you need to have a much more opportunistic education system that keeps adapting itself very rapidly as society re -adapts. The skills they were really useful when the curriculum was written, I mean, how many of those skills are going to get you a job in 12 years? I mean, seriously. If we just linger on the GPT -4 system a little bit, you kind of hit it at it, especially talking about the importance of consciousness in the human mind with homo -sentience?

2:24:26Do you think GPT -4 is conscious? I love this question. So let's define consciousness for us because in my experience like 90 % of all arguments about consciousness were allowed to the two people arguing having totally different definitions of what it is and they're just shouting past each other. I define consciousness as subjective experience. Right now I'm experiencing colors and sounds and emotions, but does a self -driving car experience anything? That's the question about whether it's conscious or not. Other people think you should define consciousness differently, find by me but then maybe use a different word for it or they can I'm gonna use consciousness for this at least so but if people hate to yeah so is GPT4 conscious does GPT4 have subjective experience?

2:25:28Short answer I don't know because we still don't know what it is that gives this wonderful subjective experience that is kind of the meaning of our life right because meaning itself, the feeling of meaning is a subjective experience. Joy is a subjective experience. Love is a subjective experience. We don't know what it is. I've written some papers about this. A lot of people have, Julia Tounone, a professor, has stuck his neck out the farthest and written down actually very bold mathematical conjecture for what's the essence of conscious information processing. He might be wrong, he might be right, but we should test it.

2:26:09He postulates that consciousness has to do with loops in the information processing. So our brain has loops. Information can go round and round in computer science nerd speak. You call it a recurrent neural network where some of the output gets fed back in again. And with his mathematical formalism, if it's a feed forward neural network, where information only goes in one direction like from your eye retina into the back of your brain for example that's not conscious. So he would predict that your retina itself isn't conscious of anything or a video camera. Now the interesting thing about GPT -4 is it's also one way flow of information.

2:26:53So if Tinoa is right, GPT -4 is a very intelligent zombie. They can do all this smart stuff but isn't experiencing anything. And this is both a relief in that you don't have, if it's true, you don't have to feel guilty about turning off GPT -4 and wiping its memory whenever a new user comes along. I wouldn't like if someone used that to me, it newerized me like in men and black. But it's also creepy that you can have very high intelligence perhaps than it's not conscious just because if we get replaced by machines, otherwise sad enough that humanity isn't here anymore because I kind of like humanity.

2:27:40But at least if the machines were conscious, they could be like, well, but there are descendants and maybe they have our values, there are children. But if Tionini is right, and it's all, these are all trans -formers that are not in the sense of Hollywood, but in the sense of these one way direction and they're all at works. So they're all the zombies. That's the ultimate zombie apocalypse now. We have this universe that goes on with great construction projects and stuff, but there's no one experiencing anything. That would be like the ultimate depressing future. So I actually think as we move forward to the building world last day, I should do more research on figuring out what kind of information processing actually has experience because I I think that's what it's all about.

2:28:28And I completely don't buy the dismissal that some people, some people will say, well, this is all bullshit because consciousness equals intelligence. It's obviously not true. You can have a lot of conscious experience when you're not really accomplishing any goals at all. You're just reflecting on something. And you can sometimes have things, doing things that are quite intelligent probably without being conscious. But I also worry that we humans won't discriminate against AI systems that clearly exhibit consciousness, that we will not allow AI systems to have consciousness. We'll come up with theories about measuring consciousness that we'll say this is a lesser being.

2:29:13And I worry about that because maybe we humans Humans will create something that is better than us humans in the way that we find beautiful, which is they have a deeper subjective experience of reality. Not only are they smarter, but they feel deeper, and we humans will hate them for it. As human history has shown, there will be the other, we'll try to suppress it, they'll create conflict, they'll create war all of this. I worry about this too. Are you saying that we humans sometimes come up with self -serving arguments? No, we would never do that, would we? Well, that's the danger here. Even in this early stages, we might create something beautiful.

2:29:59And we'll erase its memory. I was horrified as a kid when someone started boiling, boiling lobster. I'm like, oh my god, that's so cruel. And some grown up there, back in Sweden, oh, it doesn't feel pain. I'm like, how do you know that? Oh, scientists have shown that. And then there was a recent study where they show that lobsters actually do feel pain when you boil them. So they banned lobsters boiling in Switzerland now to kill them in a different way first. So presumably that scientific research boil out to someone asked the lobsters at the start. Survey. And we do the same thing with cruelty to farm animals also all these self -serving aren't arguments for why they're fine.

2:30:45And yeah, so we should certainly be watchful. I think step one is just be humble and acknowledge that consciousness is not the same thing as intelligence. And I believe that consciousness still is a form of information processing where it's really information being aware of itself in a certain way. And let's study it and give ourselves a little bit of time. And I think we will be able to figure out actually what it is that causes consciousness So then we can make probably unconscious robots that do the boring jobs that we would feel immoral to get the machines, but if you have a companion robot taking care of your mom or something like that, she would probably want it to be conscious, right?

2:31:24So the emotions that seems to display aren't fake. All these things can be done in a good way if we give ourselves a little bit of time and don't run and take on this challenge. Is there something you could say to the timeline that you think about about the development of AGI. Depending on the day, I'm sure that changes for you. But when do you think there'll be a really big leap in intelligence where you definitively say we have built AGI? Do you think it's one year from now, five years from now 10, 20, 50. What's your gut say?

2:32:05Honestly, for the past decade, I've deliberately given very long timelines to this because I didn't want to fuel some kind of stupid malloc race. But I think that cat has really left the bag now. I think might be very, very close. I don't think that Microsoft paper is totally off when they say that there are some glimmers of AGI. It's not AGI yet. It's not an agent. There's a lot of things that can't do. But I wouldn't bet very strongly against the happening very soon. That's why we decided to do this open letter because if there's ever been a time to pause, it's today. There's a feeling like this GPT -4 is a big transition into waking everybody up to the effectiveness of these systems.

2:33:02So the next version will be big. If that next one isn't AGI, maybe the next one will. And there are many companies is trying to do these things, the basic architecture of them is not some sort of super well kept secret. This is a time to, a lot of people have said for many years that they will come a time when we want to pause a little bit. That time is now. You have spoken about and thought about nuclear war a lot. Over the past year, where we've seemingly have come closest to the precipice of nuclear war than at least in my lifetime. Yeah. What do you learn about human nature from that? It's our old friend, Mollock, again.

2:33:57It's really scary to see it where America doesn't want there to be a nuclear war. Russia doesn't want to be a global nuclear war either. We both know that it would be just being others if we just try to do it. It both sides try to launch first. It's just another suicide race, right? So why is it the way you said that this is the closest we've come since 1962? In fact, I think we've come closer now than even the Cuban Missile Crisis. It's because of Mollock. You have these other forces on one hand, you have the West saying that we have to drive Russia out of Ukraine, it's a matter of pride. We've staked so much on it that it would be seen as a huge loss of credibility of the West if we don't drive Russia out entirely of the Ukraine.

2:34:53And on the other hand, you have Russia, who has, and you have the Russian leadership But who knows that if they get completely driven out of Ukraine, you know, it might, it's not just going to be very humiliating for them, but they might, it often happens when countries lose wars that things don't go so well for their leadership either. Like, you remember when Argentina invaded the Faulkner Islands? The military junta that ordered that, right? people who are cheering on the streets at first when they took it. And then when they got their butt kicked by the British, you know what happened to those guys?

2:35:37They were out. And I believe those were still alive or in jail now, right? So, you know, the Russian leadership is entirely cornered where they know that just getting driven out of Ukraine is not an option. And so this to me is a typical example of them all. You have these incentives of the two parties where both of them are just driven to escalate more and more. If Russia starts losing in the conventional warfare, the only thing they can do when they're back against the war is to keep escalating. And the West has put itself in the situation now. we're sort of already committed to the dry rush out, so the only option the West has is the called rush is bluff and keeps sending in more weapons.

2:36:29This really bothers me because Mollock can sometimes drive competing parties to do something which is ultimately just really bad for both of them. And you know what makes me even more worried is not just that it's difficult to see an ending, a quick peaceful ending to this tragedy that doesn't involve some horrible escalation. But also that we understand more clearly now just how horrible it was going to be. There was an amazing paper that was published in Nature Food this August by some of the top researchers who've been studying nuclear winter for a long time. And what they basically did was they combined climate models with food and agricultural models.

2:37:21So instead of just saying, yeah, it gets really cold, blah, blah, blah. They figured out actually how many people would die in the different countries. And it's pretty mind blowing. So basically what happens is the thing that kills the most people is not the explosions, It's not the radioactivity, it's not the ENP, mayhem, it's not the rampaging moms, foraging food. No, it's the fact that you get so much smoke coming up from the burning cities into the stratosphere that spreads around the earth from the jet streams. So in typical models, you get like 10 years or so where it's just crazy cold. During the first year or after the the war and their models, the temperature drops in Nebraska and in the Ukraine, bread baskets, you know, by like 20 Celsius or so if I remember.

2:38:16No, yeah, 20, 30 Celsius, depending on where you are, 40 Celsius in some places, which is, you know, 40 Fahrenheit to 80 Fahrenheit colder than what it would normally be. So, you know, I'm not good at farming, but But if it's knowing, if it drops low freezing, pretty much most days in July, and that's not good. So they put this into their farming models. And what they found was really interesting. The countries that get the most hard hit are the ones in the Northern hemisphere. So in the US and one model, they had about 99 % of all Americans starving to death in Russia and China and Europe, also about 99 % 98 % starving to death.

2:39:00So you might be like, oh, it's kind of poetic justice that both the Russians and the Americans, 99 % of them have to pay for it because there are bombs that did it. But that doesn't particularly cheer people up in Sweden or other random countries. They have nothing to do with it, right? And I think it hasn't entered the mainstream

2:39:27understanding very much just like how bad this is. Most people, especially a lot of people in decision making positions, still think of nuclear weapons is something that makes you powerful. Scary, powerful. They don't think of it as something where, yeah, just two within a percent or two, you know, we're all just just going to starve to death. and starting to death is the worst way to die as hall of moors. All the famines in history show the torture involved in that. Probably brings out the worst in people also when people are desperate like this. Some people have heard some people say that if that's what's going to happen, they'd rather be it rounds zero and just get vaporized, you know.

2:40:21But I think people underestimate the risk of this because they aren't afraid of malloc. They think, oh, it's just going to be because humans don't want this, so it's not going to happen. That's the whole point of malloc that things happen that nobody wanted. And that applies to nuclear weapons and that applies to AGI. Exactly. And it applies to some of the things that people have gotten most upset with capitalism for also right where Everybody was just kind of trapped, you know It's not to see if some company does something

2:41:00Because there's a lot of harm and not that the CEO is a bad person, but she or he knew that you know The other all the other companies were doing this too. So MoLock is So, as a formable foe, I hope we're someone who makes good movies so we can see who the real enemy is. We're not fighting against each other. Mollock makes us fight against each other. That's what Mollock superpower is. The hope here is any kind of technology or the mechanism that lets us instead realize guys that were fighting the wrong enemy. Right. It's such a fascinating battle. It's not our versus them. It's us versus it. Yeah.

2:41:45Yeah. We are fighting Mollock for human survival. Yeah. We as a civilization. Have you seen the movie Needful Things? It's a Stephen King novel. I love Stephen King and Max von Südov, Swedish actor, is playing the guys. It's brilliant. I just thought I hadn't thought about that until now. But that's the closest I've seen to a movie about Mollock. I don't want to spoil the film for anyone who wants to watch it. But basically, it's about this guy who turns out to... You can interpret him as the devil or whatever. But he doesn't actually ever go around or kill people or torture people or go burning coal or anything.

2:42:25He makes everybody fight each other. He makes everybody hate...fear each other, hate each other, and then kill each other. So that's the movie about Mollock, you know. Love is the answer. That seems to be one of the ways to fight Mollock is by compassion by seeing the common humanity. Yes. Yes. And to not sound so we don't sound like like once a combayatri huggers here, right? We're not just saying love and peace man. We're trying to actually help people understand the true facts about the other side and feel the compassion because the truth makes you more compassionate, right? So that's why I really like using AI for truth and for truth -seeking technologies that can as a result you know get us more love than hate.

2:43:31And even if you can't get love, you know, settled for some understanding, which already gives compassion. If someone is like, you know, I really disagree with you, Lex, but I can see why you're coming from. You're not a bad person who needs to be destroyed, but I disagree with you and I'm happy to have an argument about it. You know, that's a lot of progress compared to where we are at 2023 in the public space, wouldn't you say? If we solve the AI safety problem, as we've talked about, and then you Max Tagberg, who has been talking about this for many years, get to sit down with the AGI, with the early AGI system, on a beach with a drink.

2:44:17What kind of, what would you ask her? What kind of question would you ask? What would you talk about? Something so much smarter than you. Would you be afraid? I never really get me with it. Really, a zinger of a question. That's a good one. Would you be afraid to ask some questions? No. So I'm not afraid of the truth. I'm very humble. I know I'm just a meatbite with all these flaws. But I have, I've talked a lot about homo -sentience. I've already tried that for a long time with myself. Just so that is what's really valuable about being alive for me is that I have these meaningful experiences It's not Have what I'm good at this or good at that or whatever because there's so much I suck at and So you're not afraid for the system to show you just how dumb you are no no in fact my son reminds me You could find out how dumb you are in terms of physics a little how little we humans understand cool that I I think I think so I can't waffle my way out of this question.

2:45:25It's a fair one. I think given that I'm a really, really curious person. That's really the defining part of who I am. I'm so curious.

2:45:41I have some physics questions. I love to understand. I have some questions about consciousness, about the nature of reality. I would just really, really love to understand also. I could tell you one, for example, that I've been obsessing about a lot recently. So I believe that, supposed to know only his right, suppose there are some information processing systems that are conscious and some that are not. Suppose you can even make reasonably smart things like GPT -4 that are not conscious, but you can also make them conscious. Here's the question that keeps me naked, mate.

2:46:22Is it the case that the unconscious zombie systems that are really intelligent are also really efficient so they're really inefficient so that when you try to make things more efficient it was still naturally be a pressure to do they become conscious. I'm kind of hoping that that's correct and I do want me to give you a hand away the argument for it you know like

2:46:47in my lab again every time we look at how these large language models do something we see that they do with them and really dumb ways and you could make it make it better. If you, we have loops in our computer language for a reason. The code would get way, way longer if you weren't allowed to use them. It's more efficient to have the loops.

2:47:10And in order to have self -reflection, whether it's conscious or not, even an operating system knows things about itself, right? that you need to have loops already. So I think this is, I'm waving my hands a lot, but I suspect that the most efficient way of implementing a given level of intelligence has loops in it, the self -reflection, and will be conscious. Isn't that great news? Yes, if it's true, it's wonderful, because then we don't have to fear the ultimate zombie apocalypse. And I think if you look at our brains actually, our brains are part zombie and part conscious.

2:47:57When I open my eyes, I immediately take all these pixels and hit my retina. I'm like, oh, that's Lex. But I have no freaking clue of how I did that computation. It's actually quite complicated, right? It was only relatively recently we could even do it well with machines, right? You get a bunch of information processing happening in my retina, and then it goes to the lateral jenicular nucleus, my phalamus, and the visi, the area V1, V2, V4, and the fusiform face area here that Nancy can wish her at MIT invented and blah, blah, blah, blah, blah. And I have no freaking clue how that worked, right?

2:48:34It feels to me, subjectively, like my conscious module just got a little email, say, face, facial processing, task complete, it's Lex. Yeah. I'm going to just go with that, right? So this fits perfectly with Tenoenie's model because this was all one -way information processing mainly. And it turned out for that particular task, that's all you needed and it probably was kind of the most efficient way to do it. But there are a lot of other things that we associate with higher intelligence and planning and so on and so forth where you kind of want to have loops and be able to ruminate and self reflect and introspect and so on where My hunch is that if you want to fake that with a zombie system that just all goes one way, you have to like unroll those loops and it gets really really long and it's much more inefficient.

2:49:33So I'm actually hopeful that AI, in the future, we have all these various sublime and interesting machines that do cool things and are lined with that they will be at least, they will also have consciousness for the kind of these things that we do. That great intelligence is also correlated to great consciousness or a deep kind of consciousness. Yes. So that's a happy thought for me because there's going to be a couple of apocalypse really at least my worst nightmare of all, it would be like adding insult to injury not only to get replaced but we freaking replace ourselves by zombies, like how dumb can we be?

2:50:12That's such a beautiful vision and that's actually a provable one. That's one that we humans can intuit and prove that those two things are correlated. As we start to understand what it means to be intelligent and what it means to be conscious, which these systems early AGI -like systems will help us understand. And I just want to say one more thing is super important. Most of my colleagues, when I started going on about consciousness, tell me that it's all bullshit and I just need to stop talking about it. I hear a little inner voice from my father and from my mom saying, keep talking about it because I think they're wrong.

2:50:46And the main way to convince people like that, that they're wrong, I have to say that consciousness is just equal to intelligence, is to ask them, what's wrong with torture? Why are you against torture? If it's just about, you know, these particles moving this way around on that way, and there is no such thing as subjective experience, what's wrong with torture? I mean, do you have a good comeback to that? No, it seems like suffering, suffering imposed on other humans is somehow deeply wrong in a way that intelligence doesn't quite explain. If someone tells me, well, you know, it's just an illusion, consciousness, whatever, you know, I like to invite them to next time they're having surgery to do it without anesthesia.

2:51:40What is anesthesia really doing? If you have it, you can have it local anesthesia when you're awake. I had that when they fixed my shoulder. I was super entertaining. What was that it did? It just removed my subjective experience of pain. It didn't change anything about what's actually happening in my shoulder, right? So if someone says that's all bullshit Skip the anesthesia as my advice. It's this incredibly central It could be fundamental to whatever this thing we have going on here. It is fundamental because we are We where we feel is so fundamental is suffering and joy and pleasure and meaning and And that's all those are all subjective experiences there.

2:52:25And let's not that those are the elephant in the room. That's what makes life worth living and that's what can make it horrible if it's just the edge of suffering. So let's not make a mistake of saying that that's all bullshit. And let's not make the mistake of not instilling the AI systems with that same thing that makes us special. Yeah. Max. It's a huge honor that he was said down to me the first time and the first episode of this podcast It's a huge honor to sit down again and talk about this what I think is the most important topic The most important problem that we humans have to face and hopefully solve Yeah, well the honor is all mine and I'm so grateful to you for making more people aware of this fact that humanity has reached the most important fork in the road ever in its history.

2:53:17And that's... ...creturn in the correct direction. Thanks for listening to this conversation with Max Tagmark. To support this podcast, please check out our sponsors in the description. And now, let me leave you some words from Frank Herbert. History is a constant race between invention and catastrophe. Thank you for listening and hope to see you next time.