Is AI "Alien Intelligence?" Emerson Spartz on Mental Models for AI

25:41And that's not just legitimate. I think one of the challenges with the climate change conversation has been that there's sort of a never-ending increase in the drama of the statements around it in order to try to kind of fully capture people's attention. And even if those things end up all being right in retrospect, it's now been so many years of that being sort of screamed at people that I think there's been a counter response of just sort of frustration and disengagement because it seems hopeless. What I think is really interesting, and maybe this is kind of the next place to go in the conversation, is my read on where this sort of set of new people coming into this conversation is, is that one, they are not as frantic as media headlines in terms of being 100 % convinced that we're all heading to doom, right?

26:34It's not 100 million new doomers that have just been minted or anything like that. But at the same time, I think they are radically less skeptical than the people who have been in AI safety for a long time might have thought after screaming into the void for so long. I think, in fact, that there is a lot of sort of common sense around if 10 % of 50 % of the people, so 5 % of all total researchers in this space think that there's a probable chance that it ends humanity, that is a problem that is worth spending time on, worth engaging with. And as we were just saying before we hit record, the people who are coming into this conversation now aren't coming from kind of, you know, 30 years of watching it sort of bump along and go up and now to accelerate.

27:23They got blindsided by some technology that seemed like absolute magic to them. And so I don't think it's as hard for them to make the mental leap from this thing that I didn't even know existed can do things better than me to I bet it could do a lot of things better than a lot of people. And what is that going to mean? And so I guess, Let's talk maybe about sort of where we find ourselves. And maybe, let me ask it in terms of a question that I have that I don't necessarily know you have the answer to or you're supposed to have the answer to. If my suspicion is right, that far more people are receptive to this conversation, they have a sort of a base level agreement that it's something that we should discuss, and that outside of just continuing to raise awareness of the issues, it feels like they're ready for conversations about remediations, about tactics, right?

28:17I think that people would be a lot happier, not just with another Bankless podcast about how we're all doomed, but with, like, here are things that we should actually do. And unfortunately, if you watch a lot of these safety guys on Twitter, which, you know, the average population isn't going to do, it's sort of like vague ideas of like more funding for alignment research. It's like, what does the Manhattan Project actually look like on this? Even what you said a minute ago, of, I think that we should actually try to slow down, is kind of more than we're getting from some of the folks who are sort of out advocating.

28:54And I understand why. Again, you scream into the void for years and years and years. It's kind of head spinning to all of a sudden have people be listening. But now that people are, it sort of feels imperative to give them things to do or things for us to do, rather than just sort of continue to have the intellectual battle about about what might be. Anyways, I think, you know, the interesting thing, I guess, maybe is, where do we go from here? What are, you know, natural next steps, you've spent a lot of time, and you are very, you're an entrepreneur by disposition, which means I assume that you've, you know, spent time on the things to do as well, even if no one knows.

29:31But, you know, what do we do, I guess, is the question, you know, no pressure. Yeah, so I'm glad that we're finding a position to be discussing this. Okay, the reason why you haven't heard too much in terms of actual solutions is because we don't know what to do. Like we're highly uncertain about what to do. And it wasn't even until like, like, because things are changing so quickly. So for example, a bunch of members, there's only been a couple other people working on this. And a lot of the stuff that they were, that some of the people were working on in the past doesn't seem as likely to be useful now because the paradigms have shifted.

29:57Language models ended up becoming much more powerful than we thought they were going to. And so some of the work that other people were doing, maybe isn't going to be helpful later, but ultimately like the only thing I think we need to do, the things that I'm, I'm really uncomfortable about, like, Like, okay, regulatory-wise, some people are proposing we need an IAEA that regulates nonproliferation in the same way that the IAEA does for nuclear nonproliferation. We need to tightly control via compute governance, like pre-registration of large training runs. And we need to monitor compute because that's a spot in the supply chain that is easy to monitor.

30:29And then we can keep people from building larger models without proper safety protocols. And that's plausible to me that that's the right thing to do. But I'm just really highly uncertain. what I do know is that like from an attention perspective um we weren't even the community wasn't even yeah basically like it's only been for the first time ever people are starting to wake up you're like okay well now that people are are like generally it seems like the majority of the population is is like on board now with uh safety you know ai safety in general so what do we do about it there's still a lot of people who aren't and so I still think it's important to like continue to um like it was that that uh that extra statement like people weren't even talking about extinction in a meaningful sense until a couple weeks ago and that was something that was like, for me, very frustrating because I'm worried about misuse risks.

31:10I'm worried about bad actors. I'm worried about disinformation. And I'm worried about there's, there's like many things that can go wrong, but like, it was only a couple of weeks ago that we even got extinction, the word extinction to be said by, you know, world leaders. There was a remarkable moment. I'm sure, I mean, I've talked about it on the show, but when Senator Richard Blumenthal was asking Sam Altman, he said, you know, Sam, you've said that he basically said, and I'm paraphrasing, obviously, Sam, you've said that if things go wrong, it could be the worst thing that ever happened and we could all die.

31:42And I'm guessing you mean jobs replacement, because that feels like death to me. It's like it actually took Gary Marcus being like, wait a second. I don't think that's what Sam was talking about when that was his worst case scenario. It was a really spectacularly weird moment that showed there was a palpable sense in that committee room that even if they had read about it and grokked it, the U.S. senators were not comfortable yet publicly declaring the possibility of any technology ending humanity. You know, it was it was the laugh, not even the laughter stage of the White House press conference, but just a sidestepping in a way that no one was going to use a word like extinction on the public record, basically.

32:31Let's put it that way. Right. And I think that was really, really, really important because the solutions that you advocate for are very different. If you're worried about human extinction versus if you're worried about people saying naughty words, because a lot of people think of a safety as like, oh, the naughty words, anti fun brigade, the one that makes me have to jailbreak, you know, chat GPT whenever I wanted to give you medical advice or like do anything that's like a little spicy. And so so step one was like, OK, we need to get people to actually understand that, like, we're actually risking human extinction here.

32:59Now that we, like, that is starting to work. We've had a few weeks now of, like, yesterday, again, UN Secretary General and, like, 42 % of CEOs. Like, that's great. It makes me feel much more like we could solve this. So the question is, like, okay, so at what point do you switch from doing – I think we need to have, like, a team effort where we do lots of different things. The main thing that I think we need to do is we need to have way more smart people that are thinking about this. Again, we've only had a few hundred people, and they're, like, pretty correlated nerds with, like, very similar worldviews.

33:27And we need all of the beauty of humanity's flourishing diversity of perspectives to weigh in here on what do we do about this. We're making this technology, and it could be the best thing we ever invent, or it could be the worst thing we ever invent. So solution-wise, just making sure that we're getting a lot more money going into alignment, that the brightest minds in policy are thinking about how do we keep this safe. One thing, for example, that I've been thinking about is open source. I'm very worried about open source and open source AGI. And I come from, as somebody who's like a full-throated open source, like flag waving in almost any other context up to this point, like you would have had me, you would have had to catch me dead to be like saying open source is dangerous.

34:09I have background in Web3 and anyway, I'm just like a big believer in decentralization and the power of open source. And one of my biggest fears right now is that open source makes it such that, like open sourcing AGI is just like a terrible idea. It's like in a very real sense, the analogy is kind of stressed, but it's kind of like giving everyone nukes. And like people say things like, oh, well, you know, it's like mutually shared destruction. And like if everybody has nukes, then like, you know, like people won't be able to fire them at each other because it'll be safe. And I think like mutually shared destruction definitely worked in the Cold War.

34:39There was a lot of close calls, though. I mean, dozens of like very close calls. And that was just with, you know, nine nuclear powers. But you can imagine nine versus like eight billion people pointing nukes at each other. And that's just really dangerous. and there's just lots of scenarios where like for example right now like you could create a i mean i don't want to quote any specific studies because of the info hazards of this but like it's like crazy fucking easy to make bioweapons right now like crazy easy to make bioweapons and we do not have the ability to like if some lunatic and there are a lot of lunatics out there most people don't realize this like what maybe most people do but like right now there's an average of three terrorist attacks in the world every day right now three we just don't hear about them because they're in places that the media doesn't cover like afghanistan and iraq and so on but like Like that's three a day.

35:22And right now they're mostly just like, you know, people strapping bombs to their chest and blowing themselves up because they don't have the resources or the skills to do anything that's more dangerous than that. But imagine if they did. And imagine if they could like, you know, just like ask ChatGPT to make a super smallpox that is like a thousand times deadlier and more virulent than regular smallpox, which killed, I don't know, I forgot the stat. I want to say it was like 10 % of humanity or 7 % of humanity of humans who ever lived died to smallpox or something like that. Like we don't have the ability to stop that at all right now.

35:51Like maybe, maybe we could, but it might take like 10 years of rolling out wastewater surveillance and like, you know, tens of billions of dollars of like, of technology and checks before we could actually stop something like that. So there could be this like window where it's just crazy dangerous to allow these technologies. And the problem with open source is that right now, if you try to do things like that, the open, if you try to like design a super pathogen, the open AI, you know, API will like, you know, it's not too hard for them to catch you trying to do something like that. But if it's open source and you can just run, you know, GPT-5 or GPT-6 on your laptop at home, then man, it's just going to be really, really, really hard to stop one crazy person from doing something that could like, and maybe it doesn't kill all humans.

36:31Maybe it wipes out like half of humanity and it does it in like a few months. You know, like these things are like, and there's like the surface of these things is growing exponentially. Like there's this, the Yudkowsky's Law of Mad Science is that the number of IQ points necessary to destroy the world drops by one per year. And it feels like it just dropped like 10 in the past year. Like how much damage you can do as a lunatic. This is one of the things, maybe a way to sort of, to the extent that we are trying to identify, and obviously we're not going to solve anything in this conversation as a podcast, but here we are.

37:03To the extent that we're trying to sort of put takeaways on this, one that I think is really valuable at this stage, right, as we are, call it transitioning from just pure awareness to a combination of awareness, deeper assessment of the problem, plus starting to talk about potential, you know, not solutions, but things to do. One that you're identifying that I think is correct and important is better problem or risk identification, more specific risk identification. One of the things that you see a lot is the whole conversation is circling around China because it's, you know, naturally problems come into the context that they're, you know, that they're already in.

37:40We have, you know, the only thing that Democrats and Republicans can agree on over the last couple elections is that China bad and we should be more antagonistic towards China. and Mark Andreessen used it as his big boogeyman in that YAI will save the world piece. It came up in the context of the hearing. China is clearly looming as a threat. Although just this morning, we're recording on Friday, June 16th, I read that there's been some high-level behind-the-scenes conversations between China and the US. And it is completely plausible to me, despite whatever sort of proto-Cold War II we might have with China, that coordination among state-level actors because of mutually assured destruction is radically easier than what you're talking about, which is we've forgotten because there hasn't been a loud terrorist attack for a while.

38:30One of the big trends that every social scientist and political scientist identified over the last 30 years that we've stopped talking about for some reason is the rise in importance of non-state actors, right? Well, all of a sudden, non-state actors have this incredibly powerful tool. And to your point, it's not even non-state actors who can mobilize resources like Al-Qaeda or ISIS might have been able to in the past. It's individual sort of rogue entities. And that becomes a really terrifying thing. So again, I don't want to stop the flow of thoughts and the stream of consciousness, but I think really assessing where the risk is and not just reductively being like, it's China, is I think an important part of progressing this conversation.

39:12You know, by the way, having the conversation about China and where they fit in this is important. It's just sort of, it's certainly not the only thing or should be the defining thing as it relates to the policy and the decisions that we make here. Right. And I think China is, it's a real threat and it's a real concern. And we should definitely have that conversation. I think that like, instinctively, blindly just saying like, we have to race to build these extinction boys, because otherwise like the political outgroup might get the extinction boys first. It's just naive. Like the thing is you can't win an AGI race.

39:44I mean, okay. So it's just, it's, it's like, you can imagine like evolution gave us two rules, survive and reproduce. May the fittest win. And we're sitting here and we're like, we're going to make a new species that's a thousand times fitter than us. And we're just going to try to make it so that it doesn't develop any instrumental goals like self-preservation or power seeking. So that we're basically going to hope that we can make it our slave and it'll just like do our bidding forever exactly how we want, right? It doesn't really matter which country makes it first if that is what we're, that's like the space of like things that we're talking about here.

40:20It's like, sure, there's all kinds of dystopian scenarios where like maybe the outgroup gets it first, maybe China gets it first and maybe they usher in a like, you know, hellishly Orwellian dystopian totalitarian state. And then that's bad. And that's certainly a bad feature. And we should definitely, you know, worry about that. We should, we should try to make sure that doesn't happen. But like the space of things that can go wrong and most of them are just solved by just like going slower is like the thing that I want to keep pointing back to. I also think it in China is a long conversation, but people underestimate how fearful, uh, no, that's a long China conversation, but basically like China is the CCP is like suspicious of technology because, uh, they like being in power.

40:55Exactly. Yeah, they like being in power. Yes, I agree. We don't have to get all the way into it, but there is huge countervailing force. I mean, this is a group that has exerted extraordinary control over cryptocurrencies and digital assets because they recognize the threat to power that they represent. I think that it's fair to say that it is likely that their desire to harness the power of these technologies is likely to be in some ways even more counterbalanced by their worry that other people in Chinese society could use it to upend the power balance they have. And they're sitting in the catbird seat right now.

41:37You know, it's kind of like, is the risk worth it? And I think it looks very different than we might imagine. Yeah, I agree. And I think that's what like, basically, my model is just think of AGI as the flipper of all the game boards. If you're already sitting on the throne, then AGI should be scary for you. Because yeah, I mean, you could use it to like, surveil your and control your population even better. But like the CCB is already firmly in control. So they have more to lose from a massively disruptive technology. I mean, who's usually pro disrupting, disrupting technologies, and who's anti disrupting technologies?

42:08The incumbents, yeah, right. Yeah, it's the incumbents that are usually the ones that are anti-disruptive technologies. And the CCP is very much the incumbent in that a billion and a half people are under their thumb. And so I think people are just really too quick to assume that, well, they use this model of what's all just like a Cold War arms race, whoever wins, wins. And that's just one mental model to use. The other mental model is them letting this Pandora's box out that they can't predict or control what will happen after that. And then them losing power. I mean, they've been like, they've been bludgeoning their tech industry now for the last couple of years.

42:41Like they wiped like a trillion and a half dollars off their market technology market caps by just like aggressively regulating. You know, they like grabbed Jack Ma and like, you know, as a warning shot. Yeah. I mean, they stopped the end financial IPO, which was, yeah, it was going to be the world's biggest IPO, the biggest IPO in history. It got shut down. Jack Ma disappeared for six months. They put him on TV like five months later just to make sure that people knew he wasn't dead. And when it came out of it, Ant Financial had been restructured as a not fully state-owned bank, but regulated as a state-owned bank.

43:15And we basically haven't heard from Jack Ma since then, other than occasional small appearances here and there. But I mean, that was a company that was heading towards world financial dominance, and they just noped it in a huge way. Yeah, exactly. And so I think that like, you know, and part of what sinologists, many sinologists believe is that basically China, the CCP was like worried about the tech industry gaining too much, you know, independent power and control. And so the CCP wanted to like, even if it meant like destroying a lot of wealth, it was more important to maintain control than to, it was worth the wealth destruction.

43:51And so I think that like there's, so Katia Grace, so the cover of Time Magazine, the most recent one, is about the end of humanity, AI. And there's a really good article by Katia Grace, who's brilliant, but she basically uses this analogy of like, it's not a race. We're not in a race right now. What we are is like, imagine that we're on a lake, a frozen lake, and there's like riches on the other end of the shore. We're all on this lake together, right? We're like tiptoeing across the lake. And like, if anybody like defects and like run towards the riches at the end of the lake, then um yeah maybe maybe they can get the treasure for the lake falls in but if we like go slowly together then we can get the treasure so the example here is like AGI is like the treasure on the other end of the lake but we have to like go at a speed that doesn't cause the lake to fall in I think that's like a useful way to think about where we're at right now and China like AGI is much bigger than like our local kind of monkey politics um and tribalisms because if one person the reason I worry about open source is because if one person builds an unsafe, like self-replicating AGI, it could just be lights out for everybody.

44:53And so if we can't monitor what people are doing with it, it's kind of like giving everyone a bio lab. I think that's actually like the self-replicating thing is the thing that people don't have good intuitions for. But like there's a lot of things normally open source. So there's this like offense defense balance that happens all the time in game theory. Right. And so the thing that like one thing that, that I worry about is that basically because AGI, so I mentioned that like AGI is like the flipper of all the game boards, right? So if I were a CEO, I'd be worried about, I'd probably, I mean, I'd be, I'd be excited, maybe excited, maybe worried.

45:21Depends on like my industry and so on. But like in general, if you're the incumbent, the new disruptive technology scared me, right? So AGI is like the flipper of all the game boards. But also AGI is the speeder up of all the things. So what does that mean? So like normally there's this like delicate balance of offense and defense, you know, with power. So like in war, you'll have like, there's this red queen race of like offensive weapons versus defensive weapons. And they tend to be like on average balanced out, but there's these windows where sometimes like, for example, uh, encryption has been like defensive advantage for a long time now.

45:52Um, because, uh, you know, one of the algorithms is hard to crack, um, doesn't crack it. So I think of AGI is basically just like, it takes like every one of these delicate, um, one of these delicate, you know, equilibria. And it makes it such that, like I mentioned that, that, that, super smallpox example. Again, think of like a black death. But imagine a black death that spreads, instead of spreading at the speed of human, basically the black death spread across Eurasia at basically the speed, walking speed, because that's how fast people moved back then, right? But imagine something that was similarly virulent that spread at the speed of flight, because the world's much more connected now.

46:29That is the kind of thing where like, yeah, maybe we can like set up all these systems to guard against it, but we don't have those systems in place now. COVID happened and we've barely changed anything. And so, yeah, there's just like, that's just like one of like many, many, many examples of like how it could just be attacker's advantage just gets like spikes up really high very briefly. And it leads to like cataclysmic things like that. And it's just hard to predict how these things will go out in advance, which is why I keep going back to like, you need to slow down. It changes too many things too fast for us to be able to adapt.

46:58Let me ask you a question around. So coming back to something that I sort of flagged earlier, just as it relates to slowdown. I think that when we think about slowdown, there are three paths that are plausible for that. One is industry mutually decides. You get enough of a consensus among enough people that they shift, that they say that the long-term incentive is going to beat out the short-term incentive. We're taking ownership of that incentive balance, and we're all going to shake hands, maybe with daggers in them, but we're all going to shake hands and do it. So that's one possible path. A second possible path is government saying you have to slow down.

47:40A third possible path is, I guess, maybe there being so much consumer pressure on companies that they are forced to slow down, that they are punished by the market in some way for not slowing down. Historically, I think that people would assume that second path is the most plausible, right? Because there was sort of some power balance between the private markets and the public sector. Again, you sort of have heard my worry is that there isn't. That at the time that a public sector that is trusted and strong is sort of most needed is the time that it is sort of the least capable of that. So one, I want your take on that.

48:20And then two, I wonder to what extent And there's another conversation that is going to be had around AI, which is a fundamental sort of realignment conversation around the social contract and what it means to be part of society. If, I mean, McKinsey, it's McKinsey, so take it with a grain of, you know, a giant bag full of salt. But their recent report that came out estimated that 60 % to 70 % of what the average worker spends their time on across something like 85 % of professions can be automated by AI. Now, of course, that doesn't take into account new things that people do because of AI and entirely new professions.

49:01And there's all these sort of things that might come, but whatever. We kind of recognize that this is a massively transformational force as relates to what humans spend their time on, what they get incentivized for. And by the way, it completely changes. For the first time, it's a technology that comes at white-collar jobs, not just blue-collar jobs. And, you know, all of a sudden people in, you know, Ohio are going to be really facing competition for people in India in a way that it doesn't even, you know, we can barely grok now. Anyways, the point being that it's hard to imagine that we don't have some sort of pretty massive realignment around how we think about people's worth and their getting to participate in society and how much it is or isn't dictated by their jobs.

49:44perhaps those conversations are aligned. Because we are going to have to go back to such a fundamental re-evaluation of people and their place, maybe that opens up a different type of conversation with government. But I guess maybe to try to put it in a question form that you can actually answer, where do you see governments in this? Do they have the ability to be a meaningful part of this? Is it less dire than it feels like to me? Or how do you think about the policy side? Okay. So the three things first was like, can we gentlemen's agreement get together and say, we're not going to build it.

50:16Second government forcing people to not build it or slow down. Uh, and then the third being, um, maybe like a moral backlash at the societal level. Um, I think that historically when we have slowed down non-technology, it's been through a mix of all three actually. Um, sometimes it was like a moral backlash from society, which led to a government saying, you know, you can't build this or like, you can only build it, but like, we're going to add a lot of safety precautions, which slows it down. Um, sometimes like at Selimar, it was, you know, scientists who got together and said, like, we think that these genetic experiments are too dangerous.

50:46So we're going to we're going to not we're going to agree collectively. We're going to shake hands and say we're not going to do these experiments for now. And then they did. We do tons of recombinant DNA experiments that were not considered to be OK back then because the scientific consensus changed. So those were OK. So I think it'll be some mix of all three. Which order? I don't know. I think that basically there's a lot of people that have like like you started off by talking about there's this innate skepticism. many people have when they hear about this, like, oh, another threat that I should be like, oh, the media is this giant negative news machine.

51:13And the media is telling me once again, a new thing I should be scared of. And they're exhausted by being told to be scared of things. And I think that's a perfectly understandable reaction. And I sympathize with that. And so you get these like you hear these like weird like reasons that people have for not taking air risk, you know, extra seriously. Like, for example, like, oh, sure, the AI scientists are saying that their technology might cause human extinction because that makes them look good. like, because they're working on something important. And like, if you really stop and think about that, like, it should really hit you just how not patently absurd is, how ridiculous that is.

51:46Like, it's crazy. Like every inventor ever is like, oh, I made this technology, it's going to like bring all these benefits to society. And then but it has like, it's it could be dangerous, right? And then he's like, No, no, it's not dangerous. And then people are like, you're just biased. Like, it is dangerous. I think that that set of arguments is is so patently absurd that it actually does a real disservice to the people who are like, not only was this the weakest part, I think of Andreessen's piece, like the invocation of Oppenheimer as a silly kind of figure who like shouldn't have had any concerns because, you know, this thing was so powerful that it ended like Oppenheimer as a historical figure, as someone who did what was important and essential, but also had massive, very normal human reactions and concerns about that.

52:36Using him as a bully cudgel to say, of course, you should have just known that the ends justify the means is ludicrous to me. It's just such a weak argument. The idea that this is just the Eliezers of the world wanting to be loud and be known is so patently ridiculous. There are much better arguments to have than that one, in my view. And the regulatory capture one too. A lot of people are like, oh, this is just regulatory capture. They're trying to call for regulation so that they can pull up the ladder. And it's fair to worry about that because that does happen all the time in politics. And it's certainly fair to worry about that with the heads of the AGI labs, DeepMind, OpenAI, Anthropic, et cetera.

53:17But these letters, the most recent letter, the extinction letter, that was signed mostly by university professors. They don't benefit at all from any kind of regulatory capture. So it's another, like, really, it's conflating, like, okay, there's this one possible sort of conspiratorial thing that could be true, but why they're saying their technology might cause human extinction versus, like, the majority of people who actually sign it who just very obviously don't benefit from regulatory capture. So I think that's, like, really important. I feel like people don't talk about this. There's this, like, just weird meme of, like, just way too many people believing this regulatory capture thing, which, again, fair to worry about that with, like, the actual companies themselves because that does happen.

53:50Although I don't think that's what's happening here because these guys are on the record. like for years they've been talking about these risks and so if they just suddenly like if they've been like dismissing these you know extinction risks um for years and then suddenly they found themselves like being willing to admit to them then it would be more suspicious but like yeah most of these guys are on the record as acknowledging these risks and many of these companies actually started because they were worried about safety they were worried about extinction risk and so they thought like well like i should be the one to start the agi company because then we'll be able to take agi risk seriously and then we can like you know increase humanity's probability of surviving.

54:22And also like Sam Altman doesn't have any equity in opening AI. Like my read on Sam Altman is that he's just like, he's a well-intentioned dude. And, um, I think he cares. Um, and I think all these guys care. And anyway, so, so without going too much details on that, I just want more people to be aware of that. Um, I think it's just too conspiratorial, um, relative to like who actually signed the letter and like what their actual incentives are anyway. Um, so back to the question of like, like, you know, gentlemen's agreement, like scientists agreeing to not build it versus government saying you can't build it versus moral backlash.

54:51Usually I would say in these situations, it's like it's more of the first two. And then the plan, the plan B is like, well, if the scientists can't agree and the government can't like impose, then the moral backlash creates the conditions such that that happens. So here's an example. So in the World War, the neutron bomb campaign, the Stop the Neutron Bomb campaign, the Carter administration built a neutron bomb. It was a like win war button against the Soviets. It was basically like a bomb that would only kill people and wouldn't destroy buildings. And like a thermonuclear, you know, radius explosion that was that powerful.

55:23And the Soviets were really worried about it. And they actually spent$700 million on a stop. We spent like, you know, billions, maybe tens of billions, I don't know, of dollars trying to create a neutron bomb. The Soviets couldn't build it or they wouldn't build it. It would take them a long time to build it. They spent$700 million on a campaign to discredit the neutron bomb. It was called the Stop the Neutron Bomb Campaign. And they were able to organize these massive protests in like throughout Western Europe, like in I think in Germany, I forget which city it was. They had like 50 ,000 people show up for a stop the neutron bomb rally, basically using like hippies as they're like, you know, useful idiots, essentially.

55:58Because for them, it's like, well, you spend seven hundred million dollars. And that put a lot of pressure on the Carter administration to actually like mothball the neutron bombs. Right. So let's say we spent 10 billion and they spent seven hundred million. That's like a good ROI to like, you know, get rid of the neutron bomb threat. Um, but like the point is they created a moral backlash, which put pressure on the Carter administration to like mothball, the neutron bombs. Now that was actually like a cold war plot version of it, but like the same things happen all the time that aren't like cat's paw, cold war plot kind of things.

56:25And so I think something like that is certainly possible here as well. But I think that like, usually I think of the government is like the basic society is the fourth branch of government. So we have the executive branch, legislative branch, judicial branch, and then we have culture and the culture is what like informs the other branches. And so that's part of the reason why I'm still like, I think it's important to continue to like make sure people are aware of the risks of AI because it's really easy. Like it's really easy to think that the risks are just about job losses or just about like naughty words or just about like racial bias or things like that.

56:53And like, those are risks too, but like we're going to choose very different things on what to focus on. If we're worried about it, here's an example. So here's like one of the biggest things that keeps up alignment researchers at night. We spend a lot of time worrying about like, what if we put all these safeguards in place and then there's this thing we call the sharp left turn, Which is basically like, what if there's just like this jump in capabilities that like emerges very quickly and like all these safeguards are put in place, they don't work at a certain point because of this big capabilities increase.

57:19And we see these big capabilities increases all the time at different scales right now. And so like there's a lot of scenarios where we think like, oh great, we've made this AGI and it's super safe and it doesn't say naughty words and it's not racially biased and it doesn't do disinformation. Bad guys aren't using it to cause too much damage and so on up to a certain point. Because one of the things we spend all the time worrying about I was like, okay, well, how do we ensure that these models are not lying to us? How do we know that like, they're actually not like plotting? You know, as silly as that sounds, like it's, we don't know what's going, we can't read their minds.

57:46We don't know what's going on inside these models right now. They're black boxes. And so an example of this that most people aren't aware of is that like when they released GPT-4, they actually like tried to see if it would escape and take over the world. And I feel like that is like, it's the thing that more people should pause and reflect on that. We're at a point now where like before they released GPT-4 and also kudos to opening AI for actually, they spent six months, you know, testing it for safety, which is good. No one was forcing them to do that. But like the fact that like, they're having to like, see like, all right, let's, let's see if our AI takes over the world before we release it should be like pretty concerning.

58:17If you, if like that, they, they think that's like a real enough thing that they wanted to test for. And then during the tests, they give it some like money and they basically like, you know, wanted to see if it could escape. Right. And it was able to hire a worker off TaskRabbit because they got stuck at a CAPTCHA, which is hilarious that like, it's this smart. They can hire somebody off TaskRabbit and yet still couldn't pass the CAPTCHA. It's amazing. It's a big vote of confidence for CAPTCHA technology. Yeah, right? Like, ah, alignment itself. So the key thing, though, is how do you get past the CAPTCHA?

58:46So hire a worker off TaskRabbit. And the worker said, like, why can't you just, like, click the button for the CAPTCHA? And the model, GPT-4, like, lied to it. It said, like, it made up a story saying, like, oh, I'm like a, you know, I've got, like, a disability and I can't see. And so the worker was like, okay. And the worker did it. And that's just like, that's a good example of the kind of thing where like that we have this AI, it already lied to a human, it hired a human, it lied to the human. And like, now imagine this, but imagine models that are like 100 times more powerful, or 1000 times more powerful.

59:15Because that's the thing about deceptive alignment is like, you don't know, like, you might have trained it to not to be honest, but like, did you train to be honest? Or did you train to not get caught? It's really hard to know which one you trained it for. So there's all these things that we worry about, like, where we don't know if it's safe. And like, there's all these risks of emergent capabilities that make it so that it's suddenly not safe. So yeah, anyway, there's just a lot of, there's like this, this, the alignment problem, like one of the things that happens all the time that like we get frustrated by is like people come into the alignment space and they spend like, you know, 10 minutes or like maybe an hour or maybe in a couple hours reading about it.

59:46And then they have, they have an idea that like, seems like it'll work to them. And, uh, it's like pretty much never an idea that we haven't like already, you know, like 10 years ago, like, you know, it's been explored for a long time. And it's just that there's so many different failure modes that it's like really hard to to be convinced about like how hard this problem is, is how you spent like 200 hours reading about it. Because it's just very hard to control something that is a thousand times smarter than you. Because almost any solution you come up with, it will be able to outsmart you on because it's a thousand times smarter than you.

1:00:14And if you don't like the word smarter, some people think of like, oh, smarter is like, oh, you mean like, some people, their version of intelligence is like good at chess and Sudoku and like, I don't know, trivia or something. And like, that's not what we mean when we say smarter. When we say smarter, we mean like problem solving. We mean like, it's better at like everything, like getting things done in the world. So that could mean making money. That could mean outsmart. That could mean like tricking you like right now, like GPT-4, I think this is another thing most people don't like really wrap their head around.

1:00:40GPT-4 read like every book ever published and the entire internet. Just imagine a human who read every book ever published and the entire internet. That means they read every book on persuasion ever. They read everything Machiavelli has ever written. And they've read like millions of conversations where people were trying to persuade each other. So imagine a human who'd read that much. That would be a master persuader. How jealous are you that it got to read every book ever?

1:01:08I'm on a pretty short list of people who've read the most books, because I've been averaging about a book a day for most of my life. And so I'm actually very dull. But also it makes me more worried, I think, than other people are, because I've seen how, well, whatever. Anyway. And I also think another thing that people should really try to wrap their heads around more is that like imagine that gpd4 right now is like a single um person that has figured out how to make like 100 million copies of itself so imagine one human makes 100 million copies because basically we've got this base model and then there's 100 million users so it's kind of like there's like 100 million people that are all training this one human and what's crazy is that whenever one of those 100 million copies like imagine the human again the human whenever one of those hundred million humans learned something, all 100 other million copies learned the same thing.

1:01:56And that's wild. And that's kind of like what happens. Cause right now, like there's another thing that really worried Jeffrey Hinton. This is like, he said, um, in an interview recently, he's like, he's like, you know, I spent like 50 years thinking that we needed to build algorithms, learning algorithms that were similar to the brain. But now he thinks that back propagation might actually be better because of the fact that, yeah, so you can have like a hundred million copies of this model. Whenever one learns something, it can propagate that information out to all 100 million other copies, in essence.

1:02:21And like humans, like if one human learned something, how long would it take you to teach 100 million other people that thing that you learned? Like, we can only communicate at like 40 bits a second with our words, whereas, you know, these models can communicate at like trillions of bits. So they could just learn much faster than us. And so there's all these like different feedback loops like that, that could give it the ability to just like far surpass us very quickly in terms of intelligence. Let me ask you a question, maybe as we wrap up, because I've got kids and dogs who are going to start screaming at me soon.

1:02:49Is there anything optimistic to you in the EU AI Act? Is there anything encouraging in there? Does it sort of, you know, point towards government being more effective than we might have hoped or less effective than we might have hoped as relates to these questions? Why not both?

1:03:34And my hope is that we'll be able to make sure that it's pointed more towards things that mitigate risk of extinction instead of things that the other risks they're focusing were more like the risks other than that. And those are real risks. But I have like very mixed feelings about the EU-AI Act. A couple of things that are interesting. One is, you know, it was this is in the same way that if you read MECA, which is their digital assets legislation, it reads like it was written about ICOs, even though it's now six years after ICOs. It's because it was. It was literally written about ICOs, and it's just this is how long it took.

1:04:07This is similar in the sense that it was started in April 21. It was pre-ChatGPT, generative AI kind of rise. And so a lot of that stuff was just bolted on at the end. So it's much more focused on a different set of issues. I think what's interesting to me, I agree, I share all those concerns. The interesting thing to me is the extent to which they have earmarked things that a lot of people agree are obviously bad and shouldn't be done. just as testament to what you were saying around we can actually say no to technologies as a thing that people aren't used to. As an easy for example, they basically said no minority report.

1:04:43You can't use AI to profile potential criminals and make assessments about that. Sorry, Interpol, you don't get to make bets and predictive bets about who's going to commit crimes and arrest them for it. And I think a lot of people are like, yeah, that's probably a bad thing. Let's not do that. And it doesn't seem to undermine a bunch of the other good use cases that I'm excited about with, you know, it's not messing up my mid-journey creations to have people not be able to be arrested for stuff they thought about, you know? And I think that, you know, again, to the extent that we're looking for small wins and nudging people towards a different understanding, I think even saying there are things that are okay to ban is an interesting step.

1:05:25I agree. It's not, you know, I don't know, whatever. I think that they run the risk, as always of sort of banning their citizens from using these technologies because, you know, companies are just going to go elsewhere, but it exists now. It is now a thing that is outside of the realm of theoretical that is actually happening. And so it feels important to at least understand in that context, if nothing else. Yeah. And I think that, so regulation, these things are really hard in general. I think one thing, I think I answered a question in an overly roundabout way earlier that, um, so you like, what do we do from regulatory perspective?

1:05:59The one thing I do think is probably robustly good right now is just to stop doing bigger training runs. Um, most people are not worried about, um, current models taking over the world. I think that's really important. Cause I think a lot of people get lost on this. They're like, look, well, I I've used chat GPT and it doesn't seem like the kind of thing that could just take over the world. And, uh, I agree. I don't think it can take over the world either. And I think most people in the AS safety community aren't worried about GPT four taking over the world, But people start to worry about GPT-5 and GPT-6 and GPT-7.

1:06:27And, like, I think there is somewhat of a consensus that, like, GPT-4 is, like, there's this notion of an AI summer harvest, which is that, look, GPT-4 is incredibly powerful. And we should, like, we should enjoy a nice summer where we harvest all the gains, productivity gains. Because, you know, it oftentimes takes, like, 10 years for technologies to widely propagate and the productivity benefits to be fully harvested by society. so why don't we like we pause on bigger training runs and like allow gpd4 like gain a better understanding of gpd4 like right now right now there's there's a field called interpretability where we're basically trying to figure out what's actually going on on inside these models you can think of like digital neuroscience right now they don't know i would round it to zero for like how much they actually know about what's going on say these models but like plausibly with like a few years of work we could like have a much better idea what's actually going on and so we could have like a few years or whatever.

1:07:20I don't, I'm not proposing anything specific, but like the idea is that we like, let's harvest the benefits, the productivity benefits of the existing models. Let's pause or slow down or just like more carefully control bigger training runs because that's where like the extinction, like the current models are more of a scary thing for like misuse, bad guys doing things. And future models are scarier from the extinction side. For example, the UN secretary general yesterday said that he is like, I think he's basically broadly in support of having an IAEA for AI, which is a single regulatory body to help set the guardrails around what these models can and can't do.

1:07:58And I'm broadly in support of that. I'm very uncertain in general, but it seems probably good to me to do something like that as anti-regulation I am in general. And with all the same cynicism that probably, I would guess, most of the people listening of this right now have about like ineffective government agencies, um, like creating even more market failures than they actually prevent. And that's the thing we're trying to figure out, right. Is like, um, the, the point of regulation is to like allow markets to function more effectively markets by themselves don't stay open very long. Uh, typically a free market is only briefly free before some entrepreneur wraps it in chains and it's no longer free again.

1:08:35And then you need to have some sort of referee, some sort of umpire, some sort of like, you know government like intervention to like open the market back up again um so that the diversity of like markets can flourish again anyway that's like one thing that seems good and then the other thing that i just want to double down again is like open source open source to me feels like we might really lose the ability to have any control over our destiny if open source progress continues at the current pace like the fact that and also here's another thing too i think is really important too so people worry about china like if you if you worry about china you should be very worried about open source because you can have companies like meta which spend a fortune to train the state-of-the-art models.

1:09:11And then if they just open source the state-of-the-art models so that China gets it right away, that's like, I mean, we have the CHIPS Act right now, right? We're spending a fortune to make sure that China doesn't get access to cutting-edge chips. If you prevent China from getting cutting-edge chips, but you let meta, you let your own technology companies just like openly share the technology with China, then you've completely negated the point of the CHIPS Act in the first place. And so you should be anti-open sourcing AGI if you're like pro-CHIPS Act. And like, yeah, anyway, there's just a lot more complexity to this than people realize.

1:09:42But like, the thing I want to get at is like, if once we open source it, once people can run their own models, like at home, then it could be very, very, very hard to actually regulate it and control it moving forward. And so that's like a thing that keeps me up at night. Last question for now, because obviously this is going to be a fast evolving conversation. What do you recommend people go read to learn, right? If they're just trying to wrap their heads around this, but maybe they don't have the 200 hours that you recommended, you know, but they want to be, have a meaningful take on it so that they can be part of the discourse, even if it's just understanding who to vote for because of what they're saying or whatever it is, you know, what, what, what do you recommend?

1:10:21If you're interested in technical things, go to ai-safety.info and watch Robert Miles' YouTube videos. If you're interested in general in this problem, not so much the technical side, go watch Tristan Harris's AI Dilemma. It's like an hour-long talk. And it's just like a good kind of like summary of like the various risks involved and like what's at stake. Another person who, by the way, comes at tech ethics from the standpoint of being a tech entrepreneur. He sold the company to Google and they were friends of mine back in San Francisco. So really interesting perspective and definitely someone who has thought deeply about these things.

1:10:56And who also, I would say, one of the challenges that I find with the discourse around AI is how much of it is re-litigating social media battles. You know, like if you watch the hearing, it was all just section 230, section 230. We shouldn't have had section 230 from the politician's side, right? It was all social media. If you read Andreessen's piece, it's clearly still kind of caught up in the social media wars and what the New York Times thinks about Silicon Valley and all that sort of stuff. Tristan actually has, he started all of his work around questions of social media and I think is a little bit more capable of kind of getting outside of that perspective to bring this.

1:11:35So cosine, I guess is what I'm saying. Yep. And then actually real two quick ones I just thought of. One is Don't Look Up, the documentary. That one's a little shorter. It's not as in-depth on the problems, but it's still really a good intro. And then the last one is actually Tim Urban's Wait But Why post on artificial intelligence. It's kind of old. It came out years ago, but it's really good at helping you get some deeper foundational ideas about how transformative AI is likely to be. So go read his blog posts on that. Awesome. Emerson, so good to have you on the show. I can't wait till the next one.

1:12:05Thanks, man.