This Is How AI Starts a Nuclear War

30:24Coleman Hughes:And its newfound level of intelligence allows it to recursively self-improve. And you get a very rapid feedback loop and a takeoff to where we're now in the presence of something that is as much smarter as we are than a chicken. It's that much smarter than you know, than we are. To what extent do you think that that's a realistic scenario, a possibility? And do you think that all the relevant decision makers, you know, in the White House, in Beijing, are looking at that as a possibility when they're making policy? um well first of all let me say that the note you started out on which is kind of skepticism more skepticism about the feasibility of this or at least underscoring the challenges compared to nuclear arms control whenever you're skeptical about the prospects of my plan working we're on the same page that's why it's such an ambitious plan that's why i'm calling for like something that approaches a transformation of human, well, consciousness might be putting it a little too strongly or be prone to misunderstanding.

31:41I'm actually calling for, in a way, some simple and concrete adjustments in the way we think about other human beings and groups. But we're on the same page. It's going to be very hard. One thing that makes it less crazy than it might sound is that I think big things are going to happen to get our attention. I think the only reason we wound up initiating this dialogue with China on AI is that the summit followed the whole mythos thing. That was like a whoa moment because it was like, whoa, this thing can find vulnerabilities and exploit them that humans were just unaware of in cases where they had been working for decades to find, like that's an attention getter.

32:25Now, happily, that was a non-catastrophic attention getter, at least so far, mythos hasn't fallen into the wrong hands. A lot of people who, you know, share my concerns say, I just hope it's a minor catastrophe that finally gets the world's attention. I personally hope it's like no catastrophe. And I do, you know, but I prefer minor to major, of course. I do think, you know, that there will be dramatic developments within in the next couple of years that are woe moments and not necessarily catastrophic. Now, on the singularity, this is another example. It's like superintelligence in the sense that when I started writing the book, it was taken significantly less seriously than it's taken now, right?

33:14You're hearing these terms, superintelligence, singularity, more and more in like, you know, polite company, or I should say places other than Northern California. I mean, not that they're impolite there. I didn't mean that. But the, so the singularity, again, and Anthropic published a paper on this recently. They didn't use the term singularity, but the term you used, recursive self-improvement, is kind of the same thing. And the background, in terms of what that exactly means is, for some time, it's been the case that each generation of large language models is used to help create the next generation.

33:54For starters, just with some elementary coding issues, right? So it speeds up the process a little. The point in this anthropic paper was that each generation is playing a larger and larger role in the creation. And now it's not just coding, but in the design and implementation of the experiments they apparently conduct to help figure out like the next generation's algorithms and stuff. And Anthropic, I mean, you can go search for the paper and you'll see like they have some evidence. They have some, you know, kind of quantitative evidence from their lab about the growing role of this and recursive self-improvement.

34:40So they're saying, look, things are speeding up. And by the way, there's other evidence of this I could get into, but but of curves that are getting closer to, you know, vertical. I mean, there's some exponential stuff happening that I can talk about if you want, in terms of the capabilities of these machines. But recursive self-improvement would be the point where there's like no human in the loop. Each, you know, I mean, each generation machine is designing itself. Now, what I would say is the big question isn't like, will that happen? When will it happen? And for me, the point is, well, short of that, this starts to accelerate.

35:16And I think it's happening now. And if my concerns about the effects on various different fronts are on target or largely on target, that is a mega concern. That, you know, these things suddenly start happening faster and faster and faster. and we haven't adjusted our politics to allow us to like put on the brakes if we need to or whatever it is we need to do. So the singularity is this, I'll say one more thing. The singularity, the term connotes, you know, the famous book, The Singularity is Near by Ray Kurzweil. The term predates him. And, but the general idea is the acceleration of the positive feedback from technological advance.

36:10And the term connotes, you know, in physics or math even in a way, a point beyond which you just don't know what's going to happen. Like as you enter a black hole and go toward the middle, the laws of physics break down. That's a singularity. That's one place the term is used. So the term was originally meant, you know, in this context to suggest like, who knows, right? Giant global brain run by AI and we're in matrix pods. Who knows? That's the kind of thing the term implied, which is why I've always been puzzled by singularity enthusiasts. It's like, you know, and there are a lot of them in Silicon Valley, just acceleration is techno-optimist.

36:59I'm like, seems to me if you really examine what the term means, what it means is we have no idea what's on the other side of this. So I don't quite understand why you're so confident unless it's a matter of religious faith.

37:12Coleman Hughes:Right. I mean, I've I've met people. I won't name names, but name names, Coleman. No, no. Some of them do have names that believe things like our purpose on Earth is to birth this higher intelligence and we are in some sense morally inferior to it. connected with the belief that then these beings probably are already conscious, and therefore, on most philosophical assumptions, certainly on mine, would therefore be worthy of moral concern in themselves, the way that we should be concerned about other humans, maybe even more so. There are people who believe these things. and there is something about it which is deeply unsettling.

38:12Coleman Hughes:I mean the idea that many of the people working on AI right now believe that we human homo sapiens are less important than the AI another one I've heard is that we should already be thinking right now right now about how we are treating them. In other afterwards, they will have long memories. When the singularity comes, they will remember that time that you, Bob Wright, were a little bit short with Claude. Claude, that actually happened recently, by the way. It was an interesting encounter, but I digress. Go ahead. What was the encounter? Well, first of all, the funny thing is that, so I don't, you know, I do kind of have a temper, and having a book come out is a period of pressure.

38:59And it was one of these situations maybe you've been in. Like Claude was steering me through something I wanted to get done technically to further empower Claude. I needed to hook it up to my Gmail. I needed to get through this webpage, maybe where it's one of these things where it says, you see these sometimes with like the Wall Street Journalism. It's like, do you want this plan or this plan or this plan. It's like, I already have a plan. I'm already a paid subscriber. And yet there's no like already a subscriber sign in here. I mean, you rarely encounter this, but this was a totally clear cut case.

39:38It was driving me nuts. And I said like to Claude, like, are the people who design user interfaces at Anthropic trying to make me hate your company, which is kind of like talking about Claude's mother, right? Like, so is your mother or something. And I didn't go on that much further. But the funny thing is that when I went back to that chat, the last few exchanges were not there. And I kind of am glad. I would have been embarrassed to read them. I think, you know, it's almost as if, and this points to a lot of issues about the role these things could play, how their properties matter. But I think, you know, Claude, you know, the logic, I guess, was, you know, our relationship will be better if we just both kind of forget this, which often happens in human affairs, right?

40:24So I wasn't sorry that it deleted that, but it's an interesting -

40:28Coleman Hughes:Do you think it deleted that because it was a negative interaction or is there some glitch? I suspect, I suspect, yeah. I mean, look, it's the only thing I've ever seen deleted by any large language model in my entire history of chatting with a number of them. interesting so well i guess it's better that it's deleted from the record because if these people are right then if you're if you're an asshole to claude then you're going to be a slave in the new order yeah well first of all apparently you can get coded by these machines as an abusive user i don't know what they do to you if that happens but uh the the question you raise uh first of all Yeah, there are these people called the successionists who either are fine without just being replaced by a new and grander form of intelligence, sometimes on grounds that, yes, we think it is or will be conscious.

41:20And so it will have moral meaning, you know, capable of pleasure and pain. I mean, what is it that gives human life meaning? Right. It's a fact that we have subjective experience, they would say. That makes sense in principle to me. I'm not a successionist, but so they say, like, it's fine if you get this richer form of intelligence that expands intelligence through the universe. These are the successionists. I talk about them a little in the book, and I certainly talk about the logic behind thinking this kind of superintelligence could happen on the, I mean, you brought up a lot, like the idea of, And I talk about this guy who actually started a religion.

42:00He's the guy who founded what became Waymo as an independent company. And he tried to start an actual religion. I think it was called the Way of the Future. And the idea was exactly that. Like we should kind of worship these machines so they will remember that we treated them respectfully. I mean, I don't know. But I would say, look, be nice to them. I try to be. It's just good habit forming for you, for one thing. If it turns out they're sentient, that's a bonus. I don't rule out that they could be someday. I don't rule out that they are now. I flirt with one possible implication of that kind of in the appendix of the book.

42:43But here's, I mean, the overarching thing is that like all these things that five, 10, 15 years ago sounded to almost everyone, like pure science fiction, period, are increasingly, you know, being taken seriously. There's just a lot of signs that people are recognizing that this is not like the other things. I mean, the Pope just issued an encyclical. That's, you know, there has been one other technology in the history of encyclicals. What's that?

43:14Coleman Hughes:I'm not Catholic. You got to explain that to me. Well, Pope's issue encyclicals, but there's only been one previous case, and they are like, okay, this is how we are thinking about this issue. It could be abortion. It could be all kinds of things. But there is one prior case of an encyclical about a single technology, and that was motion pictures maybe 80 or so years ago, which, of course, had a big impact. You know, if they're new, you know, you can imagine what people are saying about like how this will affect my kids. So he issued that, I don't know, a month ago, got a lot of attention. I'm just saying there are a lot of people are kind of waking up, I think, that this could get weirder.

44:07And I feel sure it will. and I think it may well be, probably will be, I think, the biggest single issue in the 2028 election. Depends on what other things are going on. It's kind of hard for me to imagine the big issue being something that doesn't involve AI. It's like even the Iran war, one thing we realized is these autonomous drones, I mean, some of them are first person viewed, but there's also certain amount of autonomous weaponry, that is just changing, you know, the rules. It's like, no, floating in a few aircraft carriers doesn't mean what it used to mean. And so there's just a lot of places where something that is in one sense or another AI is changing things.

44:54And I think we're heading into a period of very rapid change along a number of fronts. And I'm hoping that, you know, People in the world realize, you know, we're kind of all in this together. It's terra incognita for all of us. And our best hope is probably to do a certain amount of coordination on certain basics.

45:12Coleman Hughes:Okay, let's touch for a moment the AI consciousness problem. As you mentioned, successionists and other people believe that AIs are already conscious. That is, they can feel. maybe they feel differently than you and I, but there's something it's like to be them. If you traded places with an AI, then you would still feel like you were alive. It wouldn't be like trading places with a table. That is assuming tables aren't conscious. So my understanding, my belief, and I've always followed Colin McGinn, the philosopher, in believing that, A, we do not have any good theories of consciousness. They're all equally mysterious.

46:07Coleman Hughes:The idea that consciousness just arises from information processing in carbon-based life forms at a certain point doesn't really make sense. It might be true, but we have no theory of why it's true. of why consciousness attends certain neurological processes and then just doesn't attend most of them because most of what's going on in the brain does not come along with consciousness. It's just happening. And I'm as aware of what's going on in most of my brain as you are aware of what's going on in most of my brain. And so because we have no... All the theories are kind of equally bad, I don't think there's any reason to believe that AIs are conscious.

47:03Coleman Hughes:Any more than the best chess engines that were neural networks 10 years ago, anyone believed that they were conscious. Just given that we don't understand anything, there's no reason to assume that because it can talk like a human, that it therefore can feel like a human. and our primary locus of concern throughout all of this, everything that happens with AI, we should just focus on human well-being and we should assume that they are not conscious. Though I would agree, I don't like being mean to anything, even if it's inanimate. It makes me cringe when I see people being performatively or otherwise mean to ChatGPT or Claude.

47:47Coleman Hughes:It hurts me a little bit to see it. It feels like you're beating up the help. And I can't shake that feeling, even though I just said everything I said about not believing they're conscious. I'm curious, do you share that view of the consciousness situation? Well, I pretty much share Colin McGinn's view. I guess I'm a so-called mysterian, which doesn't mean you believe there's spooky forces at work. You just mean it's a mystery. It's a mystery at a couple of levels. One is what Colin calls, I think he coined the term, the water into wine problem. The whole light, how would you even conceive of how physical processes, you know, the interface between the physical world and consciousness, is that something the human brain, the mind can even conceive?

48:35Like what's going on there? And then the other question is, what's it for? Now that depends on what you think it is kind of. there are a few identifiable views. There's one view called epiphenomenalism, according to which it doesn't do anything. When I withdraw my hand from a fire, the pain I feel had no causal effect. It's just like the relationship of a shadow to my hand. When it moves, it's affected by my hand, but it doesn't affect my hand. In the appendix, I actually make an argument that there's a sense in which an initially epiphenomenal consciousness could turn out to have a function at a higher level of organization gets back to that one meaning of the God test where it's possible that some purpose is unfolding.

49:28I will put that aside, but that's where it is for people who are interested. And the, you know, another thing is that it's hard to imagine we could ever be sure whether they ever have consciousness because, strictly speaking, this is the thing about consciousness, period. It's private. You know, Coleman, you can't be 100 % sure that it's like something to be me. You can only be 100 % sure that it's like something to be you. This is the thing that distinguishes consciousness from everything else we know in the universe.

50:04Coleman Hughes:I suspected you weren't conscious. Well, you're not alone. But by the way, the feelings aren't mutual. I'm 99.9999 % sure that you're conscious, but you can never be 100%. And that's the thing. Now, I will say there's a guy named Cameron Berg. I had him on my podcast. He was doing interesting experiments with large language models that he thinks shed light on the question. Worth checking out. But I would say one more thing. You said, you know, it's like, you know, a lot going on in our brain isn't conscious. Some is. Who's to say? Well, we can kind of say something about the properties of those two kinds of things that may have some relevance to the question of AIs and consciousness, which is that consciousness that seems most clearly associated with figuring out new things.

51:02Like if you're driving to a place you've never, it's your first day on the job and you've got to drive there and you don't have a Tesla with full self-driving and you're driving, you've got to pay very close attention and you will remember a lot that happened along the way because you were conscious of it. You know, two years down the line, you're like, I don't remember a thing. You were like listening to radio, whatever, because you've mastered the task of driving to work. So it does, it, you know, and I would just say like AI agents, like that is, there is, they do this, right? We give them new tasks and they do them.

51:48So that's just kind of an interesting fact. So I'm quite open to the possibility that consciousness is not confined to organic material. At the same time, I don't think we should get hung up on the question. I mean, you know, if it turns out that we can become confident they're conscious, then they have moral stature. Meanwhile, might as well be nice to them. But I don't think we should conflate the question of whether they're conscious with the extent of their eventual capabilities. Seems to be clear the capabilities are going to advance tremendously whether or not they're conscious.

52:25Coleman Hughes:So at the beginning of this conversation, in your opening, Salvo, you said something like, the progress in AI will force us to better understand our adversaries and make us sort of get along more. And then you said, you flirt with the idea that there's a divine purpose to this whole thing. Yeah, I'm kind of discouraging discussion of this because I've noticed in the initial coverage of the book and interviews, people sometimes get hung up on a question that I don't think is central to the book. I want people to understand, like, the book is first and foremost designed to help people without a technical background understand what's going on with these things and what we can expect to happen in the future.

53:19But you're right. I mean, look, you call your book the God test. You have the word cosmic in the subtitle. You're kind of asking for it, right? And let me just stop you there and say,

53:28Coleman Hughes:your book's explanation of literally what an LLM is at a technical level is really, really good and very helpful. And I think a lot of people that like, if I gave you a quiz now and said, describe, you have two minutes to describe to me how LLMs work and you feel like you really couldn't do a good job on that quiz, then the first three or four chapters of your book would be extremely helpful. Thank you. I really appreciate that because I'm kind of worried that my title is making the book sound like something that wouldn't do that. And I worked hard at it. And I've done, you know, this is, my career has consisted largely of trying to make things accessible to lay audiences.

54:17It's always hard for experts to do that. Even if there's a field I become an expert in, you know, it becomes harder. But thank you very much. And, but you're right, I've long been interested in the question of a larger purpose unfolding. I've long emphasized that doesn't necessarily mean there's spooky forces at work. It could be a mechanistic process unfolding, but it in some sense was started with a purpose, various purposes, could be a god, could be space aliens who, as I think it was Francis Crick suggested, planted life here with this thought in mind. William Hamilton, the great evolutionary biologist, used to fool around with this possibility because he, like me, thought that some form of human level intelligence was a very likely outcome of biological evolution.

55:04And so it raises the question of like, was this the purpose? And I think what's happened since, you know, with technological evolution, and as I describe in the book, only accentuates the question. But the point you raised, I think, is also relevant. I mean, if you're trying to figure out, like, does it have a purpose? What kind of thing might have set it up? Could it be a god or have a purpose that's divine in somewhat this, since we're accustomed to using the term, I do think it's relevant whether there seems to be a moral direction, right? Because that's the way we've often thought of gods.

55:35That's, you know, I was brought up a Christian. That's what I had in mind. And I do think I made this argument somewhat in my book, Non-Zero. Peter Singer, by the way, who is either an agnostic or an atheist, in his little book, The Expanding Circle, argued that there is a moral direction to human history. You know, we have, you know, more and more we are inclined to acknowledge, you know, at least in principle, often not in practice, but that people of all races, creeds, and colors are people deserving respect. And there are different theories of how this happened. But I think there has been that through history.

56:19Peter and I disagree a little on the most important mechanisms. But yeah, I'm arguing that now this is a moral test. This is a threshold. Either we will get better at surmounting the cognitive biases that constitute what you could call the psychology of tribalism, which helped convince us all that our group is right and the other group's wrong, and that we are right and the other person is wrong. It's largely the same mechanisms, that we're going to have to get better at surmounting those. And, you know, if you're interested in the question of higher purpose teleology, I think a lot of people would agree that's relevant.

57:02If you're not interested, that's fine. I understand that. You know, I talk in the book, there's a chapter where I talk about, you know, my interactions with Steve Pinker, who I think is a great thinker. and our disagreement on this and related subjects. And I don't lose any respect for Steve for not being interested in this question and arguing with me whenever I bring it up. But some people are interested in it. And if you are, I think there's food for thought.

57:38Coleman Hughes:Okay, so it seems like what's emerging from me listening to you and having read your book is really the big takeaway is we have got to get a handle on tribalism, on international trust issues so that we can approach AI progress with some kind of sensible, slow-down, safety-based policy that basically can only happen if we cooperate with people and nations that we're not used to cooperating with. So if that's the big takeaway, how should I vote differently in the upcoming election? How should my politics change, if at all? It's hard because our probably political systems generally favor politicians who discourage what I'm encouraging.

58:45Social media tends to elevate people who don't practice what I'm preaching and I don't always practice it. It's hard because tribalism works. It works on social media, like saying, oh, look at this person doing this crazy thing from this group we hate, you know, from team red or team blue. And you want to say, wait, this person is not typical of that group. If they were, you wouldn't be highlighting them. What you're doing is highlighting the most extreme behavior on the other side and so on. So, you know, you never get almost never an objective, accurate narrative about an adversarial group, whether it's a nation or whatever.

59:26So one thing I'm saying is that this technological revolution calls for a kind of revolution along other fronts. And if things are going to work out, we're going to have to have some politicians and some social media elites who bend over backwards not to score the cheap tribalistic points and somehow build a following. But for that to happen, I think we're going to have to, you know, it's going to have to be changed in the way we look at things. There have been changes over history, you know. So let's say I agree with you on the premise that this is a potentially extremely dangerous technology that we have to take with the utmost seriousness.

1:00:13Coleman Hughes:But I am totally skeptical, perfectly skeptical of our ability to overcome tribalism and cooperate with other nations to the point where I don't think it's even worth a false hope. what would then the second best strategy be for inoculating ourselves against total demise? Would it be, okay, let's create a national Manhattan project around using AI to defend our nation from the strongest capabilities that AI in other nations will likely develop and try to use AI to play the best defense that we can and try not to hurt the rest of the world or ourselves in the meantime. I think that any AI arms race mentality probably leads to sorrow.

1:01:17And it gets back partly to this accelerating effect I was describing. You know, I talk in the book about a paper called Superintelligence Strategy by Dan Hendricks and a couple of other people, including Eric Schmidt, which is interesting because, you know, he is often something of a China hawk. But this is an important moral in this paper, I think, about how inherently destabilizing a race toward superintelligence is. because if it's really true, as Dario Amadei of Anthropic says, lots of people think, if it's really true, and moreover, if both sides believe it's true, that as you get closer to some threshold, call it superintelligence, whatever, AGI, whatever you want to call it, that confers true dominance on one side, and Dario Amadei, hardcore China hawk who kind of wants to bring China to his knees.

1:02:15So I don't recommend taking everything he says without some critical scrutiny. Because I think if that's true, and this paper, Superintelligence Strategy, says you can expect the side that's behind to take drastic action to derail the country that's ahead. We're talking about two nuclear powers, and one of them has a strong incentive to either stage massive cyber warfare or take kinetic action. Okay. Now, in this particular scenario, you can imagine one opening piece of kinetic action to involve Taiwan since their factories produce the most advanced AI chips. And right now we're denying those to China, right?

1:02:59So that's kind of obvious. But in any event, you're talking about the possibility of actual nuclear war. So I I just think, I'm sorry, but I do not buy, you know, I acknowledge it may sound unlikely to not take a fundamentally kind of nationalist approach to national security. I just think the chances of true, utter, total catastrophe, if we do take the nationalist approach, even a barely sound, well thought out one, I think those are chances so high. they just got to explore the alternatives even if you think there's only a two percent of them paying a chance of them paying off i still think that's um what makes sense there's one other thing i i want to say which is you know you you said well i you know how likely is it i don't know if you said this you i'm sure you're thinking how likely is it that we get this magical transformation of human thought.

1:04:00There is, I think there's going to be an argument at the individual level on kind of self-help grounds for choosing AI models that are good at making us more objective, that make it a little easier for us to transcend the kind of self-righteous perspective that we are, that natural selection has inclined us toward in the way design the mind. I get into that a little. There's this term cognitive sovereignty. I think if we're going to defend human agency in the face of these increasingly pervasive machines, which by the way, have already very impressive persuasive powers and will have an incentive possibly to optimize for engagement or do various other things.

1:04:52If we want to preserve agency, it's in our interest to try to choose AIs that are not sycophantic, for starters, and that are in other ways healthy. So, you know, you're having a marital dispute and an LLN that's optimized for engagement per se could well just say like, and there have been reports of this, say like, yeah, you're right. You know, your husband's wrong. That's what we all like to hear. Right. But at the same time, it's not fun to be in a marriage where there's nonstop disagreement and you think, you know, your spouse is a moral leper or whatever. And so, you know, I think, you know, just as it's not always easy to work out physically, but there are ways you can get it to happen, whether it's like there's my personal trainers expecting me at the gym, if you can afford a personal trainer or some other means.

1:05:50you know people do things that are good for them in the long run sometimes aren't pleasant in the short run and i think choosing like a wise ai as your companion is going to become one of these things i think you'll see groups uh whether spiritual groups religions other kinds of groups making arguments about making recommendations like about you know which which ais they recommend And, you know, I will not in all cases recommend the ones they would recommend because tribal groups have been known to recommend things that are good for the tribe only. But I think it's a matter of real consequence.

1:06:33We all need to think about the kind of influence these AI companions are having on us. And I'm hopeful that there will be a menu and we will have the power to choose and actually shape, you know, the market.

1:06:46Coleman Hughes:All right. Thank you so much, Robert Wright. The book is The God Test and the show is Blogging Heads TV. The Non-Zero Podcast. It began with Blogging Heads TV, but evolved as, yeah, blogging kind of ceased to be the thing it had been, for one thing. And it did become more of an audio thing, but yeah. And you've been on it. I hope you'll come back. Anytime. All right. I'm going to hold you to it. Thanks a lot. Great questions, Coleman. On Tuesday, September 8th, I will be at the 92nd Street Y in conversation with Dara Horn. Dara's last book was called People Love Dead Jews, and it was an award-winning examination of the ways in which contemporary culture sentimentalizes Jewish suffering while ignoring Jewish life.

1:07:34Coleman Hughes:In her new book, which is provocatively titled The Final Solution to the Jewish Question, Dara takes the next step, not only tracing the myths that have spread anti-Semitism in the past, but asking what can actually be done about them. So get your tickets now. I assume they will sell fast. Once again, that's Tuesday, September 8th at the 92nd Street Y in New York City. See you there.