Why Do People Who Think AI Could Kill Us All Still Work on AI?

0:00Today on the AI Breakdown, we're asking why people who are concerned about the extinction risk of AI continue to work on AI. The AI Breakdown is a daily podcast and video about the most important news and discussions in AI. Go to breakdown.network for more information about our newsletter, our Discord, and our YouTube. Hello friends, welcome back to another Artificial Intelligence long read. Today we are reading a piece from June by Daniel Eath that asks what I think is a really fascinating and important question. The piece was published on Medium and is called Given Extinction worries, why don't AI researchers quit?

0:37Well, several reasons. And it was inspired in part, it seems, by the one-sentence statement that had been signed right around that same time by many prominent AI researchers that said simply, Mitigating the risk of extinction from AI should be a global priority, alongside other societal scale risks such as pandemics and nuclear war. Daniel points out that this was signed by an absolute veritable who's who. Jeffrey Hinton and Yoshua Bengio, two of the most cited AI researchers in the world, and both Turing Award winners, the CEOs of OpenAI, Google DeepMind and Anthropic, the chief scientist of OpenAI, the chief AGI scientist at Google, the CTO and the chief scientific officer at Microsoft, even Bill Gates.

1:15Daniel says, okay, so why are these researchers still working on AI, and why aren't CEOs pulling the plug on their own labs? Now, Daniel caveats that the reasons that he lists here are not exhaustive, and he also points out that there are many AI researchers who don't think AI poses a risk of human extinction. But he says, this piece isn't about them, as their reasons for continuing to work on AI aren't as confusing. And at this point, we begin to actually read Daniel's words, and for that, I turn this over to the most recent version of the NLW voice created by Eleven Labs. I think you'll find this has some pretty significant advances over the last time we did an AI longreads, and I would love to hear your feedback on how this compares or what you thought about it.

1:57Once again, this is Given Extinction Worries, Why Don't AI Researchers Quit? Reason one, their specific research isn't actually risky. The term AI is an incredibly broad term referring to many different types of technology, from large language models like GPT, to chess-playing software like Stockfish, to self-driving car technology. Worries about extinction from AI typically involve artificial general intelligence, a GI, a hypothetical future type of AI that could outcompete humans generally across many domains, such as long-term planning, social persuasion, and technological development. If your AI research involves, for instance, improving self-driving car technology or applying current AI techniques to better diagnose cancer, then it is unlikely to actually accelerate us closer to AGI, and thus it may be irrelevant for extinction risk.

2:44Some AI research even makes it more likely that if AGI is ever developed, it'll be easier to control and thus less likely to cause human extinction. For instance, mechanistic interpretability research tries to make AI systems more understandable, instead of being opaque black boxes, And scalable oversight research attempts to make it easier for a human overseer to provide feedback to an advanced AI system so that it may internalize goals aligned with the desires of its designers. There is absolutely nothing hypocritical about an AI researcher who is pursuing either research that's not on the path to AGI or alignment research to be sounding the alarm about the risks of AGI.

3:19I, consider if we had one word for energy researcher which included all of A, studying the energy released in chemical reactions, B, developing solar panels, and C, developing methods for fossil fuel extraction. In such a situation, it would not be hypocritical for someone from A or B to voice concerns about how C was leading to climate change, even though they would be an energy researcher expressing concerns about energy research. Reason two, belief that AGI is inevitable and more likely to go better if you personally are involved. As a piece of background context, it's worth noting that a large portion of technologists are, to first approximation, technodeterminists in the sense of believing technological progress is basically inevitable, with regulation or other social forces only temporarily slowing down technology or moving around where it gets developed.

4:05Whether or not this view is accurate is outside the scope of this piece. From that assumption, many researchers conclude that AGI will get developed eventually, whether or not that's a good thing. Additionally, many AI researchers think there is an unstoppable race to AGI, so even if most technological development isn't inevitable, AGI in particular might be. Again, whether or not this unstoppable race framing is correct is beyond the scope of this piece. Many people assume it is, though others disagree. And from the view that AGI is inevitable, researchers who are concerned about extinction risk may conclude that if they personally get involved, they might be able to help steer the ship in a better direction.

4:40I've heard several reasons for thinking this, but two are particularly common. Being a voice of caution might push their lab or the field to be marginally safer. Remember, while many AI researchers are concerned about extinction risk from AI, it's not a universal concern among AI researchers. So a researcher at a top lab pushing to take the risk seriously could in principle affect how much the lab as a whole does. Alternatively, a prestigious researcher pushing to take the concern seriously might make other researchers also take it seriously. Hinton and Ben-Guy Oh, for instance, seem to be successfully acting in this manner now, though I have no reason to assume their recent pro-safety stances were planned far ahead of time.

5:18Working at a relatively more cautious or ethical lab may make that lab more likely to win the AGI race. AI labs as a whole vary in how generally cautious they are and how seriously they take extinction risk. An AI researcher that works for a relatively more cautious lab might consider that they're helping a more cautious lab win against less cautious labs in a race to A-G-Y, which they might think would decrease extinction risk on net. Similar arguments are also sometimes used for advancing the AI capabilities of one country over another. Reason three, thinking A-G-I is far enough away that it makes sense to keep working on AI for now.

5:54Some researchers think A-G-I will eventually present an extinction risk, but not for a while. These researchers might still support alignment research so that we're in a better spot when A-G-I is closer, but think that calls for slowing down or restricting AI development now are premature. In an extreme case, someone with this view might consider worrying about extinction risk at all to be fruitless. For instance, Andrew Eng, co-founder of Coursera and former head of Baidu AI Group and Google Brain, likens worrying about extinction risk from AI to worrying about overpopulation on Mars, perhaps something that'll become an issue in hundreds of years but not for a while.

6:28It should be noted that AI researchers are all over the map regarding how far away they think AGI is. Some think AGI is less than 10 years away. Some think it'll take a few decades, and some think it'll take over a century. Reason four, commitment to science for science's sake. Many scientists and technologists have unwavering support for scientific progress as a matter of principle, or otherwise an impulse towards advancing scientific progress, regardless of how any cost-benefit analysis of any specific scientific advance would come out. This sentiment was neatly captured by Robert Oppenheimer's statement, When you see something that is technically sweet, You go ahead and do it, and you argue about what to do about it only after you have had your technical success.

7:07That is the way it was with the atomic bomb. Similar statements have been made by AI researchers regarding AGI. Hinton himself says that he used to paraphrase Oppenheimer's above statement when asked how he could work on a technology so potentially dangerous. Others are drawn to the creation of AGI as an intellectual challenge. Reason 5. Belief that the benefits of AGI would outweigh even the risk of extinction. In order to understand this viewpoint, you have to understand just how powerful many technologists expect AGI could be. Many technologists expect AGI to relatively quickly become far smarter than the smartest human, providing corresponding technological advances.

7:44This viewpoint is illustrated nicely in the hit blog Wait But Why, which is particularly popular among the tech community. Note that Wait But Why uses the term ASI, meaning artificial superintelligence, for an AGI that's far smarter than the smartest human. What sorts of technologies could such an intelligent AI create? Many consider that it could perform thousands of years of technological advancement within a few calendar years. A common view is that it could relatively quickly unlock advanced nanotechnology, that is, atomically precise manufacturing, the ability to build complex objects with atomic precision, and AGI combined with advanced nanotechnology might allow for some particularly spectacular technologies.

8:23Above all, many consider the possibility that AGI might enable a cure for aging, allowing those alive at the time to live for thousands or even billions of years in good health and biological youth, even if they are chronologically much older. This sounds crazy because people typically think of aging as inevitable. But while it's inevitable today, the thought goes, that's only because we don't have a robust treatment for it today. At base, the deterioration process in aging is simply a matter of the atoms in your body being rearranged into configurations where you become more frail and your health declines.

8:53In principle, a comprehensive enough understanding of the relevant processes, combined with the ability to rearrange molecules and fix subcellular damage should allow for rejuvenation from the damages of aging itself, allowing, for instance, someone who is chronologically 70 years old or even 700 to become biologically 25. It's not uncommon for techies to claim in public that AGY might allow for a cure for cancer and then in private admit that what they're really excited about is a potential cure for aging. This coyness isn't generally a cynical attempt to deceive anyone. It's just that scientists recognize that if you talk about curing aging in public, you'll usually be labeled a crackpot versus you can talk about curing cancer all you want.

9:32But understanding what scientists really imagine here is key for understanding why many are willing to run an extinction risk. Almost no one would seriously risk human extinction in exchange for a cure for cancer. But there are many people who would seriously risk human extinction in exchange for a fountain of youth. The same post from Wait, but why referenced above shows the way that many who think about this issue grapple with it. To quote the author Tim Urban,

10:39reason 6. Belief that advancing AI on net reduces global catastrophic risks via reducing other risks. This reason is somewhat of a variation of reason 5. There are a few different specific lines of argument that this reason can take, but the most common are that AI is key to solving either climate change, pandemics, or the risk of nuclear war, or some combination of those three. AI could, for instance, help develop better solar panels or more energy-efficient materials, combating climate change. It could help develop better vaccines or other pandemic preparedness measures. I'm somewhat less clear on how AI is supposed to help with the risk of nuclear war.

11:15Though it should perhaps be noted that there are other pathways by which AI may exacerbate these risks. As AI advances, it's expected to use more and more energy as models scale up and are deployed more, exacerbating climate change, make the creation of novel virus easier, which is the most concerning situation for pandemics, as engineered pandemics could be much more lethal and transmissible than what we tend to see naturally, and might even destabilize nuclear balances. Basically, AI might allow for accurate enough targeting that it could allow for a first strike to nullify second strike capabilities, providing an incentive for nuclear powers to go first and take out their adversaries' nuclear capabilities.

11:52Reason 7. Belief that AGI is worth it even if it causes human extinction. Some technologists believe that creating AGI would be of cosmic significance outweighing the extinction of humans, often rooted in the idea that this significance is due to AGI's superior intelligence. This belief is often expressed as something along the lines of AI will simply be the next step in evolution after humans, said in an approving manner. For one example of this view, the AI researcher Hugo de Garris has said, These machines might, for whatever reason, wipe out humanity. Wait, there's always that risk. They'll be godlike.

12:23As an AI researcher myself, am I prepared to risk the extinction of the human species for the sake of building an AGI? Because that's what it comes down to. Another alleged example of this view comes from Larry Page, co-founder and ex-CEO of Google. Both Elon Musk and MIT physics professor Max Tegmark have claimed that Larry has dismissed concerns about extinction risk from AI, based on the idea that it would be speciesist to prioritize the survival of humanity over the creation of AGI, Max having made this claim in his book Life 3.0, Elon having made it in various venues, including an interview with Fox News.

12:59All right, and at this point, I will come back to not only NLW, but real, not AI NLW. Daniel's piece concludes with a reminder that this list wasn't meant to be exhaustive, or even try to defend any of these positions, but simply, as he puts it, to say that there's nothing inherently unexplainable about both believing AI poses a substantial risk of causing human extinction and continuing to work on AI research. I guess my small subset of some of these that I will add to the mix for consideration would be summed up as something like confidence or optimism in our capacity to avoid worst case outcomes as they start to become more clear.

13:39Now, of course, many don't believe that this will be possible. They think that there's too much risk, that as the machines become super intelligent, they will try to hide that from humans until it's too late. I tend to find that the presumption in a lot of these conversations is that from this point of it being fairly theoretical to the point of it being very real, we'll somehow sleepwalk through all of the stages and gradations between those two extremes. I simply have a hard time believing that's the case, and I think that we've got a lot more capacity to navigate that difficult line than we might be giving ourselves credit for.

14:11But then again, who knows? I'm still quite clearly trying to absorb as much information about possibilities as I can, and create space to come to ever-evolving conclusions about it, and hopefully these articles and these long reads are helping you do the same. A big thanks once again for you guys for hanging out on a weekend and listening to the show. Until next time, peace.