TIME Asks: Is AI the End of Humanity?

0:00Today on the AI Breakdown, we're reading selections from Time Magazine's cover story, The End of Humanity. The AI Breakdown is a daily video and podcast about the most important news and discussions in AI. Like, subscribe, and share, and go to breakdown.network for more information. Welcome back to the AI Breakdown. Last Sunday, I shared with you guys what I told you was an experiment. It was a reading of a recent article that I thought was illuminating or at least worthy of discussion, and the feedback has been really good. This doesn't necessarily surprise me, as Long Read Sunday has been a long-beloved feature of the other breakdown show, But I think when it comes to AI, there is such a frenzy around new news that having a chance to actually reflect on an essay that someone has taken the time to construct can be really valuable.

0:45Now, Time Magazine has a forthcoming cover story called The End of Humanity, subtitled How Real is the Risk? And this, of course, captures the cultural moment we're in where people are really taking AI risks seriously in a way that we weren't just a couple months ago. So for this week's reading, I'm going to actually read two selections from that issue of Time. The first is called AI is not an arms race. And the second is the Darwinian argument for worrying about AI. Now, for those of you hoping for the regular Saturday Weekly Recap, fear not, it will be coming tomorrow instead. The first selection, AI is not an arms race, was written by Katya Grace.

1:22Grace is the lead researcher at AI Impacts, which is an AI safety project at the non-profit Machine Intelligence Research Institute. Katya writes, The window of what AI can't do seems to be contracting week by week. Machines can now write elegant prose and useful code, ace exams, conjure exquisite art, and predict how proteins will fold. Experts are scared. Last summer, I surveyed more than 550 AI researchers, and nearly half of them thought that if built, high-level machine intelligence would lead to impacts that had at least a 10 % chance of being, quote, extremely bad, e.g. human extinction.

2:01On May 30th, hundreds of AI scientists, along with the CEOs of top AI labs like OpenAI, DeepMind, and Anthropic, signed a statement urging caution on AI. Quote,

2:23Why think that? The simplest argument is that progress in AI could lead to the creation of superhumanly smart artificial people with goals that conflict with humanity's interests and the ability to pursue them autonomously. Think of a species that is to Homo sapiens what Homo sapiens is to chimps. Yet while many fear that AI could mean the end of humanity, some worry that if we, usually used to mean researchers in the West or even researchers in a particular lab or company, don't sprint forward, someone less responsible will. If a safer lab pauses, our future might be in the hands of a more reckless lab.

2:57For example, one in China that doesn't try to avoid substantial risks. This argument analogizes the AI situation to a classic arms race. Let's say I want to beat you in a war. We both spend money to build more weapons, but without anyone gaining a relative advantage. In the end, we've spent a lot of money and gotten nowhere. It might seem crazy, but if one of us doesn't race, we lose. We're trapped. But the AI situation is different in crucial ways. Notably, in the classic arms race, a party could always theoretically get ahead and win. But with AI, the winner may be advanced AI itself. This can make rushing the losing move.

3:32Other game-changing factors in AI include how much one party's safety investments reduce the risk for everyone, whether coming second means a small loss or major disaster, how much the danger rises if additional parties pick up their speed, and how others respond. The real game is more complex than simple models can suggest. In particular, if individual uncoordinated incentives lead to the sort of perverse situation described by an arms race, the winning move, where possible, is to leave the game. And in the real world, we can coordinate our way out of such traps. We can talk to each other. We can make commitments and observe their adherence.

4:08We can lobby governments to regulate and make agreements. With AI, the payoffs for a given player can be different from the payoffs for society as a whole. For most of us, it may not matter much if meta beats Microsoft, but researchers and investors chasing fame and fortune might care much more. Talking about AI as an arms race strengthens the narrative that they need to pursue their interests. The rest of us should be wary of letting them be the ones to decide. A better analogy for AI than an arms race might be in a crowd standing on thin ice, with abundant riches on the far shore. They could all reach them if they step carefully.

4:41But one person thinks, if I sprint, then the ice may break and we'd all fall in. But I bet I can sprint more carefully than Bob, and he might go for it. On AI, we could be in the exact opposite of a race. The best individual action could be to move slowly and cautiously. And collectively, we shouldn't let people throw the world away in a perverse race to destruction, especially when routes to coordinating our escape have scarcely been explored. All right, back to NLW here. The quick thing that I want to note after this piece is that the main way that I'm seeing this arms race analogy happen is less between us or US companies and another global political power and more a capitalist competition between Google, Microsoft, and Meta.

5:25This, for example, it seems like is what Jeffrey Hinton has been talking about as a catalyst for him leaving Google. That when there was an incredible market pressure for Google to move fast, they were being more responsible. Once ChatGPT started to threaten their core search business, they started moving in a much more aggressive way. In other words, this particular arms race isn't like the Cold War. It's viewed as a zero-sum market competition, in which the next decades of business are going to be decided in the next few months or years. There are massively powerful financial incentives aligned around that, and so that might be the point at which there needs to be some sort of intervention.

6:00Second, we read the Darwinian argument for worrying about AI. This piece was written by Dan Hendricks, who is the director of the Center for AI Safety. Now, it was, of course, the Center for AI Safety that coordinated that one-sentence statement signed by so many that was shared earlier this week. Dan writes, A broad coalition of AI experts recently released a brief public statement warning of the risk of extinction from AI. There are many different ways in which AIs might become serious dangers to humanity. And the exact nature of the risks is still debated. But imagine a CEO who acquires an AI assistant.

6:33They begin by giving it simple, low-level assignments, like drafting emails and suggesting purchases. As the AI improves over time, it progressively becomes much better at these things than their employees. So the AI gets promoted. Rather than drafting emails, it now has full control of the inbox. Rather than suggesting purchases, it's eventually allowed to access bank accounts and buy things automatically. At first, the CEO carefully monitors the work, But as months go by without error, the AI receives less oversight and more autonomy in the name of efficiency. It occurs to the CEO that since the AI is so good at these tasks, it should take on a wider range of more open-ended goals.

7:10Design the next model in a product line, plan a new marketing campaign, or exploit security flaws in a competitor's computer systems. The CEO observes how businesses with more restricted use of AIs are falling behind and is further incentivized to hand over more power to the AI with less oversight. Companies that resist these trends don't stand a chance. Eventually, even the CEO's role is largely nominal. The economy is run by autonomous AI corporations, and humanity realizes too late that we've lost control. These same competitive dynamics will apply, not just to companies, but also to nations.

7:42As the autonomy of AIs increases, so will their control over the key decisions that influence society. If this happens, our future will be highly dependent on the nature of these AI agents. The good news is that we have a say in shaping what they will be like. The bad news is that Darwin's laws do too. Though we think of natural selection as a biological phenomena, its principles guide much more, from economies to technologies. The evolutionary biologist Richard Lewontin proposed that natural selection will take hold in any environment where three conditions are present. One, there are differences between individuals.

8:15Two, characteristics are passed on to future generations. And three, the fittest variants propagate more successfully. Consider the content recommendation algorithms used by social media platforms and streaming services. When particularly addictive algorithms hook users, they result in higher engagement and screen time. These more effective algorithms are consequently selected and further fine-tuned, while algorithms that fail to capture attention are discontinued. This fosters the survival of the most addictive dynamic. Platforms that refuse to use addictive methods are simply out-competed by platforms that do, leading to a race to the bottom among competitors that has already caused massive harm to society.

8:51In the biological realm, evolution is a slow process. For humans, it takes nine months to create the next generation, and around 20 years of schooling and parenting to produce fully functional adults. But scientists have observed meaningful evolutionary changes in species with rapid reproduction rates, like fruit flies, in fewer than 10 generations. Unconstrained by biology, AIs could adapt, and therefore evolve, even faster than fruit flies do. There are three reasons this should worry us. The first is that the selection effects make AIs difficult to control. Whereas AI researchers once spoke of designing AIs, they now speak of steering them.

9:26And even our ability to steer is slipping out of our grasp as we let AIs teach themselves and increasingly act in ways that even their creators do not fully understand. In advanced artificial neural networks, we understand the inputs that go into the system, but the output emerges from a black box with a decision-making process largely indecipherable to humans. Second, evolution tends to produce selfish behavior. Amoral competition among AIs may select for undesirable traits. AIs that successfully gain influence and provide economic value will predominate, replacing AIs that act in a more narrow and constrained manner, even if this comes at the cost of lowering guardrails and safety measures.

10:02As an example, most businesses follow laws, but in situations where stealing trade secrets or deceiving regulators is highly lucrative and difficult to detect, a business that engages in such selfish behavior will most likely outperform its more principled competitors. Selfishness doesn't require malice or even sentience. When an AI automates a task and leaves a human jobless, this is selfish behavior without any intent. If competitive pressures continue to drive AI development, we shouldn't be surprised if they act selfishly too. The third reason is that evolutionary pressure will likely ingrain AIs with behaviors that promote self-preservation.

10:35Skeptics of AI risks often ask, couldn't we just turn the AI off? There are a variety of practical challenges here. The AI could be under the control of a different nation or a bad actor. Or AIs could be integrated into vital infrastructure, like power grids or the internet. When embedded into these critical systems, the cost of disabling them may prove too high for us to accept, since we would become dependent on them. AIs could become embedded in our world in ways that we can't easily reverse. But natural selection poses a more fundamental barrier. We will select against AIs that are easy to turn off, and we will come to depend on AIs that we are less likely to turn off.

11:10These strong economic and strategic pressures to adopt the systems that are most effective means that humans are incentivized to cede more and more power to AI systems that cannot be reliably controlled, putting us on a pathway towards being supplanted as the Earth's dominant species. There are no easy, surefire solutions to our predicament. A possible starting point would be to address the remarkable lack of regulation of the AI industry, which currently operates with little oversight, much of the research taking place in the dark. Regulation needs to be done proactively rather than reactively.

11:40If something goes wrong in this domain, we may not get the chance to fix it. The problem, however, is that competition within and between nations pushes against any common-sense safety measures. AI is big business. In 2015, total corporate investment in AI was$12.7 billion. In 2021, this figure had grown to$93.5 billion. dollars. As the race towards power systems quickens, corporations and governments are increasingly incentivized to reach the finish line first. We need research on AI safety to progress as quickly as research on improving AI capabilities. There aren't many market incentives for this, so governments should offer robust funding as soon as possible.

12:18The future of humanity is closely intertwined with the progression of AI. It is therefore a disturbing realization that natural selection may have more sway over it than we do. But as of now, we are still in command. It is time to take this threat seriously. Once we hand over control, we won't get it back. All right, so back to NLW here again for a quick wrap-up. Folks who deal with AI safety and AI risk, people who have designed their careers or their lives around those issues, have been screaming into a void for a very, very long time. The interesting place, however, that we find ourselves now is that alongside this emergence of chat GPT in particular, but AI in general, as a force in broad consumer consciousness, a mainstream force that is already impacting people's jobs and lives.

13:04The discussion of AI risks has moved from the fringes into the mainstream. Time putting the end of humanity, how real is the risk on the cover of their issue this week, is example of exactly that. What's more, we had a really interesting moment earlier this week. A blog reported from a military conference that a colonel in the U.S. Air Force had said that they had run an AI simulation in which an AI drone killed its own operator because they view that operator as getting in the way of its mission. This seeming to be exactly the type of scenario that AI risk people had worried about, it was like lightning and went all over the internet extremely quickly.

13:41Publications around the world and in the very, very mainstream of America published this as though it were real. Fast forward to the next morning, the US Air Force denies that any such simulation ever took place, and finally the colonel comes back up and says actually it was theoretical, but that still shouldn't change the fact that that's the type of thing that could happen. Now, if you listened to my episode about that, you heard that I was pretty frustrated with the colonel and basically argued that this is exactly the type of thing that's likely to diminish and make people take the AI safety conversation less seriously because they are reasonably allergic to and have their guards up against being manipulated and lied to.

14:17However, I had a few comments on that YouTube video that I thought were really insightful. And basically they said, sure, it sucks that this guy was kind of fibbing, even if you don't want to call him a full liar. Let's call it over-dramatizing a theoretical, and maybe not doing so much to clarify that it was just in fact a theoretical. But what had happened was effectively an immune response from humanity. The fact that people cared as much as they did was, these comments argued, a really good sign. I think that's a pretty fair point. My belief continues to be that the hundreds of millions of new people who are paying attention to AI now that weren't six months ago really are still trying to wrap their heads around questions of AI safety and AI risk.

15:00Those questions sit alongside much more prosaic and personal questions of how AI is going to impact their lives. It strikes me that given the risk and safety conversation is no longer on the fringes, it may be time to start to transition the core of that conversation away from things like why we should care, because there seems to be lots of evidence that people do care. And instead, we should probably start to talk about specifics. You see this happening a little bit already. People weren't keen on the idea that open source researchers would have to get licenses for their projects in a way that could just reinforce regulatory capture for big companies like Meta and OpenAI and Microsoft.

15:35But maybe they had less of a problem with the idea that things above the power level of GPT-4 had to go through some sort of licensing process. That's an example of getting into specifics. I think even thornier are going to be questions of how to address the market mechanism which creates such incredible incentives for people to wire every system with AI as fast as they possibly can. It strikes me that it's going to bring up maybe the biggest conversation we've had about the nature of free markets and capitalism in a very, very long time. I certainly don't know what the right answers are, but it does feel to me like we've turned a corner where we can begin to have a different and much more practical conversation.

16:10Anyways guys, that was it for this week's Long Reads Saturday, I guess, instead of Long Reads Sunday. Again, that Time Magazine cover story is called The End of Humanity, How Real is the Risk? And this has been the AI Breakdown. If you found this useful, please like, subscribe, and share. Hit the notification button so you don't miss any episodes. Go check out the podcast or the newsletter version. And thanks for taking the time to listen. Until next time, peace.