AI Researchers Put Risk of Human Extinction from AI at 5%

0:01Today on the AI Breakdown, we're looking at a new extensive survey of AI researchers on whether they think we should be concerned about AI's threat to humanity. Before that on the brief, with an updated keyboard for the first time in 30 years, is Microsoft kicking off the AI PC era? The AI Breakdown is a daily podcast and video about the most important news and discussions in AI. Go to breakdown.network for more information about our Discord, our YouTube, and our newsletter. Welcome back to the AI Breakdown Brief, all the AI headline news you need in around five minutes. Today, we start with a narrative that whether true or not is definitely going to be marketed and sold to you, and that is the idea that we are heading to a new era of the AI PC.

0:45We've seen this sort of rhetoric from a number of different companies. Intel, when they introduced new chips at the end of last year, talked about how we were heading towards this AI PC era. And now just today, we've gotten news that Microsoft is introducing a co-pilot key on its keyboards. The co-pilot key will replace the right control key, and as many are pointing out is the first major change to Windows-based keyboards in around 30 years. Basically, when you're in any program that has a co-pilot AI assistant, you'll be able to press that button to get right to that feature, and in areas that don't have any sort of co-pilot assistance, the key will just bring up Windows Search.

1:22Now, part of the timing of the announcement comes, of course, with the Vegas-based CES starting, as the first laptops and keyboards to feature this key are starting to roll out. Now, on the one hand, this is a small thing. It's just a shortcut to a feature that'll exist in a bunch of different programs, right? But on the other hand, having that button front and center is likely going to push some meaningful portion of people to press that and use those features more often than they might otherwise have. To the extent that one sees things like Microsoft's CoPilot as intro or gateway experiences for people exploring an AI-powered computing era, I think this could be a meaningful update.

1:55Now, beyond that, when it comes to this question of the AI PC year in general, I certainly think that one of the major themes that you are going to see is generative AI models sizing down to be able to run on different types of hardware, and more specifically, more personal types of hardware. It's quite clear that Apple is absolutely racing to that, and many of the open source competitors such as Mistral are clearly innovating in the area of smaller models, which might be able to run locally on device. The types of things that that will enable could be significant. And so what we have here, I think, is one of those marketing memes that actually has some truth in reality.

2:31Moving on today to fundraising news, the big story is that Perplexity has confirmed their latest funding round. The information reported that this was in the works last year, but the company has now confirmed it. They've raised around 74 million new buckos from a variety of investors, which include notably Jeff Bezos. The investment values the company at$520 million post-money. Now for those of you who haven't used perplexity before, it's basically an alternative approach to search. You can ask it something like, what is the breakdown network? And it answers not only with an AI-generated summary, but an AI-generated summary that actually cites its sources.

3:05It also gives you a set of related topics that can lead you down different research rabbit holes. The company reports that perplexity is being used by around 10 million people each month right now, which is obviously a far cry from the number of people who use the dominant search engines like Google. But on an anecdotal and personal level, I know many people in the AI space who have largely shifted their behavior from Google to perplexity and think it has a real shot at disrupting the search experience of the future. Right now, the company makes$5 to$10 million in annual revenue from subscriptions, but is not yet profitable.

3:35Moving over to the regulatory sphere for a moment, OpenAI is seeking to shore up regulatory risk in the European Union by consolidating its European operations under its Ireland subsidiary, OpenAI Ireland Limited. In an email sent to users on December 28th, the company wrote, We have changed the OpenAI entity that provides services such as ChatGPT to EEA and Swiss residents to our Irish entity, OpenAI Ireland Limited. Now, on top of that, their new privacy policy for Europe was also updated. Basically, OpenAI is trying to get out ahead of GDPR privacy compliance issues. And to me, the story is less that specific set of actions and more the fact that I think that in 2024, you're going to see a lot more of this type of scrambling to get into compliance rather than fighting the global legal tide.

4:17Now, speaking of legal battles, Midjourney is of course one of the image generators that is locked in a number of different lawsuits around its approach to training its models. Recently, a list of artists that were allegedly used to train Midjourney has been leaked to much chagrin. The so-called Midjourney Stylus spreadsheet has more than 16 ,000 artist names, as well as listing specific time periods, artistic movements, medium genres, and video game softwares that people can use as part of their prompts. Court documents claim that the list was first published to Discord in February 2022 by MidJourney CEO David Holes, who added the message, I think you're all going to get mind-blown by the style feature.

4:51I should be clear, it's not just genres, it's also artist names. Of course, artists who are against unlicensed AI training on their works are none too happy about this list. In the US, as election season ramps up, progress on any sort of comprehensive AI legislation is very slow, but that doesn't mean that we're not seeing bills. They're just tending to happen at the state level, as every jurisdiction tries to figure out what AI is going to mean for its citizens. In California, a senator has filed a bill prohibiting state agencies from working with unethical AI companies. Basically, the bill requires California's Department of Technology to develop a new set of standards around services that use AI.

5:29In addition, it prohibits the state of California from contracting with any services unless the provider of those services meets those established standards. Another bill introduced simultaneously would establish a California AI research hub designed to provide additional compute resources and other data for academics. Finally, in a move that will be sure to generate extreme emotions in one direction or another, a company in London is set to create a holographic AI version of Elvis Presley for a new stage show called Elvis Evolution. The company Layered Reality writes, A life-size digital Elvis will share his most iconic songs and moves for the very first time on a UK stage.

6:03Elvis fans can look forward to a memory-making experience like no other. Through AI and groundbreaking tech, you'll be able to witness iconic Elvis Presley performances as if you were really there, and celebrate defining moments in Elvis Presley's extraordinary life and career. Like it or not, friends, this is the new reality. However, that is going to do it for today's AI Breakdown Brief. Next up, the main AI breakdown. Quickly, a brief word from today's sponsor. As a listener of this show, I suspect you like to stay up to date on all things AI and tech, which is why you have to check out the chart-topping podcast Web3 with A16Z Crypto.

6:37Produced by venture firm Andreessen Horowitz, Web3 with A16Z is the perfect companion podcast to the AI breakdown. Web3 with A16Z Crypto is your definitive resource for the future of the internet, whether you're interested in the convergence of AI and crypto or simply curious about what's next. If you need a place to start, they recently released an excellent episode with Stanford cryptography professor Dan Bonet and former Google X engineer Ali Yahya in conversation with host Sonal Choksi about the intersection of AI and crypto. From fighting deepfakes and proving humanity to large language models like ChatGPT, they cover it all.

7:11I highly recommend checking it out, especially if you'd like to learn more about how AI and crypto will impact our everyday lives. Beyond crypto and AI, this show is for creators seeking more ways to truly own their work, for business leaders trying to prepare for the future today, and for innovators exploring trending tech topics. Don't miss out. Follow Web3 with A16Z Crypto on Apple Podcasts, Spotify, or your favorite listening app.

8:02artificial intelligence for years. And as the world got acquainted with ChatGPT and MidJourney and other tools like that, that group got consequently louder, trying to let people know that they believed that there was a great concern that people should be paying more attention to. That discourse begat, of course, the six-month pause letter, and then Jeffrey Hinton's famous defection from Google and the media tour that followed. And by the middle of the year, questions of X-risk or extinction risk were everywhere in mainstream media outlets. Surveys throughout the second half of last year have suggested that people are extremely concerned about the negative potential implications of artificial intelligence, and indeed that concern in and of itself and the media surrounding it has inspired a louder counter-reaction in the form of the EACCs or the Effective Accelerationists.

8:46We head into 2024 then in the midst of full-fledged narrative warfare between the folks on the one hand who think AGI will unleash unmeasurable prosperity and opportunity, and the group who, on the other hand, thinks that it could lead to the very end of humanity. Now, that is the context into which a new research paper surveying more than 2 ,700 AI and ML researchers comes into play. The paper was called Thousands of AI Authors on the Future of AI and was led by a team at AI Impacts from Berkeley, California. Now, we're going to dig into many of the findings from this survey, but the first thing that I want to note is how much the big banner headline statistic is serving as a Rorschach test for how people think about AI X-risk in general.

9:27So, for example, the new scientist writes, There's a 5 % chance of AI causing humans to go extinct, say scientists. In the largest survey yet of AI researchers, a majority say there is a non-trivial risk of human extinction due to the possible development of superhuman AI. Basically, the statistic that they're referencing is that around 58 % of the researchers that they surveyed said that there was a 5 % chance of human extinction or another extremely bad AI-related outcome. Now, it's clear that the survey's authors think that this means we should take X-Risk more seriously. Said Katya Grace, who's one of the authors of the paper, it's an important signal that most AI researchers don't find it strongly implausible that advanced AI destroys humanity.

10:06I think this general belief in a non-minuscule risk is much more telling than the exact percentage risk. In other words, what the authors are saying is that a group of researchers saying that there's a 5 % chance that their field causes human extinction is a really worrying note. But then you have Business Insider, whose headline reads, There's only a 5 % chance of AI making humans extinct, a study featuring 2 ,700 AI researchers has found. Now, of course, the implication of that only is that it's something that we shouldn't be concerned about, that this number feels lower than one might have expected.

10:37The lead of the piece seems to reinforce that perspective. Quote, Over the last year, we've heard a lot about the risk of AI destroying humanity. Industry leaders in AI heavyweights said the rapid development of the technology could have catastrophic consequences for the world. But, while most AI researchers recognize the possibility of existential threats, they don't think that dramatic outcomes are very likely. Now, I think that if you go to the actual paper itself, what it shows is a meaningful amount of concern, but also a lot of ambivalence in not knowing. One gets the sense that people are trying to rapidly update their mental models because things are changing so quickly.

11:10For example, one of the things that they asked was, if science continues undisrupted, what are the chances that unaided machines outperform humans in every possible task? The chance of that was estimated to be 10 % by 2027 and 50 % by 2047. That 2047 estimate was 13 years earlier than the estimate that was reached in a similar survey that was conducted just a year earlier. In other words, a single year of progress in AI has had this group of AI researchers pull forward their estimate of when machines could outperform humans in every possible task by 13 years. The chance of all human occupations becoming fully automatable was forecast to reach 50 % by 2116, which was up from 2164 in the 2022 survey, a 38-year increase.

11:55Now again, to my mind, what matters is not the specific year, but more the fact that the evidence in front of their eyes is forcing people to grapple with AI moving much more quickly than they had previously imagined. Now in terms of the ambivalence, the way that the paper frames it is, most respondents expressed substantial uncertainty about the long-term value of AI progress. While 68.3 % thought good outcomes from superhuman AI are more likely than bad, of these net optimists, 48 % gave at least a 5 % chance of extremely bad outcomes such as human extinction, and 59 % of net pessimists gave 5 % or more to extremely good outcomes.

12:30Between 37.8 % and 51.4 % of respondents gave at least a 10 % chance of advanced AI leading to outcomes as bad as human extinction. More than half suggested that substantial or extreme concern is warranted about six different AI-related scenarios, including spread of false information, authoritarian population control, and worsened inequality. There was disagreement about whether faster or slower AI progress would be better for the future of humanity. However, and this is the upshot of the whole paper, there was broad agreement that research aimed at minimizing potential risks from AI systems ought to be prioritized more.

13:02Now, the AI Safety Memes account on Twitter pulled out a number of different statistics that suggest that the Business Insider glib interpretation may not be exactly what the paper really conveys. For example, the paper found over 95 % of the researchers were concerned about dangerous groups using AI to make powerful tools like engineered viruses. Over 95 % are concerned about AI being used to manipulate large-scale public opinion. Over 95 % are concerned about AI making it easier to spread false information. Over 90 % are concerned about authoritarian rulers using AI to control their populations.

13:32Over 90 % are concerned about AIs worsening economic inequality Over 90 % are concerned about bias, such as AIs discriminating by gender or race Over 80 % are concerned about a powerful AI having its goals not set right, causing a catastrophe For example, developing and using powerful weapons Over 80 % are concerned about people interacting with other humans less because they're spending more time with AIs Over 80 % are concerned about near-full automation of labor, leaving most people economically powerless Relatedly, over 70 % are concerned about near-full automation of labor making people struggle to find meaning in their lives.

14:04And only 20 % say it's likely that we'll be able to understand what AIs are really thinking by 2028. Now again, what I argued is that the upshot of the research is that these researchers by and large want to focus on AI safety research in a bigger way. Statistics that validate that, 86 % say the AI alignment problem is important versus only 14 % that say it's unimportant, which is a 7 to 1 ratio. and 70 % want to prioritize AI safety research more versus just 7 % who want to prioritize it less. In other words, by 10 to 1, these researchers think that AI safety research should be prioritized more.

14:36Now, this account also pointed out that although that headline reported statistic of 58 % of researchers finding a 5 % chance of extinction, that the mean across all the researchers for their p-doom on the question, what probability do you put on future AI advances causing human extinction or similarly permanent and severe disempowerment of the human species, was actually 16.2%, which, as they point out, is basically the same odds as Russian roulette. So where should this study leave us? Well, I think what I've thought for basically the entirety of last year as well, which is that this is a classic type of modern discussion that is dominated by extreme voices on either side.

15:13That domination by extreme voices is aided and abetted by both the media infrastructure as well as social media algorithms, which promote more extreme views versus more nuanced and moderate takes. But I think that to the extent that one is looking for optimism, there is clearly a lot of common sense space in the middle. I think by and large that people feel that technology should work for us. It should be on behalf of us. And to the extent that there are risks that technology is not in the service of humanity, that's something worth really considering. Now where it gets tricky, of course, is that the market incentives to race ever faster forward are not slowing down anytime soon.

15:45And so the question becomes, if markets aren't creating enough of an incentive to prioritize safety research or just appropriate speed of development, what are the countervailing forces? There are a lot of people who are uncomfortable in general with state intervention who feel like the state is the only solution in this case, and that's going to be a great big discussion throughout 2024 and beyond. But no matter what, even if you disagree with these results, even if people come up with problems of the methodology, in general I am for having more information for more different types of people, and especially those with experience in the field, so I welcome these numbers as one more piece of evidence in a very important discussion.

16:20However, that is going to do it for today's AI Breakdown. Let me know what you think about all this. Hit me in the comments. Come join us on the AI Breakdown Discord, bit.ly slash AI Breakdown. And until next time, peace.