Regulatory Capture? Meta AI Chief Accuses OpenAI, Anthropic of Stoking Fears

0:01Today on the AI Breakdown, we're looking at the increasingly contentious debate between the pro-open-source AI community on the one hand, and the AI safety and extinction risk community on the other. The AI Breakdown is a daily podcast and video about the most important news and discussions in AI. Go to breakdown.network for more information about our YouTube channel, our Discord, and our newsletter.

0:24Welcome back to the AI Breakdown. A quick reminder that I am currently off on a very short vacation for my 10th wedding anniversary, so this is pre-recorded. it may not have the most up-to-date information, and that's also why you're not getting a brief as normal. However, today's topic is still very topical and very much of this week. The week kicked off on Monday with President Biden's executive order on artificial intelligence, and then in the middle of the week we had the AI Safety Summit in the UK, which was meant to really put the issues of AI safety and in particular, the biggest concerns around the most dangerous scenarios of AI front and center in the global political discourse.

1:03Now, as that was happening, tensions between people with different views of AI safety, the role of open source in the industry, and what's going on in those policy spheres spilled out into the open in a major, major way. As Nathan Binatch summed up, it's a big week for AI safety and the future course of our industry. So we're going to look at two instances this week of high-profile members of the AI community basically arguing that the major tech companies and AI labs are, as Lior at AlphaSignalAI put it, spreading fear of AI to gain control over the industry. So the first of these conversations, and a person who's very much at the center of them, is Chief AI Scientist at Meta and NYU Professor Jan LeCun.

1:45Jan is of course a 2018 Turing Award winner alongside Jeffrey Hinton and Yoshua Bengio, but has basically the opposite perspective as them when it comes to AI safety and AI risks. Back at the end of September, as news came out that the AI Safety Summit was meant to focus almost entirely on these existential risks, Jan tweeted, The UK Prime Minister has caught the existential fatalistic risk from AI delusion disease or afraid. Let's hope he doesn't give it to other heads of state before they get the vaccine. Now at the beginning of the week, Max Tegmark from the Future of Life Institute, who was one of the main organizers of that six-month pause letter, quote tweeted that late September post and said, Jan, I'd love to hear you make arguments rather than acronyms.

2:27Thanks to UK Prime Minister Rishi Sunak and EU Commission President Ursula von der Leyen for realizing that AIX risk arguments from Turing, Hinton, Bengio, Russell, Altman, Hassabis, and Amode can't be refuted with snark and corporate lobbying alone. Now, that statement absolutely set Jan off and led to an extremely notable post and frankly accusation. Jan responds, Altman, Hassabis, and Amode are the ones doing massive corporate lobbying at the moment. They are the ones who are attempting to perform a regulatory capture of the AI industry. You, Jeff, and Yoshua are giving ammunition to those who are lobbying for a ban on open AI R &D.

3:04If your fear-mongering campaigns succeed, they will inevitably result in what you and I would identify as a catastrophe. A small number of companies will control AI. The vast majority of our academic colleagues are massively in favor of open AI R &D. Very few believe in the doomsday scenarios you have promoted. You, Yahshua, Jeff, and Stuart are the singular but vocal exceptions. Like many, I very much support open AI platforms because I believe in a combination of forces. People's creativity, democracy, market forces, and product regulations. I also know that producing AI systems that are safe and under our control is possible.

3:40I've made concrete proposals to that effect. This will all drive people to do the right thing. You write as if AI is just happening, as if it were some natural phenomenon beyond our control. But it's not. It's making progress because of individual people that you and I know. We, and they, have agency in building the right things. Asking for regulation of R &D, as opposed to product deployment, implicitly assumes that these people and the organization they work for are incompetent, reckless, self-destructive, or evil. They are not. I have made lots of arguments that the doomsday scenarios you are so afraid of are preposterous.

4:12I'm not going to repeat them here. But the main point is that if powerful AI systems are driven by objectives, which include guardrails, they will be safe and controllable because we set those guardrails and objectives. Now about open source. Your campaign is going to have the exact opposite effect of what you seek. In a future where AI systems are poised to constitute the repository of all human knowledge and culture, we need the platforms to be open source and freely available so that everyone can contribute to them. Openness is the only way to make AI platforms reflect the entirety of human knowledge and culture.

4:42This requires that contributions to those platforms be crowdsourced, a bit like Wikipedia. That won't work unless the platforms are open. The alternative, which will inevitably happen if open-source AI is regulated out of existence, is that a small number of companies from the west coast of the US and China will control AI platform and hence control people's entire digital diet. What does that mean for democracy? What does this mean for cultural diversity? This is what keeps me up at night. Tegmark responded, Thanks, Jan, for the long and thoughtful reply. You're conflating two separate questions.

5:14One, is superintelligence an X-risk? Two, is open-source and good? You criticized Rishi Sunak over one. I replied, but you still gave no arguments for why it's not an X-risk, just repeated the trivially falsifiable claim that almost no AI researchers are worried about this. Instead, you changed the topic to two. It's obviously great to have lots of AI open-sourced. We love open-source here at MIT. And it's obviously stupid to open-source absolutely everything, like bioweapon-optimizing code. So there's an interesting discussion to be had about where to draw the line. If you support open sourcing superintelligence, you presumably have an argument for one about why it's completely safe, and I'm still waiting to hear this argument of yours.

5:52And now, a word from today's sponsor. Are you interested in how two top-of-mind trends, AI and crypto, can work together? If so, I have the perfect podcast recommendation for you. Web 3 with A16Z Crypto, the chart-topping show brought to you by venture firm Andreessen Horowitz. Web 3 with A16Z Crypto is your definitive resource for the future of the internet, whether you're already building in these spaces or simply curious about what's next. If you need a place to start, they recently released an excellent episode with Stanford cryptography professor Dan Bonet and former Google Xer Ali Yahya in conversation with host Sonal Choksi about the intersection of AI and crypto.

6:29From fighting deepfakes and proving humanity to large language models like ChatGPT, they cover it all. I highly recommend checking it out, especially if you'd like to learn more about how AI and crypto will impact our everyday lives. Beyond Crypto and AI, this show is for creators seeking more ways to truly own their work, for business leaders trying to prepare for the future today, and for innovators exploring trending tech topics. So go ahead, listen to Web3 with A16Z Crypto wherever you get your podcasts. Now, I will say for folks who have read this before, I apologize for basically just reading this discussion, but A, a lot of you won't have read it yet, and B, given that this is Twitter, it's dense and hard to piece through and surrounded by other comments, and so hopefully this is useful because I do think that it is a marker of this particular platform that we actually get to see these conversations out in the open.

7:17Anyways, Jan responds to Max again. He says the reason that number two, which by the way was Max's question is open sourcing good, is in question and open source AI R &D is threatened is one, the question of superintelligence. Those questions are not disconnected. If you ask, give me arguments for why turbojets and rocket engines won't kill people, I could respond by pointing to a dozen treatises on how to design and build reliable turbojets and rocket engines. Those treatises don't exist yet for superhuman AI, because superhuman AI has not been invented yet. And people who think scaling up autoregressive LLMs will get us there are just wrong.

7:50So the question becomes, what will superhuman AI systems look like? What is the blueprint? My answer to this, which you've heard many times, is that future AI systems, just like intelligent animals, will be objective-driven. Whatever outputs or actions they produce will have to satisfy a set of objectives, some of which will be guardrails that guarantee their safety and controllability. We will set those objectives and guardrails, and AI systems will not be able to violate them by construction. These systems will stay under our control because their primary objective will be to be subservient to us.

8:19Now, we are not going to build super-intelligent, objective-driven AI systems right away. We are going to build very simple systems first, and make sure the guardrails we put in place guarantee their safety. Then we'll work our way up, building smarter and smarter systems, adjusting the objectives and guardrails as we go. It will be like turbojet and rocket engine reliability, an iterative process of refinement and careful engineering. Much of the X-risk hyperventilating is due to two fallacies. One, just because a system is intelligent, it will want to take over. Two, the minute we turn on a slightly flawed superintelligent system, it will take over and destroy humanity.

8:52Number one is simply false. The desire to dominate is not linked to intelligence. That's not even true within the human species. The people who seem most motivated to dominate are not the smartest among us. To want to dominate, you need a hardwired drive to do so. Only a few social species have such a thing, including humans, baboons, chimps, wolves, etc. Non-social species like orangutans don't. But if you are still scared, we can hardwire objectives into AI systems to make them submissive to humans within limits. Number two reveals a complete misunderstanding of how new technologies are developed and deployed.

9:22The same way we don't build a giant turbojet airliner and put 1 ,000 passengers in it in the first trial, we won't build a super-intelligent AI prototype and let it loose on the first try. We'll build cat-level AI first and figure out what guardrails are necessary as we work our way up the intelligence level. Now, there are so many places that we could take this conversation at this point. One of the really interesting dimensions that I think has to be included as part of this is the impact of market forces on the whole conversation. I think one of the things that makes this smartest among the AI safety and X-risk folks that I know most concerned is that we're not just operating in a scientific vacuum.

9:57Instead, we are operating in a market context, which rewards speed over absolutely everything else. But that's not actually the biggest point of the conversation here, and there is so much more to get into, so let's continue. Now, of course, the big place this started was that Jan was arguing very vociferously that the big AI labs are lobbying the U.S. political establishment against open systems and for regulation in order to perform a sort of regulatory capture where their leads are preserved and they have an advantage because of the cost of compliance. Epic's Tim Sweeney agreed with this argument.

10:30He quote-tweeted Jan and said, took Apple and Google 20 years to stoop to asking regulators to endorse their monopolies, quote-unquote, for safety. Now when Google DeepMind CEO Demis Hassabis was asked about this on CNBC, he clapped back. Hassabis said, I pretty much disagree with most of these comments from Jan. I think the way that we think about it is there's probably three buckets or risks that we need to worry about. There's sort of near-term harms, things like misinformation, deepfakes, these kinds of things, bias and fairness in the systems that we need to deal with. Then there's sort of the misuse of AI by bad actors repurposing technology, general-purpose technology for bad ends that they were not intended for.

11:05That's a question about proliferation of these systems and access to these systems. So we have to think about that. And then finally, I think about the more longer-term risk, which is technically AGI or artificial general intelligence risk. So the risk of themselves making sure they're controllable. What value do you want to put in them to have these goals and make sure they stick to them? Now, Hassabis said that it was important to start the conversation about how to regulate AGI now, even though we're not there, because as he put it, I don't think we want to be doing this on the eve of some of these dangerous things happening.

11:34Now, as I told you right up front, it wasn't just Jan making these comments this week either. Insider ran a piece about Andrew Ng called, Google Brain co-founder says big tech companies are inflating fears about the risks of AI wiping out humanity because they want to dominate the market. Basically, in a conversation with the Australian Financial Review, Ng said that the biggest tech companies were hoping to trigger strict regulation with the, quote, bad idea that AI could make us go extinct. He continued, There are definitely large tech companies that would rather not have to try to compete with open source, so they're creating fear of AI leading to human extinction.

12:05It's been a weapon for lobbyists to argue for legislation that would be very damaging to the open source community. He expanded upon these points in a tweet in a blog post. He wrote, My greatest fear for the future of AI is if overhyped risks such as human extinction lets tech lobbyists get enacted stifling regulations that suppress open source and crush innovation. In a blog post he wrote, In recent months I sought out people concerned about the risk that AI might cause human extinction. I wanted to find out how they thought it could happen. They worried about things like a bad actor using AI to create a bioweapon, or an AI system inadvertently driving humans to extinction, just as humans have driven other species to extinction through lack of awareness that our actions could have that effect.

12:43When I try to evaluate how realistic these arguments are, I find them frustratingly vague and nonspecific. They boil down to, it could happen. Trying to prove it couldn't is akin to proving a negative. I can't prove that AI won't drive humans to extinction any more than I can prove that radio waves emitted from Earth won't lead space aliens to find us and wipe us out. Such overblown fears are already causing harm. High school students who take courses designed by Kira Learning, an AI fund portfolio company that focuses on grade school education, have said that they are apprehensive about AI because they've heard it might lead to human extinction, and they don't want to be a part of that.

13:14Are we scaring students away from careers that would be great for them and great for society? I don't doubt that many people who share such worries are sincere, but others have a significant financial incentive to spread fear. Individuals can gain attention, which can lead to speaking fees or other revenue. Non-profit organizations can raise funds to combat the phantoms that they've conjured. Legislators can boost campaign contributions by acting tough on tech companies. As he sums up, I firmly believe that AI has the potential to help people lead longer, healthier, more fulfilling lives. One of the few things that can stop it is regulators passing ill-advised laws that impede progress.

13:45Some lobbyists for large companies, some of which would prefer not to have to compete with open source, are trying to convince policymakers that AI is so dangerous, governments should require licenses for large AI models. If enacted, such regulation would impede open source development and dramatically slow down innovation. Now this got its own set of responses, including from Jeffrey Hinton. Hinton wrote, Andrew Ng is claiming that the idea that AI could make us extinct is a big tech conspiracy. A data point that does not fit this conspiracy theory is that I left Google so I could speak freely about the existential threat.

14:15Andrew responded, I didn't say it's a conspiracy, but I think overhyped fears about AI leading to human extinction are causing real harm. He then pointed to the same examples he used in his letter. young students being discouraged, and hype about harm being used to promote bad regulation. He concludes, I know you're sincere in your concerns about AI and human extinction. I just respectfully disagree with you on extinction risk, and I also think these arguments, sincere though they may be in your case, do more harm than good. Now, as we try to wrap up, there are a couple things that I think are worth noting.

14:45First of all, I think that these conversations are a lot harder in the abstract than they are in the specific. What I mean by that is that when all of these debates started, we didn't yet have President Biden's executive order. Once we did get that order, we could get specific about different points of disagreement. For example, it actually seems like the executive order is going to be a lot worse for the big AI labs, who have been accused of fear-mongering to get regulatory capture, than it will be for open-source developers. But at the same time, there is still a concern among the open-source community that the very core approach to regulation, which is based on how much compute is used or the size of the models, is not the right way to go about things.

15:25Clem from Hugging Face wrote, In my opinion, compute or model size thresholds for AI building would be like counting the lines of code for software building. Regulation based on this will most likely be easily fooled, create hurdles and worries for companies to compete on bigger models, so concentration of power, and slow down innovation without solving any of the safety challenges. What about focusing on use cases and industry-specific evaluations of risks like we do for software. Jan, for example, agreed with that, saying, yes, regulate end product deployments. Don't regulate R &D with an arbitrary computer or model size threshold.

15:57Now, my point is that one still might disagree with this, but there's a lot more profitable conversation to be had with that specific conversation than the much more general one. Anyways, it is a fascinating time in the evolution of this space and in the evolution of public attitudes around it. As I mentioned in my first episode about the Biden executive order a few days ago, it is very clear that this was not the end point, but the starting gun when it comes to these debates rising to the biggest spheres of the public conversation. And so, of course, what that means for you guys is now is a good time to get involved.

16:27If you have strong feelings about this, it's worth sharing them because a heck of a lot of people on both sides of this issue are going to claim to be doing things on your behalf, whether you express those feelings or not. So if you've got them, you might as well actually share them. For us, that is going to wrap up for today's episode. Till next time. Peace. Thank you.