SB1047: The World's Most Important and Problematic AI Policy Debate

0:00Today on the AI Daily Brief, we are talking about SB1047, the world's most important and problematic AI policy debate. The AI Daily Brief is a daily podcast and video about the most important news and discussions in AI. To join the conversation, follow the Discord link in our show notes.

0:23Hello, friends. Quick note before we dive into today's show. As you will see, it is much longer than our normal episodes. And because of that, there will only be this main discussion. We will not do the headlines, but we will be back tomorrow with our normal format. But for now, let's dig into SB 1047. Welcome back to the AI Daily Brief. Perhaps the most controversial discussion in the artificial intelligence space right now is the discussion around California's SB 1047. On the one side, you have those who say that this is common sense, light touch regulation, exactly the type of guardrails the industry needs to evolve in a way that is good for everyone and not just good for the companies driving AI.

0:59On the other side, you have those who see this as having the potential to absolutely chill AI development and innovation, a fraught, if some believe, well-intentioned attempt to regulate that will do nothing but create more legal landmines that drive entrepreneurs to simply build less. Now, as you might guess from the fact that I do this daily AI podcast and I have an AI enablement company and super intelligent, I find myself more aligned with one side than the other. However, for the purposes of this episode, I'm going to try to represent each side faithfully with the idea that all of you are smart enough to make up your own minds around what you think about this.

1:31However, I'm also going to try to explain why the discourse around this seems so fraught. If you've spent any time reading about this or seeing how people talk about it, it feels more tense, more fraught than you might expect from a state legislature bill. Perhaps that is just the current political climate in America in general, but I think it actually might be reflective of something more. In short, there is an underlying and more fundamental disagreement about the risks that we face from AI that makes it very hard to find common ground with this discussion. So let's first do a little bit of background.

2:02Here's how the Wall Street Journal sums it up. SB 1047 requires that developers of large AI models conduct safety tests to reduce the risk of catastrophic harm from their technology. It defines catastrophic harm as cyber attacks that cause at least 500 million in damage or mass casualties. Developers must also have a kill switch that ensures that AI can be shut down by a human. The bill has a computing power threshold defined in terms of how much it costs to train. That level is set at around$100 million. In terms of where the bill is, SB 1047 has been passed by the California Senate and two of the state's assembly committees.

2:32The Democrat-controlled legislature in California is now back in session, and so the push to get changes made has gotten frantic now, as the bill could be on its last steps before being sent to Governor Gavin Newsom for signing a veto. The bill was drafted by Scott Wiener, who is a Democratic state senator from San Francisco. He was my representative when I lived there back in the teens, and Wiener has tried to argue strongly that he is not anti-innovation and that he is not anti-oversource. He argues that the bill is not overly prescriptive. In fact, that it only codifies safety standards that the industry has already set on its own.

3:01GDSuper.com gives a few additional details that help understand the bill. First, as we discussed, the bill would regulate AI models based on the amount of money and computing power used to train them. They point out, however, that the bill also seeks to regulate covered model derivatives, which include copies of covered models, modified versions of covered models, and covered models combined with other software. The bill would also set up a new state agency called the Frontier Model Division. Next, there would be a variety of compliance obligations for developers of covered models. Quote, before training a covered model, a developer would have to implement administrative, technical, and physical cybersecurity protections, the capability to promptly enact a full shutdown of the AI, and a comprehensive written safety or security protocol the developer would have to provide to the state.

3:38Again, that's before training the model. Now, before releasing the model, the developer would have to assess whether the model is reasonably capable of causing or enabling critical harm, implement reasonable safeguards to prevent the model and its derivatives from causing or enabling critical harm, ensure to the extent reasonably possible that the covered model's actions and the actions of covered model derivatives can be accurately and reliably attributed to them. You might already beginning to see some of the problems for people who are against this legislation. There's a lot of words like reasonable, reasonable safeguards, reasonably capable, that create a ton of legal ambiguity.

4:08Legal ambiguity, of course, leads to expensive legal fees. In addition to the compliance rules around the developers, the legislation would also regulate operators of computing clusters that were used to train covered models. Those operators would have to obtain a prospective customer's identifying information and business purpose, assess whether a prospective customer intends to use the computing cluster to train a covered model, retain records of customers' IP addresses used, and implement the capability to promptly enact a full shutdown of any resources used to train or operate models. Basically, it would give operators of the computing clusters extreme surveillance requirements and significant power and responsibility over their customers.

4:41Next, there would be significant labor protections for employees of the developers, basically protecting them if they have to become whistleblowers. This, I will say, is probably the least controversial part of this whole bill. So let's talk now about the pro position for this. We'll get into some specifics, but here's how I might sum up some of their points. First, there is a general belief that industry can't be trusted to regulate itself. There is an argument that only big models are implicated, and so the concerns around little tech and open source are overblown. There is an argument that this is light touch relative to the extreme AI existential risks, a point that we'll come back to.

5:16There is an argument that the rules have the flexibility to be applied in ways that accommodate the reality as the reality becomes clear over time. In other words, that this doesn't lock us into overly prescriptive policy. And there's an argument that this largely reflects voluntary commitments that have already been made. And so why the logic follows, wouldn't companies who already have made those voluntary commitments be willing to enshrine them in the letter of the law. Sonny Gandhi, the VP of political affairs at Encode Justice, wrote an op-ed called California Shouldn't Buy Big Tech Talking Points That AI Regulation Will Hurt Innovation.

5:46He writes, You might think this is a debate between big tech and slow government. You might think this is a debate between those who would protect technological innovation and those who would regulate it away. Or you might think that this is a debate that decides if AI development will stay or leave California. These arguments could not be more wrong. SB 1047 is about ensuring that the most powerful AI models, those with the potential to cause catastrophic harm, are developed responsibly. We're talking about AI systems that could potentially create bioweapons, crash critical infrastructure, or engineer damage on a societal scale.

6:13These aren't science fiction scenarios. They're real possibilities that demand immediate attention. He goes on to argue that the arguments from the tech industry that say that this will drive innovation out of California is simply not true because the bill only applies to companies spending hundreds of millions. He calls this fear-mongering. He also explicitly calls out the effective accelerationist movement. He calls them tech zealots that, quote, dream of a world where AI develops unchecked, regardless of the consequences. They list concepts like sustainability, social responsibility, and ethics as enemies to be vanquished, and feverishly dream of a world where technology replaces humans.

6:43Sonny argues that he is mischaracterized as being anti-technology. He says he rejects that completely. Quote, I'm a digital native that sees AI's immense potential to solve global challenges. I'm deeply optimistic about the future of technology, but I also understand the need for guardrails. Now I want you to put a pin in that section about the effective accelerationists and the characterization of them as tech zealots, and that line that these aren't science fiction scenarios. Recently, some of the loudest supporters of the PAWS movement, Turing Award winners Yoshua Bengio and Jeffrey Hinton, have also written a letter expressing their support for the bill.

7:13That letter was also signed by Lawrence Lessig and Stuart Russell, and they described the bill as, quote, the bare minimum for effective regulation of this technology. They say there are fewer regulations on AI systems that could pose catastrophic risks than on sandwich shops or hairdressers. They write, as some of the experts who understand these systems most, we can say confidently that these risks are probable and significant enough to make their safety testing and common sense precautions necessary. In an email to Time magazine, Ben Gio reinforced the argument that industry can't be trusted to regulate itself.

7:39He wrote, I worry that technology companies will not solve these significant risks on their own while locked in their race for market share and profit maximization. That's why we need some rules for those who are at the frontier of this race. That was of course why Jeffrey Hinton said that he left Google last year, namely that he was concerned that the competition around AI was making them act in ways that he no longer agreed with. The letter says, Relative to the scale of the risks we are facing, this is a remarkably light-touch piece of legislation. Now, the Pause AI movement goes even farther than the four authors of this open letter, saying the language that Benji O.

8:09Hinton, Lawrence, and Russell use should tell you. SB 1047 is better than nothing, but it's not enough to keep us safe. We need to globally stop AI companies from gambling with our future. We need an international pause. By the way, they quote tweeted Dan Hendricks, the director of the Center for AI Safety, who was integrally involved in writing this legislation, which, as we will come to, is part of the challenge with having any sort of common ground discussion of it. Now let's talk about the con side of this argument. A huge variety of industry organizations, led by venture capital firms and the tech companies themselves, have organized an effort to stop SB 1047.

8:41The arguments against it are very diverse. There are competitiveness concerns, particularly with China, as well as in terms of keeping California as a bastion of innovation versus seeing startups locate elsewhere. Some see this as a boon for regulatory capture, with only the biggest companies able to comply. Some point out that this doesn't address immediate concerns, and that it's not based on specific immediate risks. Some argue that it's too early to know what the actual risks are. People point to arbitrary lines and concerns around developer liability, and to the fact that this creates significant new governmental power.

9:11StopSB1047.com has a collection of many of these arguments. The website lists a set of unintended consequences of SB1047. They write that it could put a chilling effect on AI investment and development in California, that it penalizes developers for outcomes that are obscure and requires testing that doesn't exist yet. They write that the bill's vague definitions, coupled with stiff legal liability, create substantial uncertainty and economic risk for AI developers and business owners. They argue that it could unintentionally make AI systems less secure by driving research into the shadows, that the bill creates systemic disadvantages for open-source and startup developers, and that it disincentivizes AI research and innovation in the US, giving other countries, notably China, the opportunity to overtake the US in AI development.

9:48Andreessen Horowitz, who has been one of the loudest voices in this fight, wrote a 14-page response to Senator Wiener in response to a letter that he had written regarding A16Z's public comments about SB 1047. They write, Your July 1st, 2024 letter made certain legal claims that are inconsistent with the plain reading of the proposed bill in its current form. Notably, they point to, SB 1047 applies to startups because of its arbitrary and shifting thresholds. SB 1047's criminal penalties will deter innovation. SB 1047 creates new civil liability. 1047 will hurt California's economy. 1047 is troublingly vague.

10:20SB 1047's kill switch requirement imposes executive burdens on new AI models, and SB 1047 privileges closed-source models over their open-source counterparts. I'm not going to read all 14 pages of this, but you can hear these themes coming through over and over again. I think from an outsider's perspective, some of these specific discrete concerns that ring most clear are around the vagueness of the language. Anyone who has spent any time around regulation knows that the more vague things are defined, the more interpretation there is, which means more power in the hands of regulators and lawyers and higher burdens and compliance costs on the subject of the regulations.

10:52And this is true even in the case of well-intentioned regulation. And indeed, well-intentioned regulation is exactly what Fei-Fei Li, who is sometimes called the godmother of AI as opposed to Jeffrey Hinton's godfather of AI, says the bill is, even though she thinks it's wrong. Her main arguments in an open letter are that 1. SB 1047 will unduly punish developers and stifle innovation. She writes, In the event of misuse of an AI model, SB 1047 holds liable the party responsible and the original developer of that model. It is impossible for each AI developer, particularly budding coders and entrepreneurs, to predict every possible use of their model.

11:24SB 1047 will force developers to pull back and act offensively. Her second argument is that SB 1047 will shackle open-source development. She argues specifically that the kill switch requirement would devastate the open-source community. Third, she writes that the bill would cripple public sector and academic AI research. She writes, open-source development is important in the private sector but vital to academia, which cannot advance without collaboration and access to model data. She also says, Most alarmingly, this bill does not address the potential harms of AI advancement, including bias and deepfakes.

11:52And again, I'm sorry to keep saying put a pin in this, but this really will become part of the central tension of this entire conversation. Ultimately, she writes, In various conversations with President Biden over the past year, I have expressed the need for a moonshot mentality to spur our country's AI education, research, and development. SB 1047, however, is overly and arbitrarily restrictive, and will not only chill California's AI ecosystem, but also have troubling downstream implications for AI across the nation. She argues that she is not against AI governance, but that we should, quote, work together to craft AI legislation that will truly build the technology-enabled human-centered society of tomorrow.

12:25A few other con arguments just before we wrap up. Today's episode is brought to you by Superintelligent, the platform that helps teams maximize AI. Super is, of course, the platform that we've been building that pairs fun, fast, practical video tutorials with step-by-step instructions to get you actually using AI. And from there, unlocks information about hundreds of use cases that show you how people are actually getting value out of AI right now. Now, we have just launched Super for Teams. This is a new add-on experience that allows teams to share more information about what AI they're using, how it's working, and how to get more value out of it.

13:02Whether your company is 25 or 2 ,500, 100, Superintelligent is going to be the best platform for unlocking information about how to get the most out of AI right now. If you'd like to learn more about the Superintelligent Teams offering, go to besuper.ai slash partner and send us a note so that our team can get right back to you. Once again, that's besuper.ai slash partner. Today's episode is brought to you by Venice. Venice is a private, uncensored generative AI app. It accesses open source models to enable text, image and code generation without the fear of being spied on or having your data exploited.

13:37Discuss anything with Venice without concern about it being monitored, sold, or given to advertisers and governments. Venice is different because your conversations and creations are kept securely within the browser, never stored or accessible by Venice. Unlike other AI apps, Venice won't tell you what's okay to say or not. Venice won't patronize you. It simply provides direct access to machine intelligence. No topics are off-limits, no ideas are taboo. With Venice, you're in control of the AI as you should be. Pro subscriptions are available for$49 a year or$8 per month. AI Daily Brief listeners receive a 20 % discount on Venice Pro.

14:09Visit venice.ai slash nlw and enter the discount code nlwdailybrief. That's nlwdailybrief, all one word. The first notably comes from Anthropic. Anthropic is of the big tech labs, the one most associated with AI safety and the effective altruism movement, which is part of the origination story of this bill. In short, Anthropic does not support the bill. In a letter they wrote, ensuring the safe development of AI technology is a worthy goal, but the current version of SB 1047 has substantial drawbacks that harm its safety aspects and could blunt America's competitive edge in AI development. Our letter proposes to refocus the bill on frontier AI safety and away from approaches that aren't adaptable enough for a rapidly evolving technology.

14:49And then there's the academic community. Russell Wald, the deputy director at the Stanford Institute for Human-Centered AI says, SB 1047 does not offer the regulation California needs, and instead attempts to combat unsubstantiated X-risk while not focusing on the immediate harms of AI. And this is really where we start to get into the rub of this. Another open letter comes from a group of University of California students and faculty from seven campuses. They write, as academics, we unequivocally oppose this bill for concern that it may seriously harm both the research and educational objectives of California universities in AI.

15:21We call on our representatives to seriously consider these harms when weighing whether to pass this legislation. Industry actors are not the only ones who vociferously oppose this bill. But more than any other open letter, this note calls out the point that the entire bill at core is predicated on a belief in a very particular set of AI ex-risk. They write, As experts in artificial intelligence, machine learning, and natural language processing, we must stress that both the existence of meaningful AI risks and the proposed methods of assessing model risk alluded to in SB 1047 are deeply dubious.

15:55There is no consensus in the scientific community on whether and how language models or other frontier AI systems may pose a threat to the public. It is questionable whether a language model can act as an information-generating system that is any more empowering to a nefarious actor than a search engine. Any would-be bioweapon developer can just as easily do Google searches and scour textbooks to find any of the kinds of chemical, biological, radiological, or nuclear weapon harms that have so far been demonstrated. Furthermore, we do not believe that existential or catastrophic risks from AI development, the primary motivator for most in the AI safety community, are sufficiently evidence-based to guide policy.

16:26We find it deeply troubling that a law with potentially severe intellectual and economic harms is built atop such shaky intellectual ground. For example, Senator Wiener's response to earlier criticisms of SB 1047 leveled by Y Combinator and Andreessen Horowitz notes a point of common desire to quote regulate AI for safety between the bill's promoters and its critics, and that the late feedback on the bill is a surprise to him as he posted an outline in September 2023 for the purpose of soliciting early feedback. We would like to emphasize that only those who already believe in existential risks of AI systems, a group that, while containing some field luminaries, does not hold a consensus position in the AI research community, would participate in discussions around safety legislation in its early stages.

17:05As we reject the frame that the risks alluded to in the bill are real enough to warrant regulation, it is hard to offer constructive criticisms to improve the bill, and we instead call on our representatives to seriously reconsider passing such a bill at all. Any fact-based debate around this bill must start from the questionably factual nature of AI risk. And this is the real heart of the problem. I think that was incredibly clearly and succinctly put. But to try to simplify this, what this group of academics and students are saying is that this bill is predicated upon the belief in the AI X risks that the AI safety community is so concerned about.

17:42When I read that letter from Sonny, he made clear that these were quote not science fiction scenarios. When I referenced the letter from Bengio and Hinton, it said relative to the huge scale of the risks. But again, that is an argument predicated on a belief in those risks. The point that these folks are making is that there is not consensus on those risks. In fact, there is fairly significant disagreement about it. And so when a bill is explicitly designed to address those risks, there is no common ground to start from. In other words, this bill didn't start from a question what are the real risks of AI and how might we address them, it started from a very specific assessment of the risks of AI that come from the AI safety movement.

18:22Ars Technica pointed this out in late July as well. In a section of an article called Attack of the Killer AI, they pointed to the shutdown provisions and said, this kind of language in the bill likely reflects the particular fears of its original drafter, Center for AI Safety co-founder Dan Hendricks. In a 2023 Time magazine piece, Hendricks makes the maximalist existential argument that evolutionary pressures will likely ingrain AIs with behaviors that promote self-preservation and lead to, quote, a pathway toward being supplanted as the Earth's dominant species. If Hendricks is right, then legislation like SB 1047 seems like a common-sense precaution.

18:53Indeed, it might not go far enough. However, critics argue that AI policy shouldn't be led by outlandish fears of future systems that resemble science fiction more than current technology. They quote writer Daniel Jeffries, who says, SB 1047 was originally drafted by non-profit groups that believe in the end of the world by sentient machines like Dan Hendricks' Center for AI Safety. You cannot start from this premise and create a sane, sound, light-touch safety bill. Tech policy expert Neerat Weissblatt adds, if we see any power-seeking behavior here, it is not of AI systems but of AI doomers.

19:20With their fictional fears, they try to pass fictional-led legislation, one that according to numerous AI experts and open-source advocates, could ruin California and the US's technological advantage. And so here we really come down to it. Why there is so much acrimony around this discussion. Hold aside the details of where lines are drawn around how big a model has to be to be implicated by this, the specific structure of the new regulatory bodies that have power over these rules, the culpability of developers of open-source systems, all of those things are incidental to the core disagreement, which is about just how risky AI is.

19:53In many ways, this is really a debate about the precautionary principle, the idea that some technology is dangerous enough that we need to regulate it in advance of it coming to fruition. In Forbes, James Brogel, a senior fellow at the Competitive Enterprise Institute, writes, The precautionary principle which advocates for banning potentially harmful technologies until they can be demonstrated to be safe can be an appropriate response in cases where there are clear and imminent existential threats. For example, it makes sense to restrict public access to fissile materials like enriched uranium or to prohibit the everyday sale of grenade launchers.

20:22The problem is that existential threats from AI are not obviously imminent and the US is locked in a race with China. Moreover, the likely benefits of AI, from revolutionizing research to curing some of the world's most intractable diseases, are so substantial withholding them from the public is nowhere near realistic. The battle then here really is not about specific remediations to AI risk, but about the nature of AI risk in general. On the one side are the EA or effective altruist aligned organizations that are extremely focused on AI x-risk. This includes groups like Open Philanthropy, the Center for AI Safety, and the Future of Life Institute.

20:54On the other side are the Accelerationists, the group that Sonny Gotti called tech zealots who dream of a world where AI develops unchecked regardless of the consequences. The group that he argues feverishly dream of a world where technology replaces humans. There is absolutely a transhumanist element to parts of the accelerationist movement, but there are also a huge number of people who disagree fundamentally with the AI safety community's assessment of X-Risk, who have absolutely no desire to see technology replace humans, and who frankly would be pretty pissed at being painted with such a broad brush.