95 Theses on AI Safety

0:00Today on the AI Daily Brief, we're reading 95 theses on AI, and specifically AI safety. The AI Daily Brief is a daily podcast and video about the most important news and discussions in AI. Follow the link in the show notes to join our Discord and join the conversation.

0:24All right, friends, happy weekend. It being a weekend, of course, that means it is time for a long read. And this week, we have something pretty interesting. I was sent this piece, it's on Medium, by a friend who I consider an extremely cogent thinker in and around the AI safety space. The person who shared it with me is an extreme techno-optimist, but also finds himself concerned about some AI safety issues and spends a lot of time thinking about the middle space for that, so I tend to listen when there are things that really capture his attention. Now, the piece is by Samuel Hammond, and it's called 95 Theses on AI.

0:55Obviously, it is modeled off of Martin Luther's 95 Theses and has sparked a ton of conversation in the AI safety space. So let's read it, and then we'll figure out where we want to take the conversation. Section 1. Oversight of AGI labs is prudent. 1. It is in the US national interest to closely monitor frontier model capabilities. 2. You can be ambivalent about the usefulness of most forms of AI regulation and still favor oversight of the frontier labs. 3. As a temporary measure, using compute thresholds to pick out the AGI labs for safety testing and disclosures is as light-touch and well-targeted as it gets.

1:274. The dogma that we should only regulate technologies based on use or risk may sound more market-friendly, but often results in a far broader regulatory scope than technology-specific approaches. See the EU AI Act. 5. Training compute is an imperfect but robust proxy for model capability and has the immense virtue of simplicity. 6. The use of the Defense Production Act to require disclosures from frontier labs is appropriate given the unique affordances available to the Department of Defense and the bona fide national security risks associated with sufficiently advanced forms of AI. 7. You can question the nearness of AGI, superintelligence, or other dual-use capabilities and still see the invocation of the DPA as prudent for the option value it provides under conditions of fundamental uncertainty.

2:058. Requiring safety testing and disclosures for the outputs of$100 million plus training runs is not an example of regulatory capture, nor a meaningful barrier to entry relative to the cost of compute. Section 2. Most proposed AI regulations are ill-conceived or premature. 1. There is a substantial premium on discretion and autonomy in government policymaking whenever events are fast-moving and uncertain, as with AI. 2. It is unwise to craft comprehensive statutory regulation at a technological inflection point, as the basic ontology of what is being regulated is in flux. 3. The optimal policy response to AI likely combines targeted regulation with comprehensive deregulation across most sectors.

2:414. Regulations codify rules, standards, and processes fit for a particular mode of production and industry structure, and are liable to obsolescence in periods of rapid technological change. 5. The benefits of deregulation come less from static efficiency gains than from the greater capacity of markets and governments to adapt to innovation. Six, the main regulatory barriers to the commercial adoption of AI are within legacy laws and regulations, not most prospective AI-specific laws. Seven, the shorter the timeline to AGI, the sooner policymaker and organizations should switch focus to bracing for impact.

3:10Eight, the most robust forms of AI governance will involve the infrastructure and hardware layers. Nine, existing laws and regulations are calibrated with the expectation of imperfect enforcement. Ten, to the extent AI greatly reduces monitoring and enforcement costs, the de facto stringency of all existing laws and regulations will greatly increase absent a broader liberalization. 11. States should focus on public sector modernization and regulatory sandboxes and avoid creating an incompatible patchwork of AI safety regulations. Section 3. AI progress is accelerating, not plateauing. 1. The last 12 months of AI progress were the slowest they'll be for the foreseeable future.

3:432. Scaling LLM still has a long way to go, but will not result in superintelligence on its own, as minimizing cross-entropy loss over human-generated data converges to human-level intelligence. 3. Exceeding human-level reasoning will require training methods beyond next token prediction, such as reinforcement learning and self-play, that, once working, will reap immediate benefits from scale. 4. RL-based threat models have been discounted prematurely. 5. Future AI breakthroughs could be fairly discontinuous, particularly with respect to agents. 6. AGI may cause a speed-up in R &D and quickly go superhuman, but is unlikely to foom into a godlike ASI given compute bottlenecks and the irreducibility of higher-dimension vector spaces, i.e.

4:19Ray Kurzweil is underrated. 7. Recursive self-improvement and meta-learning may nonetheless give rise to dangerously powerful AI systems within the bounds of existing hardware. 8. Slow takeoffs eventually become hard. Section 4. Open source is mostly a red herring. 1. The delta between proprietary AI models and open source will grow over time, even as smaller open models become much more capable. 2. Within the next two years, frontier models will cross capability thresholds that even many open source advocates will agree are dangerous to open-source ex-ante. 3. No major open-source AI model has been dangerous to date, while the benefits from open-sourcing models like Llama 3 and AlphaFold are immense.

4:544. True open-source means open-sourcing training data and code, not just model weights, which is essential for avoiding the spread of models with sleeper agents or contaminated data. 5. The most dangerous AI models will be expensive to train and only feasible for large companies, at least initially, suggesting our focus should be on monitoring frontier capabilities. 6. The open-versus-closed-source debate is mainly a debate about meta, not deeper philosophical ideas. 7. It is not in meta-shareholders' interest to unleash an unfriendly AI into the world. 8. Companies governed by non-profit boards and CEOs who don't take compensation face lower powered incentives against AIX risk than your typical publicly traded company.

5:279. Lower-tier AI risks, like from the proliferation of deepfakes, are collective action problems that will be primarily mitigated through defensive technologies and institutional adaptation. 10. Restrictions on open-source risk undermining adaptation by incidentally restricting the diffusion of defensive forms of AI. 11. Trying to restrict access to capabilities that are widely available and or cheap to train from scratch is pointless in a free society, and likely to do more harm than good. 12. Nonetheless, releasing an exotic animal into the wild is a felony. Section 5. Accelerate vs. Decelerate is a false dichotomy.

5:561. Decisions made in the next decade are more highly levered to shape the future of humanity than at any point in human history. 2. You can love technology and be an accelerationist across virtually every domain—housing, transportation, healthcare, space, commercialization, etc.—and still be concerned about future AI risks. 3. Accelerate vs. Decelerate imagines technology as a linear process when technological innovation is more like a search down branching paths. 4. If the AI transition is a civilizational bottleneck, a great filter, survival likely depends more on which paths we are going down than at what speed, except insofar as speed collapses our window to shift paths.

6:295. Building an AGI carries singular risks that merit being treated as a scientific endeavor, pursued with seriousness and trepidation. 6. Tribal mood affiliations undermine epistemic rationality. 7. EACC and EA are two sides of the same rationalist coin. EA is rooted in Christian humanism, EACC in Nietzschean atheism. 8. The de facto lobby for accelerationism in Washington, D.C. vastly outstrips the lobby for AI safety. 9. It genuinely isn't obvious whether Trump or Biden is better for AI ex-risk. 10. EAs have more relationships on the Democratic side but can work in either administration and are a tiny contingent all things considered.

7:0211. Libertarians, EACCs, and Christian conservatives, whatever their faults, have a far more realistic conception of AI and government than your average progressive. Twelve, the more one thinks AI goes badly by default, the more one should favor a second Trump term precisely because he has so much higher variance. Thirteen, Steve Bannon believes the singularity is near and a serious existential risk. Janet Haven thinks AI is Web3 all over again. Hello friends, quick note before we get back to the show, I'm so excited to share that Superintelligent is now live. Superintelligent is a platform for fast, fun, and super practical, useful AI learning.

7:38We have something like 300 video tutorials adding 30 to 50 each week, covering every topic in AI you can imagine from LLMs to image generators to case studies, use cases, basically everything that tells you how to use AI and what to use it on in short, fast, four to seven minute tutorial videos, which are paired with step-by-step instructions that help you actually use these tools as well. It's$20 a month for unlimited access. and I would love to see you there. Check it out at bsuper.ai. That's bsuper.ai. Section 6. The AI wave is inevitable. Superintelligence isn't. 1. Building a unified superintelligence is an ideological goal, not a fait accompli.

8:202. The race to build a superintelligence is driven by two or three US companies with significant degrees of freedom over near-term developments, as distinguished from the inevitability of the AI transition more generally. 3. Creating a superintelligence is inherently dangerous and destabilizing independent of the hardness of alignment. 4. We can use advanced AI to accelerate science, cure diseases, solve fusion, etc., without ever building a unified superintelligence. 5. Creating an ASI is a direct threat to the sovereign. 6. AGI labs led by childless Buddhists with alt accounts are probably more risk-tolerant than is optimal.

8:507. Sam Altman and Sam Bankman-Fried are more the same than different. 8. High-functioning psychopaths demonstrate antisocial behaviors in their youth but learn to compensate in adulthood, become adept social manipulators with grandiose visions, and a drive to win at all costs. 9. Corporate malfeasance is mostly driven by bad incentives and techniques of neutralization, convenient excuses for overriding normative constraints, such as, if I didn't, someone else would. Section 7. Technological Transitions Cause Regime Changes 1. Even under the best-case scenarios, an intelligence explosion is likely to induce state collapse or regime change and other severe collective action problems that will be hard to adapt to in real time.

9:26Two, government bureaucrats are themselves highly exposed to disruption by AI and will need firmware-level reforms to adapt and keep up, i.e. reforms to civil service, procurement, administrative procedure, and agency structure. Three, Congress will need to have a degree of legislative productivity not seen since FDR. Four, inhibiting the diffusion of AI in the public sector through additional layers of process and oversight, such as Biden's OMB directive, tangibly raises the risk of systemic government failure. Five, the rapid diffusion of AI agents with approximately human-level reasoning and planning abilities is likely sufficient to destabilize most existing U.S.

9:57institutions. Six, the reference class of prior technological transitions, agriculture revolution, printing press, industrialization, all feature regime changes to varying degrees. Seven, seemingly minor technological developments can affect large-scale social dynamics and equilibrium. See social media and the Arab Spring. Section 8. Institutional regime changes are package deals. One, governments and markets are both kinds of spontaneous orders, making the 19th and 20th century conception of liberal democratic capitalism, a technologically contingent equilibrium. Two, technological transitions are package deals, e.g., free markets and the industrial revolution went hand-in-hand with the rise of big government.

10:32Three, the AI-native institutions created in the wake of an intelligence explosion are unlikely to have much continuity with liberal democracy as we now know it. Four, in steady state, maximally democratized AI could paradoxically hasten the rise of an AI leviathan by generating irreversible negative externalities that spur demand for ubiquitous surveillance and social control. Five, periods of rapid technological change tend to shuffle existing public choice and political economy constraints, making politics more chaotic and less predictable. Six, periods of rapid technological change tend to disrupt global power balances and make hot wars more likely.

11:02Seven, periods of rapid technological change tend to be accompanied by utopian political and religious movements that usually end badly. Eight, explosive growth scenarios imply massive property rights violations. Nine, a significant increase in productivity growth will exacerbate, bumbles cost disease, and drive mass adoption of AI policing, teachers, nurses, etc. 10. Technological unemployment is only possible in the limit where market capitalism collapses, say, into a forager-style gift economy. Section 9. Dismissing AGI risks as sci-fi is a failure of imagination. 1. If one's forecast of 2050 doesn't resemble science fiction, it's implausible.

11:352. There is a massive difference between something sounding sci-fi and being physically unrealizable. 3. Terminator analogies are underrated. 4. Consciousness evolved because it serves a functional purpose and will be an inevitable feature of certain AI systems. 5. Human consciousness is scale-dependent and not guaranteed to exist in minds that are vastly larger or less computationally bounded. 6. Joshabach's cyberanimism is the best candidate for a post-AI metaphysics. 7. The creation of artificial minds is more likely to lead to the demotion of humans' moral status than to the promotion of artificial minds into moral persons.

12:058. Thermodynamics may favor futures where our civilization grows and expands, but that doesn't preclude futures dominated by unconscious replicators. 9. Finite time singularities are indicators of a phase transition, not a bona fide singularity. 10. It is an open question whether the AI phase transition will be more like the printing press or photosynthesis. Section 10. Biology is an information technology. 1. The complexity of biology arises from processes resembling gradient descent and diffusion guided by comparatively simple reward signals and hyperparameters. 2. Full volitional control over biology is achievable, enabling the creation of arbitrary organisms that wouldn't normally be evolvable.

12:403. Super-intelligent humans with IQs on the order of 1 ,000 may be possible through genetic engineering. 4. Indefinite life extension is a tragedy of the anti-commons. 5. There are more ways for a post-human transition to go poorly than to go well. 6. Natural constraints are often better than man-made ones because there's no one to hold responsible. 7. We live in a base reality and in nature there is no such thing as plot armor. Whew, what a piece. That is the dense definition of a long read. Obviously we are now off Samuel's piece, and I am back as NLW discussing this again. And where I want to focus is, let's hold aside some of the deeper philosophical pieces, even things about regime change, politics, post-human futures, etc.

13:18Basically the second half of things. Where I want to focus is on the first half. And what I think is valuable about this is the extent to which it tries to obliterate the sides that are trying to calcify this debate into AI safety or AI risk on the one hand, or accelerationism on the other. Samuel takes positions that are much more coherent in many ways with the EACCs in areas like his section on most proposed AI regulations are ill-conceived or premature, even in the section on open source, where, for example, he talks about the benefits from open sourcing models currently being immense. The flip side, his first section, and what feels like perhaps the biggest takeaway if he was forced to identify one, is the AI safetyist sort of position that oversight of AI labs is something we should be doing.

13:58For a long time now, I've really wanted the discourse to move beyond the hyper-theoretical and instead get into the very practical and specific. And I like the idea of people picking up this conversation and really just honing in on this one question of monitoring labs on the basis of the compute they're using. Now, while this question is to Samuel not complicated, to many people it is. There are slippery slope issues, there are government power versus market issues. But at least when we hone in there, we can kind of ignore some of the other stuff and just focus on the issues in the context of themselves, which I think is likely to lead to some better decisions.

14:30Anyways, that is going to do it for today's LRS. Appreciate you listening or watching as always, and until next time, peace.