#00 - Daniel Schmachtenberger - AI, Capitalism, Misalignment & Moloch

31:16the competition, the Cold War between the USSR and the USA was not, which was being framed as two different political economies, right? Communism and capitalism and competition was not capitalism, right? And the fucked up things that were happening inside of the ussr were not capitalism so we're not talking like the critique that we're about to offer of capitalism is not saying some previous economic system or political economy was better and actually moloch instantiated itself through those systems as well capitalism was more effective and it did get selected because it was more effective at both good things and up things right which is kind of what you mentioned that it wins a war but it has to sacrifice important stuff to do so.

32:01So the thing that can be reductionist and win certain critical metrics, but harm other stuff in the process where eventually the cumulative effects of those harms are either catastrophe or dystopia, a world that nobody really wants. And those are the two primary attractors right now. You have catastrophes and to prevent all the catastrophes, to make sure that people can't build catastrophe weapons in their basement, what type of surveillance is needed to make sure that you have enough controls on all the things, if you really have the ability to control the entire landscape of things that could lead to catastrophic risks that are radically decentralized, most of those solutions look pretty dystopic.

32:39And so we want a future, a third attractor future, that is neither catastrophes nor dystopias. And without saying what it is first, let's just say that it's not those things. And we can almost all universally agree that we would prefer not those things, which means that we need something that has the power to be able to prevent catastrophic risk, but also needs checks and balances on its own power, right? It doesn't have unchecked power and capturability or corruption or those types of dynamics that are then uncheckable. It's beyond the scope of this video to talk about what that third attractor solution is.

33:15But so we have, you know, when we talk about capitalism as a kind of dominant global economic system and of course we don't have pure laissez-faire capitalism we have this kind of hybrid um political economy but roughly this thing it does not subsume all of moloch like we said moloch was operating under feudalism and under communism and under other systems and in the competitions between them it is more fair to think of it as kind of um the god of game theory uh but as capitalism being such a powerful part of that stack we can think of it as a metaphor for a moment and say capitalism is well let's just start by the a couple key aspects of the incentive dynamics vast majority of human history and tribal type dynamics pre-agriculture all of our kind of genetic fitness in that environment we didn't have the ability to store a lot of surplus right that happened post the plow and grain and storage technologies and whatever uh in which case there were all these kind of sayings with various tribes the best place to store extra food is in your neighbor's belly because rots otherwise and you know etc and tribe are invested in so but as soon as we start getting to uh private property ownership and the ability for a lot of surplus where i could differentially make it through a famine better than somebody else i could have a higher quality of life than somebody else i could pass on inheritance.

34:48As soon as I have private property as a possibility, now there is an incentive to try to turn more of nature into my private property, right? And to try to turn more of other people's actions in my private property. But when the property is actual real commodities and goods, or the agreements with people who can do services, let's stick with goods because it's easier for right now, there's a diminishing return on the value of any of those based on the illiquidity of them or the difficulty of moving them. I get more lumber at a certain point. I have more lumber than I can use, and I can't even move it around to sell all that quickly.

35:24And so I don't really want all that much more of it. And the same would be true with ore of a certain kind or whatever. But as soon as we move to a kind of a currency mediated system where I can sell it in real time and turn it all into something that has no intrinsic value, but the optionality for every form of value well now there's no fungibility fungibility now and obviously we started with things that had intrinsic value but still got used to mediate it like gold but then you know we got to fiat so i'm just going to do this huge jump to fiat because it's the current system even though it has no intrinsic value what the value that it has is maximum speed of optionality, right?

36:08Maximum optionality and maximum kind of liquidity and speed. And so in that situation, there's no diminishing return on getting more, like more is more, right? And whether I want to convert that money into military power or convert it into public opinion through media and campaigns and stuff, or convert it into technological power of one kind or another kind or land ownership, the money allows me the ability to do all that. So you can think of it as just units of power or units of game theory, right? Units of OODA loop. And then when you add money on money dynamics to it, compounding interest as the beginning, then of course, all of the financial services that become possible with more capital, but just compounding interest, not only is there not a diminishing response to as I get more and more ore, it becomes less valuable to me because I can't use it fast enough.

37:01Now, as I get more money, it is actually exponentially making money on itself. So when I have private property, I have the ability to turn all of that into fungible units of capital, and it makes money on itself. There is now a maximum incentive to turn as much of the world as possible into capital in my holding. And because other people are, and they could use that against me, there is now an arms race for me to do it faster than that guy. Right. And that's decentralized. And

37:37now, of course, we can see that when we're talking about there are types of power that don't directly just relate to dollars, right? The number of Twitter following is one or the amount of covert political influence or, you know, many, many other things. There are military generals that have more total power than the amount of money they have. But obviously they influence a huge amount of money in terms of military assets, how much they cost and things like that. And so this is why I say I don't want to reduce it exclusively to money, but if we had to pick a single metric that has the most kind of optionality for all other types of metrics, that would be the one.

38:18So if we're thinking about Moloch as a whole, we can see that whether we're talking about the environmental issues or whether we're talking about the increasing polarization because of social media algorithms or whether we're talking about any of these things, the rapid race that is not orienting towards safety enough on new technologies, that this set of dynamics is underneath it, right? And this is why that kind of frame Scott Alexander and others have put forward, which is who is engineering this thing? Well, Moloch is engineering this thing, like that thing overall. And now this is where I want to stop and go into the, what is a misaligned AGI?

39:03What is a paperclip maximizer for a moment? Because it actually makes Moloch clearer and then Moloch makes it clearer. Do you want to construct the kind of paperclip maximizer scenario for people? You know, a lot of the people in the kind of AI risk space. So the paperclip maximizer is like a thought experiment that is basically of like an extreme super intelligence gone wrong. Because just because you can build something that is by definition, super intelligent, you know, and if we define intelligence in what I think is the best definition, which is the ability to optimize and navigate a very broad range of terrains in order to achieve whatever your goals are.

39:48So if we can achieve, you know, define intelligence as that, your ability to basically get stuff done across a wide range of environments. That does not necessarily guarantee that you also have the wisdom to decide what your goals should be in the first place. It's called the orthogonality thesis, the idea that like, you know, maybe intelligence and wisdom are perfectly aligned, but there's actually a very large possibility that they are completely unaligned. You know, They're just orthogonal to one another. And so the paperclip maximizer is like the extreme, a silly example of that, an arbitrary example, whereby let's say you want to build a machine.

40:30You have a factory that builds paperclips, and then you happen to get a hold of a superintelligence that will help you build them as fast as possible so you make maximum profit. Your superintelligence then is able to, because it's so capable at navigating a broad range of goals, turn every atom that it comes across into more paperclips until the universe becomes tiled with them. So yeah, it's basically a very somewhat oversimplified, but at the same time kind of comically salient example of a deeply misaligned, but nonetheless super intelligent system. Right. So just to construct a couple of key parts of it, Paperclip is obviously a silly and kind of cute on purpose example of whatever it is, right?

41:21Whatever commodity, whatever widget that you're optimizing for. And without even saying super intelligence, let's just say increasingly good, increasingly competent and generalizable artificial intelligence gets applied to the corporation, which is already happening everywhere. and it has just two features which is it can work to achieve a goal whatever its objective function is in this case make more paperclips and it can recursively improve itself so it gets better at doing that thing that's key right um so of course at first it does a bunch of stuff that we want it figures out how to turn off lights when people aren't there to save cost and energy and how to make more efficient supply chains and negotiate better deals and all those kinds of things and just makes a more efficient business and of course the reducto ad absurdum is once it has done all of the easy good stuff it still has the objective function make more paperclips and then it has to start doing stuff that is not just obviously easy good right where there's some trade-offs that are happening somewhere else but the things that are being harmed are not part of its objective function right its objective function isn't make the most paperclips wow don't doing any harm not do harm anywhere else because that the wow don't do harm anywhere else is actually incredibly hard to specify in an easy computational way which is the heart of what we'd call you know the alignment problem and so if you did have something that could recursively increase its ability to achieve a goal like that and then had enough generalized intelligence that it could outcompete anyone that was competing against it right it could increase its capability faster than say we as humans could and we're like, oh, fuck, we don't want you to be making paperclips out of our food sources.

43:01We don't want you to be making paperclips out of, but it figures out how to beat us at those games. Then yes, eventually it just starts turning everything into the substrate for paperclips. And in general, the idea is you have an objective function, whatever the AI is optimizing for, whatever that is. And there's different ways that people will describe it in nuance. And, you know, it's worth reading Yudkowsky and Bostrom and the other, you know, kind of seminal thinkers on what the nature of the AGI alignment problem is. But roughly, if you have an artificial intelligence that is general and autonomous, autonomous meaning it is working on its own.

43:41You don't have to keep giving it prompts, right? It's doing its own thing. It has agency. And where you can't pull the plug on it, right? That's a key part. and it can upgrade its own capability to do whatever it is that it's seeking to do and it can upgrade its capability faster than we can because the smarter one then is capable of making even smarter one and we already see early signs of this we already see ai starting to create better internal ai functions to be able to achieve their um the goals that are set for them the idea of do we want an autonomous general intelligence that is comprehensively smarter than us that is trying to fulfill a goal before we know that its fulfillment of that goal isn't going to really mess stuff up for us right we're like no no we don't want that thing right we would like to prevent that because it's entirely possible that it could do some things that are totally not what we want in pursuing that goal.

44:48So obviously, if its goal was to maximize GDP, there's a lot of nasty ways to maximize GDP. It can go up with war. It can go up with addiction. It can go up with and whatever it is that the objective function is, there's a lot of perverse instantiations of that thing being fulfilled in a way that totally messes up other stuff. And so the AGI alignment question is, can we actually ensure that before the thing is truly a general autonomous intelligence that it is aligned, aligned with our interests, our intention, our good or something? What the nature of alignment means is actually a deep question.

45:20I'm going to put that on hold for a moment, but roughly aligned with us such that that much power would be a safe thing for us to have exist. and what i the thought experiment of an intelligence that was say as much more intelligent than us as we are than chimps or ants or whatever uh looking at how our increase in intelligence has voted for all the other inhabitants of the planet uh uh you know that's a very concerning thought now this is where i want to actually use the analogy of that in autonomous so it's doing stuff on its own right it's it's autopoetic it's self-authoring it's self-upgrading and it's orienting towards an objective function And I would basically like to say that you could call the current global system, and just to simplify it, let's call it global capitalism, even though it's not that.

46:26Calling it Moloch would be better, right? But let's just talk about the capitalism part because the metrics are kind of clear. You could say that it is already a general autopoetic superintelligence. it is has an objective function which is to convert as much of the world people's creativity ideas labor natural resources everything into capital and so that's the paper clips right and which is interesting because it has no real value just optionality for real value and there's always this assumption that there's more real value out there but that stops being true forever So if I, the tree sequesters CO2 and produces oxygen and I need to breathe oxygen and it does a lot of other important things like supports pollinators and cleans the water and stabilizes topsoil and all these things for me.

47:21But if I cut the tree down, turn it into lumber, there's still enough oxygen for me. I didn't actually ruin my, if I cut the whole acre of trees down, there's still oxygen. In fact, there's no differentially seeming less oxygen for me doing that. But now I have the money of all the lumber of these trees and I can do real tangible shit for me and my family or my corporation with that. And so the optionality value I get allows me to still access real value, but I'm destroying real value in the process. So I'm going to say, well, you're not destroying it because you're making lumber. Well, yes, lumber is actually radically less complex than a tree that has less total types of value that it does.

47:57So we're converting the self-organizing, self-repairing, complex world into an increasingly simple or complicated, fragile world that has less types of value to less types of actors. The tree has value of many different types to many different types of actors, right? So you can't just say, well, it's carbon sequestration. No, it's stabilizing topsoil. It's a million things. Biodiversity, et cetera, yeah. So when we talk about artificial intelligence, we talk about what type of computer system it runs on. Hardware-wise, it runs on CPUs or GPUs or TPUs or whatever it is, and what type of algorithms that it runs.

48:44So it's interesting that we can already say humans are general intelligence. And capitalism is running parallel process across all the humans. right and we know when you think about the cloud and why parallel process was so powerful it's also why gpus are so powerful um capitalism is basically as a decentralized incentive system incentivizing all humans to both do novelty search figure out new ways of making money and exploitation take the existing ways and do the most of them that you can those that do better at it get more influence in the system and in turn influence the system in ways that support them to do more of it.

49:21Those that oppose the system are also opposing those who are doing well at the system. So even though the system as a whole doesn't have agency, those who do keep in check those that would oppose them. So it is as if the system has agency. And you go from barter to currency, to fiat currency, to fractional reserve banking, to AI high-speed trading of derivatives and credit default swaps. And that is basically the recursive upregulation of the algorithm, right? It is getting more and more capable of doing more and more financialization of the world to incentivize people to do more and more things.

50:01So you can see that you've got something that is already running on all these general intelligences. And as a result is super intelligent. It has an objective function. The objective function is misaligned with the long-term well-being of the world. And it advances narrow value metrics at the – and it's not that like – but everything is getting better Pinkert-Rostling type arguments are like saying, but look at how many paperclips we have. Aren't we all stoked that we're getting cheaper paperclips? and multiply by all the types of paperclips. Like, yes, narrow metrics are being advanced at the cost of lots of wide metrics that end up being critical to either the breakdown of life support.

50:44So you get catastrophe or the breakdown of the quality of life, you get dystopia. How would that then sort of using that definition apply to the difficulty of the alignment problem with an AGI? So here's the thing. We think about the superintelligence. And a misaligned superintelligence is a very scary idea. And if you want a sense beyond the silly paperclip maximizer of how scary it is, read some of people like Eliezer Yudkowsky and others on what AGI misalignment means, superintelligence misalignment means.

51:25before we go all the way to Moloch we could already say that collectives of lots of people interacting in particular ways whether we say a public corporation or say a nation state let's say we take a large public corporation right well beyond what we have evolutionary history for we had evolutionary history for tribal type size things below the Dunbar number where everybody could talk to everybody. So possibly the human scale ethics and whatever could perpetuate through the whole thing. At much larger scale, it changes, which is why the beginning of large civilizations started to have different properties.

52:00So you take a public corporation. Who's in control of it? Kind of nobody, right? Like the people in the corporation answer up to the executive team, answer to the CEO. The CEO answers to the board. The board answers to the shareholders. the the board has a fiduciary responsibility to maximize profit returns to shareholders the shareholders are pension funds and whatever where the managers of the shareholders are trying to get money back to the 401ks and the whoever is in there they end up being pieces of law that are bound up through the whole thing which is the the liability limiting status of the corporation that can privatize gains and socialize losses and the fiduciary responsibility of the directors to maximize shareholder profit and the on and on.

52:47So who's really in charge of it? You can get rid of a CEO and put a new one in. You can get rid of a director, put a new one in. You can sell some of the shares, get a new shareholder. It's kind of this thing that gets set in motion where it has an objective function, which is now maximize profit within the domain of how it figured out how to do that thing. And so, but because it is engaging, it's running on all these human intelligences, which are already general intelligence. So it can do things that the humans can do plus things that none of those humans on their own could do, right? It takes a lot of humans together to do a large Hadron Collider or a Hubble or an Exxon, right?

53:23It has a capacity that nobody could do on its own. So it is super intelligent. It's beyond human intelligence in that way, not just in a narrow way, because it's engaging people that are already generally intelligent. And then beyond just that, it's already engaging computation. So it's engaging the narrow, but very powerful data processing. And now we add AI to that. And so we can say already that a nation state or a public corporation is kind of a cybernetic general intelligence that is already misaligned. Can you just define what you mean by cybernetic? Like cybernetics, just the field of study for any type of system that kind of self-regulates, right?

54:07How the control mechanisms work, how the regulatory mechanisms work. So a corporation has feedback loops. It has feed forward loops. It has regulatory processes to be able to maintain what it's doing and upgrade what it's doing, right? NationSafe has that. And, but it's, it is dealing with internal, but also external pressures that force it to be what it is. maybe Google didn't want to release its large language model yet, but as soon as its business model gets attacked by Microsoft releasing one and adding it to being in the possibility of search, then it has to. And so this was where the multipolar trap, the Molokian type dynamic comes in is the individual organization is not totally sovereign because it's for it to keep existing.

54:50It has to deal with the pressures defined by others. And so either a sociopath can start something that then everybody has to deal with, or everyone assuming the other one is about to do it next and no sociopath has a situation that is functionally sociopathic. We talk about a corporate person, 14th amendment, kind of giving personhood rights to a corporation in this weird way. If we were to talk, so there's already a framework for thinking of it as an agent, right? The superintelligence as an agent, but the fiduciary responsibility to maximize profit makes it kind of an obligate sociopath, right?

55:26That kind of thing has to take the opportunities it has so long as it is not illegal within the confines of law, but it can work to change law, which is what all big corporations lobby, right? And that ends up being one of the very profitable things that a company does is rather than the regulator limiting it too much, it figuring out how to get the regulator to change regulation more aligned with its interests. um and so for it to say hey uh a shareholder profit maximization fiduciary responsibility in an oil company and solving climate change are incommensurable a shareholder fiduciary profit maximization and military industrial contractors and a world of peace that would de-necessitate all the demand or incommensurable, right?

56:15And so then those competing with each other, right? So then any of the A groups are like, hey, I can't really do safety because the other ones, we all kind of have to race at this. And then maybe the whole US is we don't want to regulate it because we would rather our guys get there before China gets there. Because whoever has, it's going to run the world and we'd at least like it to be US companies. So that Molokian dynamic makes it to where each of these cybernetic super intelligences interacting with each other creates a meta cybernetic super intelligence that you can call moloch right which is why i wanted to talk with you about it is you can see moloch as an emergent property of the systems of incentives and the uh dynamics of coordination that are built into the system where it is employing human general intelligence.

57:10It's employing computational capabilities and increasingly artificial intelligence and the whole rest of our tech stack. It is upregulating through competitive dynamics, but upregulating in this narrow benefit kind of way. And so we could say this thing that is driving climate change and driving species extinction and dead zones and oceans and coral loss and desertification and arms races and polarization and all like that, that is a misaligned superintelligence that nobody can pull the plug on. It's already autonomous. It's already nobody can pull the plug. And it is building AI. Because corporations - It's not actually the people within.

58:01They think they're the ones building it. Well, but they're building it within corporations that have a fiduciary responsibility for profit maximization that are in multipolar traps with other companies that are racing to do it, that have to look at how do we commercialize this thing, whatever it is, right? Or they're building it within nation states that have to be able to compete with another nation state. And what that means is that some narrow value metrics that define what wins the competition get prioritized over wide value metrics. And so it is fair to say that we already have a misaligned autopoetic super intelligence running the world, running on and running all of the people to various degrees.

58:44It is already employing all of the computational power. It is developing more computational power in AI. The AI is being built by it in service of itself. So the AI risk scenario that Yudkowsky or Bostrom or others put forward of a thing where you can't pull the plug, it is upgrading its capacity to do what it does. It has an objective function it's pursuing, but it harms stuff that we wouldn't want harmed in its subjective function in the pursuit of it. What I'm arguing is that we are already there. And it is our world system. And that AI is simply accelerating it. and that we don't have to get to AGI to have the effect of it because you already have GI.

59:29We already have general intelligence in the form of the corporations, nation states, and the overall system where then adding AI, even if it is not fully generalized to that system, you already have something that is autonomous in general that is now getting increasingly potent capacities, even if it's within a bunch of narrow domains, right? And so before we get to the case of just autonomous AI being its own risk, the existing AI in this landscape is driving the entire risk landscape, is driving the overall, is accelerating the topology that is already in place.

1:00:13and this is why i said i think the misaligned agi as a thought experiment helps people understand Moloch. But what the reality of Moloch helps people understand that without getting to a total AGI, that the nature of the risk there is already happening. Then with that, we have to say, what would it take to prevent those risks? And it's a different calculus. It's a different way of thinking about it. So what do we do? Let me make it a little bit more tangible first and talk about sub-AGI within this context, this Molokian metacrisis context, what do the actual risks of AI look like? I'll give a few different categorizations.

1:00:59And I want to say I am not an AI alignment expert or a AI risk expert. There are a lot of experts at places like Miri and Redwood and other places that I think people should listen to, pay a lot of attention to. I'm familiar enough with those arguments and then very specifically with the metacrisis arguments to see how it relates. And that's what I'm speaking to here.

1:01:26There's something very unique about AI relative to all other forms of technology. I'll speak to the deeper part in a moment. But to begin with, synthetic biology is very powerful. Like, obviously, it's very, very powerful. There are awesome applications. there are awful applications um but synthetic bio does not automatically give us the ability to make better drones it doesn't give us the ability to make better high-speed trading doesn't give us the ability to make better nukes nukes don't automatically give us better bioweapons ai gives us better all of them right that's an important thing is that ai has the capacity to do optimization across all the things, which means the good things, which is what we all want.

1:02:12We want to have AI work on protein folding for immuno-oncology to cure cancer and on receptor sites for new drug discovery and to make supply chains more efficient and things like that. But everything that AI can optimize, it can also break. Run it in reverse, right? The AI that was doing drug discovery, I think it was Oak Ridge National laboratories was ran in reverse and came up with a bunch of chemical weapons very rapidly and uh minus sign in front of it essentially and an ai that can optimize supply chains can also optimize exactly how to break them right can optimize terrorist attacks on them even just through cyber and things like that a ai that can do protein folding for amino oncology can also make fucked up bioweapons and so the first principle is that as you it's very hard to advance ai right it takes massive gpu farms it takes only a few companies in the world that can even do the chip manufacturing to do that kind of thing um it takes a lot of computer science talent massive amounts of data and etc but once it's developed and then it is connected to the internet like a large language model can run on a lot less compute than it's trained on.

1:03:35It takes a lot to train. It doesn't take that much to run it. That's a big deal. And it takes a lot to figure it out. Once it's been figured out and you publish the paper is, and this is one of the key things is. It's like building software. You know, it's very hard to build, you know, you need programming knowledge and so on to build a software, but once you've built it and you've built the sort of user interface, any, any old schmuck can use it and get, and like read the benefits from it. So if you have a company like a Google or an OpenAI or whatever that says, hey, we're going to put safety parameters on this.

1:04:04I think there's a bunch of arguments against why that is. Even if they could do it, won't be adequate, but they can't because, you know, you had the I think it was Llama or Alpaca, the Stanford meta one that ended up getting leaked through some GitHub leak. and then somebody downloaded it onto a computer, started sharing it. And that means the full power of something that probably cost tens of millions of dollars to train is now unlocked and will be available for anybody to use for all the purposes. So the safety is not there. There are some projects working on decentralized AI that are about to make at least GPT 3.5 level kind of unlocked, widely available.

1:04:50That thing is impossible to avoid. and so the thing to understand is that it takes a lot of work to develop the new capacity once it's developed the barrier of entry to be able to do the things that it allows to be done has been radically lowered for everybody that all the good things come from that we can all do more creative stuff that's exciting all the bad things come from that and so there's this principle that we could say all technology is dual use right you're developing it for some positive purpose but it has a military application but it's not just dual every technology is kind of omni use meaning it will get used for all the uses that people have incentives to use it for who are capable of using it so however much you lower the barrier of entry anyone who can go into that barrier of entry now will use it for the things they have incentives for so you develop it for we're going to cure cancer but now you've got that ability for biological engineering really easily available widely right um and so i think the the couple things the first thing to understand about ai is ai can make better cyber weapons better nuclear weapons better drone weapons better all of those things better info weapons better population centric weapons and so it increases the capacity to increase all the other risks in a way no other thing does.

1:06:20You could kind of say, well, oil does that, right? Energy, every industry needs energy. Yeah, but it's not doing the novelty search part of figuring out new, better ways to make those domains. It's simply just allowing them to do more of it, right? The AI both allows you to scale the stuff, but also allows the innovation of way new, better stuff. So that's novel. That's a very novel thing about it. So the first principle is anything AI can optimize, it can break. You develop it for one purpose. It'll get used for all the purposes that anybody can figure out how to use it for. You try to make safeties and whatever, but you create an incentive now for a bunch of cat and mouse type dynamics on how to utilize that.

1:06:59and obviously we can think about the ones that just involve synthetic media and increasing hypernormal stimuli and ubiquitous deep fakes and really dreadful things like that right like there's in terms of population-centric warfare and breakdown of government and public trust and there's a lot that are very very near term so this is one set of risks anything that the eye can optimize, it can also break. That's kind of the bad actor case. But the other case, the more Molokian case, is just accelerating the thing that is already happening. The externalities that aren't included in the optimization function is accelerating the externalities when we're already hitting the tipping points on the externalities.

1:07:47And so you could say, yeah, but AI is going to make it to where we can produce things so much more efficiently that it'll actually save the environment um it's not that there isn't a way to do that and those are the things that i want us to pursue but we're not on track for that you've got this kind of jevons paradox that when you increase the efficiency of energy you don't actually use less energy use more energy because now energy is cheaper which opens up a whole bunch of new markets that weren't open before the same is true with compute you make compute cheaper you use more compute, not less compute.

1:08:22And so AI makes some stuff more efficient. More efficient just means there are more things to which I can apply energy to get more energy. As long as there is positive return, we will go for it. Now, to not go for all those things, you can't do with purely incentive. You have to do with deterrent, with agreement, with law, with some other thing, which is not adequate in speed to the overall situation right now. So one risk of AI is that anything you can use to optimize, you can also use it to break, it increases all of the other risks in a way nothing else increases all of the other risks. It increases the total complexity of the risk landscape, et cetera.

1:09:01The other one is that even when you're using it for the positive purposes and you're succeeding at whatever your positive purposes are, you're also speeding up externality right as we're hitting the tipping points. The next problem is you're increasing the info complexity. I think Yudkowsky calls it inscrutable matrices of floating point numbers talking about the large language models and like nobody actually knows what the fuck is going on inside of them right those kind of black boxes so the only thing that could figure out is your ai actually doing the thing it's supposed to do or not how do you if we wanted a if we wanted a law in some way to be able to regulate it or adjudicate what's happening it would take another ai that's more powerful to do it and so now you end up getting an increase of the race towards the info singularity where people can't actually make sense of or adjudicate any of what's happening right the total complexity of everything is beyond our ability to process and that just means the unsolvability of everything increases so what i'm saying is that if you add something as fastly recursive and powerful as the increasingly generalized AIs that we have to an already misaligned super intelligence near the boundary points of breakdown.

1:10:24That's a problem. And that we should figure out alignment first. So what I'm suggesting is in the way that the AI risk community is saying we should really try to figure out alignment before we race forward on developing more powerful AIs. I'm saying, yes, we really need to figure out alignment, but it doesn't just mean alignment of the AIs. It means alignment of the existing general intelligences, the cybernetics general intelligence. The capitalist model, essentially. So it's not fair to call it capitalism. It's more like game theory. Capitalism just makes it easy to think about because there is actually a metric.

1:10:59And it turns out that the capital can be used for population-centric warfare or supply chain or most any of those things. It ends up being kind of a unit of power pretty widely. But yes, that system, and it's hard to even call it a system, right? That set of perverse incentives and the coordinations that arise from it is misaligned, reaching criticality. Alignment in that thing has to be figured out because a misaligned context cannot develop aligned AI. It can't emerge from it. Because that's, I think, some people's hope is that just give it enough capability and some emergent magic will essentially come.

1:11:45This is, I think, why the orthogonality thesis is important, which is to say it is possible to get very good at optimizing and not get good at picking good goals. Those are two separate things. And we already see that in the world. We're already much more good at creating tech than we are at creating a world that everybody thinks is a world that really makes sense. So this is the exponential tech gives us the power of gods. We have not yet seemed to demonstrate the love and wisdom of gods needed to bind it. We have to figure that thing out or this thing kind of caps out. Same with the AI. Right.

1:12:19And we've handed it over to the shitty god. The worst one. So AI has fast enough feedback loops and enough power that it is one of the only things that could help change the other thing in time, change the paperclip maximizing nature of the global system, right, of the Moloch system in time. But only if it was developed in association with the cybernetic systems that were actually aligned. And aligned here doesn't just mean with our intent. Aligned means actually with our long-term well-being. And this is one of the critical issues is when we talk about alignment, aligning AIs with human intent would not be great because human intent is not awesome so far, right?

1:13:07Like that's kind of the point is that whether we are looking at the overfishing of the oceans or proxy wars or whatever it is, we're like, do we want to give exponentially more power to this species? with its intent caught in multipolar traps the way that it is it it is not a good steward of power in the bronze age it wasn't in the iron age it wasn't it still isn't but with exponentially more power exponential externalities and exponential conflict both eventually break the finite playing field and so how do we and you know solving those coordination failure multipolar trap type dynamics are necessary for the wisdom to be able to prevail adequately so i am very hopeful of the very of the uniquely positive things to support coordination that computational capabilities and artificial intelligence in particular can help with they are not being developed for those purposes and in contexts that have the right frameworks and the right incentives currently.

1:14:16It's very much the opposite thing. So rather than build them to be able to change Moloch, they're being built by Moloch in its service. Even though no one building them would say that, but the nature of the capital that they are building it with has that built right into it. Inherently, yeah.

1:14:40so the the sort of aesthetic of the thing that to me is the anti-moloch but i don't even like to call it the anti-moloch because it is something that operates that by calling it the anti-moloch says it's on the same plane as it the same dimensionality and it's something higher than that uh is this thing you know if moloch is the god of lose-lose games negative sum games what's the god of positive sum games i call it win-win everyone has different names for it you call it omnia it is that the direction of the type of artificial intelligence we need to build and if so how would we go about doing that could we use a whole bunch of info technologies including artificial intelligence including we can already see that all the problems that we saw in the social dilemma that you show in your second Moloch video, and some in Beauty Wars, the social media-related ones, well, that is actually already a certain kind of AI, right?

1:15:42It's AI that is curating the news feed, aligned with an objective function. The objective function is either time on site or engagement or something like that, some combo of metrics. And so, of course, if the objective function is to maximize your engagement, things that are addictive will do, things that piss you off and tribalize will do and so it gets to take now what's interesting about the moloch part is so far facebook or tiktok whatever has not been creating its own content but it has been incentivizing all people to create content that will rank and as the people pay attention to what ranks or doesn't as you saw beauty filters were one thing and there are a lot of other things even so far as that all of the legacy media now does stuff that will make it rank on Facebook and Twitter and whatever, because that's increasingly where the eyeballs are coming from.

1:16:29So even that, which would seem like an alternative source is still actually influenced by that thing. So the, to be able, because it's so powerful to direct all human attention at the thing, right? So whatever the algorithm is, it's going to direct all human attention is also going to direct all the innovation in the direction of what wins that algorithm. And so, and then because it's customizing the newsfeed for every person, it's split testing what do the people click on and engage with and share and etc so it's basically just objectively maximizing for personal engagement without paying attention to if it is positive or negative reward circuits and it happens to be that negative reward circuits are easier to hack than positive ones most of the time and because the positive ones you want to get the fuck off the computer and go do other stuff and then doom scroll but the doom scroll mass share thing the negative reward circuits feed better on so that's already an example of a bad objective function and an AI that has made it to where there are almost no adolescent girls with a good body image.

1:17:31Right? Dysmorphia is kind of ubiquitous. It has made it to where polarization is as extreme as it is and on and on. The externalities are massive. What it's done to attention span.

1:17:44Now, with synthetic media, we're talking about not just an AI that can curate but can create. Now you can imagine a feedback between those. When one is creating things, it will be maximally sticky to you based on personal dynamics, split testing, multiple created ones. And the other one is curating, but now not just getting all the humans to do decentralized creation, but also the... You can see that kind of feedback. But of course, if we had, if we changed the objective function of social media as AI, right, and rather you did things like, it's not just how much engagement it gets, which can be fighting that is polarizing, but something that, say, gets positive engagement across political divides and ideological divides and mimetic clusters, which is stuff that we could do, right?

1:18:33We have the infoscience to do that, which means that it's identifying places where there is shared agreement or shared perception and upregulating that rather than the most divisive stuff. You'd have a totally different world, and the technology could totally do that. It would not be as good for ad sales right now because it actually wouldn't keep people online quite as long. And so this is where the fiscal model fucks up the application of the tech because we're not just saying all social media is bad. were saying the incentives make us develop the fucked up versions of it right and so we could make social media that was exposing people to different ideas rather than reinforcing their existing ones and growing their network to include people of very different types and like so similarly rewards them to go out and touch grass essentially and you know almost rewards you for time spent off the app or something like that there are probably two there are i mean now one of the challenges is of course the ai company can say we're just giving people what they want right but it's manufacturing demand on bad reward circuits in the same way mcdonald's can say we're giving people what we want but you have a much more obese nation after it yeah exactly right and so on the other side you're like well it sounds kind of um paternalistic that we should pick what are the good reward circuits.

1:19:53But it's like, no, if you have that much asymmetric influence over humans, to not take some responsibility for what the statistical changes in their life are is actually silly, right? Now, do we want the corporations to do that themselves? Do we want the government to do it? We don't trust any authorities adequately right now for good reason. So there's some very deep conversation around how do we ensure that the power of that technology is optimizing for things that actually increase quality of life in meaningful ways. But the same with all new types of AI. Could we use it to radically improve governance where we could actually be able to have the large language model, see what people's beliefs and sentiments across the entire space are and find the topics that actually a lot of people agree on, that supermajorities would agree on and start there and be able to make platforms for candidates to actually be able to represent the the wills of the people better because we actually can see what they feel and believe at scale and um would it be possible for it to work on identifying things that a lot of people would believe or at least for it to give information about the stack ranking of the distribution of values to a proposition crafting process so that it could craft better propositions right so we're not saying that there are not awesome applications including maybe maybe the ai apply applications that are critical to be able to fix these Moloch dynamics because they can actually help coordination.

1:21:20But if you are not trying to actually understand the coordination failures deeply enough and say, what should we really be trying to solve in terms of fixing coordination that makes this whole technosphere compatible with the biosphere, compatible with human nature, compatible with meaningful definitions of human flourishing? If it is as powerful as it is serving anything other than that, the externalities in those areas will become increasingly catastrophic or dystopic or both. Are there any promising projects that you think are trying to harness AI in this sort of, again, I don't want to say anti-monic in this win-win way, or using info technologies in a way that is more aligned with what is actually good for human nature in the biosphere?

1:22:08I think we can see like Audrey Tang's work in Taiwan with creating digital democracy where they look for unlikely consensus and they're starting to apply large language models and things like that I believe and you know a lot of the work that say the Ethereum and some of the web3 communities try to do with public goods is obviously thinking about some elements of that and um it's not that nobody is but nobody with the giant capital in gpu farms and etc has that as their primary objective um and the primary objectives of all of those ones have molokian dynamics involved even if they also have some good dynamics involved and that's problem.

1:22:56And that's what I would most hope to start to shift is in recognition of the total power that is there, the total number of things that will be affected by it, the downstream end-order effects of it, the speed, the nature, the irreversibility. And there is a precautionary principle thing that who's right, Yudkowsky or Christiano on alignment or whatever it is. Well, given that most of the people who have studied AI alignment very deeply are concerned with the pace and direction with which we're moving, that's a pretty good sign that we should pay attention. And where there is disagreement between experts, but where there's radical uncertainty, but also maximum consequentiality and irreversibility, go as fast as possible is not the right answer what if there was a way to just remove the competition side of it entirely like i know it sounds very pie in the sky but like if all the major companies you know which yes let's say the companies themselves are by definition misaligned or like aligned with moloch but if the companies were to essentially hang up their competition hat not compete and work together as a sort of single entity, almost like a, you know, a collaborative science project, essentially, would that solve the problem?

1:24:22Fascinatingly, when we talk about the way that laws get built that end up supposedly having good purposes, and they do, but they also end up being part of MOLOC, like the fiduciary responsibility, the director or whatever. Currently, I think antitrust laws would try to prevent that thing from happening. um that you're mentioning and so the government would actually have to get involved but uh would i like to see the major uh ai labs together with the major academic researchers together with the regulators take very seriously uh short medium long-term futures of ai that factor all these types of assessments that factor the Molokian dynamics, the metacrisis, the way that the AI that is, the safeties won't totally last, right?

1:25:17That will get decentralized. The safeties will be taken off. It will be used for all purposes, the kind of omni-use nature of it. And to look at what does responsible movement forward in light of that look like? I don't see any good answers that don't involve that. Yeah, because that's the thing. The one thing Moloch requires, you know, one thing game theory requires is competition of some kind, some kind of either fabricated or real scarcity. And in this instance, there isn't really, you know, there are no intellectual powerhouses trying to compete to be the first to do a thing. Everyone is working on the same project together.

1:25:58It's the ultimate cooperation thing. um but yes it seems like a very almost intractable salute you know method because it would actually need to be an international collaboration as well it's not just you know i mean we do have the one advantage that it seems like the majority of the major players at least are all in western hemisphere and they all speak the same language and to a degree share similar values um you know Well, maybe not so much that, but that is one starting point. There are some very advanced AI labs in China. And the government-run ones versus the corporate ones are slightly different.

1:26:40There's obviously AI that is not just large language models for public deployment that is still also risky for many other purposes. I think the large language models for public deployment is a unique case because when the whole world starts utilizing them for the many, many purposes they will, it'll become nearly impossible to roll it back once we see the risks that are associated because we will have created so much economic and cultural dependence on it. so I think even if all we're talking about is as a start all the major GPU farms and large language models being in cooperation that would be a thing but I do think it does need to be wider than that and not just corporate and international and And I think the, I don't think there are any existential risks near term if we don't.

1:27:42I don't think large language models destroy the world. I think there will be problems and then we'll create solutions to those and whatever. But I think it absolutely accelerates the overall metacrisis across many, many vectors, which is why one has to actually take that seriously. to then really try to take responsibility for this thing, for which there is so much incentive to not take responsibility. Absolutely. Is there any specific call to action you want to put out there if there happened to be any major technologists listening to this? I mean, to begin with, it's really to engage with the wider risk arguments, not just in super intelligent AGI taking off, but the acceleration of all of the risks that the release of these things cause, to really engage with those arguments seriously and to really think about the world that's creating and be like, all right, despite the humongous incentive for me to rush ahead with this,

1:28:50is the risk calculus high enough, the speed and the irreversibility high enough that I am actually inclined to figure out something, to figure out better coordination around this, and to simply start engaging in more percentage of their total energy going into risk analysis than opportunity advancement, to have that more attached to the actual governance and choice making. I think all companies working on AGI should kill the fiduciary agreement to maximize shareholder profit. Obviously, some have talked about that. I think that should be killed. And I think the earnest real engagement between the AI labs, the people in AI safety research, and regulators should be a very actively happening thing.

1:29:46Don't be more lucky. No. You can print me one of those bumbersickers.