#31 - Kristian Rönn - Escaping The Moloch Trap

32:41Like it's way easier to game than if you have a completely decentralized system where everyone has a say and where the market can instantly react and like have a proper debate of you know what what what is actually happening and what are we actually valuing here so reputational markets is one possible solution to sort of essentially creating a new set of incentives yeah the long-term things that matter and not just optimized for the short term what other possible solutions are there i mean other possible is solutions like involve more centralized type of mechanisms so for instance I mean we might reform the UN like the problem right now with the UN is that they can't enforce like global laws like it doesn't matter if Putin is is like on the you know ICC wanted list or whatever like in the international criminal court says like arrest him like there's nothing they can do because they have no global police to enforce laws.

33:38And the thing about centralized mechanisms that make it so hard is that it's against the incentives of like a nation state to give up their power in the first place. But there are, however, some exceptions. And I think the European Union is like an excellent exception because with the European Union, like 14 % of nation states have willingly given up their sovereignty. And it has been stably giving up their sovereignty over time. Unlike the UN, which was formed right after a world war. And okay, we need to prevent this from ever happening again. It was formed after catastrophe. The European Union was also formed after the Second World War, to be fair.

34:23But it has constantly expanded because the incentives to join for a lot of countries are actually higher than not being a part of it. And the key here is the free market. Like for a lot of countries, like being a part of the European free market is so beneficial. So you're willing to give up certain parts of your judiciary and self-governance in order to join. The thing is that, you know, I think over time we might have something close to like a world government through you know a union of of unions but the time scales at which that operates is just so incredibly uh slow and i think one could argue that it's it's not gonna happen like at all i agree uh that said it's not strictly obvious to me that while there are benefits that come from for example having you know this very centralized powerful global um essentially adjudicator of the world including like an ability to police stuff uh is that that takes away like that that opens up the realm of potential tyranny yeah um extremely and that is arguably as terrifying in many ways to me because we've seen sort of like over oversteps like when because because no centralized agency is perfectly good at what it's meant to do and in fact often they're like the more centralized and like powerful they become the less error correction mechanisms there are within them uh and this that thus the more likely they are to go awry like you see that with like the who which cleared about as bad a job with covid as it could possibly yeah it managed to both fail to contain the disease and also completely destroy public trust yeah in important disease mitigation things.

36:23So we're arguably in a worse state. Shouldn't the next pandemic come along, which may be much worse than COVID, I think we're going to be even less equipped to deal with it because so many people are quite rightfully distrustful and pissed off with like centralized agencies around like public health. That's just one example. So really like a long-term solution I think that we need in that realm is yes having more powerful centralized agencies will help us certain things around resolving the tragedy of the commons but then that opens up this risk of like essentially so like if moloch is the god of decentralization do decentralized game theory gone wrong i there's the god of centralized sort of bureaucracy gone wrong yeah i like to call it this thing called norm um because it's like what's like what's the vibe if molly's like ah you know like you but i must win yeah they're using i must win this war and then screw the consequences that's like that's molly's vibe what's the god of of bureaucracy where it's just like this stagnant paper pushing like have you ticked this box oh you haven't and then like they you know they're very like rule stickly sticklery and um inflexible and oh this little dust devil's coming there's my titties yeah very uh got something in my eye um yeah yeah i call it the norm like it's like for northern yeah it's like uh how do we design a system that is takes the best bit of centralization but also the best bits of decentralization such that you know it's like a third attractor state as daniel schmack but i good likes to call it yeah um which minimizes the risk of of like game theory driven chaos and death by bureaucracy and potential risk of totalitarianism forever yeah what's the third option i mean first of all like the totalitarianism like really worries me because I mean there's depth of bureaucracy and stagnation is one aspect.

38:34I think another aspect is like good heart's law which is essentially like when you have a particular metric it ceases to become a good metric because you can sort of hack that metric. Right. And you see that in bureaucracy like a lot right. I mean let's say for instance you know policing in the United States. A lot of police departments it might seem super reasonable to have some sort of ticketing quota out to make sure that police are doing their jobs, bringing in like revenue to the police department and the county overall. But that leads to sort of an incentive structure where the best way of doing so is just to be super aggressive in your policing.

39:12And then it, you know, derodes trust, deteriorates trust in local communities and so on. And I think like on a global scale, it could be much worse. Like, let's say your only success metric is we want to reduce existential risks at all costs. Then we get something that, you know, like Nick Postrom calls this high-tech panopticon potentially, where, you know, you build a surveillance state where everyone is surveyed at all points in time to make sure that they're not doing something risky and you just have this like global tyranny, which would be absolutely horrible. Something like this wouldn't be allowed to exist but no exactly it's very like laissez-faire like it's a phrase safety third yeah exactly you're way too free here it's the most alive i feel at this place it's like yeah yeah if you want to climb that big structure you want to go 40 feet in the air without any safety measurements go for it i fall at your own risk you know um but that is you know the inverse of like this sort of like safetyist culture which um many people who are very pro-centralization want yeah and that is yeah extremely terrifying you know it varies from person to person some people find i think depending on your risk tolerance some people find the idea of like uh decentralized chaos more scary and others find uh centralized tyranny more scary um i think i find both yeah it'd be terrifying i want something i want neither you know um but I just interviewed Nate Silver who just wrote this book on basically the different appetites of risk and these different communities he calls it the river and the village and the village is kind of like the sort of the centralized establishment which tends to be very risk averse and more of wanting the safetyist culture and then the river he calls it is like the sort of mixture of gamblers and VCs and like techies and you know silicon valley yeah and like people who are very like they love calculated risk yeah or even uncalculated risk but they're much more risk tolerant and much more into decentralization so it's it's really again um it's like this this tension and like a lot of this sort of culture war we're seeing going on is this is this tension between those who want more centralization and those who want more decentralization yeah so it's like what is the win-win answer where both sets because both have some yeah are right about some things and are wise about something so how do we find a win-win between both yeah i completely agree first of all and i think like another sort of two communities that are fighting mostly online but i think that fight will have like real ramifications in the next decade or so it's like the effective accelerationists that are very much sort of you know the vc community mark and reason like i mean And look at, like, if we just have bureaucracy and centralization, look at what innovations it will prevent us from creating.

42:17I mean, we might not, like, use AI to cure cancer or, like, go out and colonize the galaxy or do, like, amazing things. We might not be able to have a reputational welcome. Yeah, exactly. We might not be able to have that either, right? Yeah. And then you have, like, I think on the other camp, like, the more risk-averse sort of effective altruists that are, like, we need to stop this, like, right now. at any cost, more or less. To be fair, I don't think it's just like a lot of effective altruists are not wanting to stop AI. No, I agree. But it's true that there is a subset within who are like very much like ready to stop AI.

42:49Exactly. I think the average effective altruist is like very pragmatic and sort of you'll see both sides. But then you have sort of very, very extremes of effective altruism, right? And I think you need to just be pragmatic about it. Like centralized mechanisms are better for some things. decentralized mechanisms are better for other things. And I mean, if you look at sort of the global economy and nation states, the states that perform the best is the ones that have sort of a mixed economy. And you have a very pragmatic approach of where do we need centralization? I mean, we might need it for healthcare and basic infrastructure and R &D.

43:28Like, where do we not need it? I mean, we don't need it in terms of like five-year plans, like the Soviet Union, because then it stifles like innovation and just creates, you know, perverse incentives through Goodhart's law. So more concretely, like how can you combine the both? I think like AI presents like such a good opportunity to combine both of them. Because if you look at sort of the global system of like artificial intelligence and the value chain of AI, it is actually highly decentralized. So, and what I mean by that is different nation states are dependent on each other. And especially China and US are depending on each other in a way that is not talked about enough.

44:12So now, for instance, you have US export controls. We, you know, the US don't want to give access to high end GPUs to China, because they might use that to create like more and more capable AI agents that can be used for like military purposes and whole host of things. So they are frozen out from getting access to the high-end ships. The interesting part here is that TSMC and NVIDIA and sort of the companies that are under the US sphere of influence, they're completely dependent on rare earth minerals and sort of mineral inputs into the semiconductor industry. And when you look at that, like China and its allies in sub-Saharan Africa has complete control over that.

45:02So that level of decentralization could create the right conditions where eventually China could say that, you know, okay, we're going to have like export controls against you. We're not going to give you access to all of these rare earth minerals. And there might be a trade war for a while. And then both of them might realize that, you know, actually we need each other to create sort of you know AGI and we might do so through some sort of you know these somewhat decentralized multinational AGI consortium where we sort of agree on okay let's build like an AI together because clearly we need each other throughout the value chain and supply chain it might be a little bit of sort of an optimistic and picture and there might be flaws to that argument.

45:49It sounds nice, but like, that seems like an unlikely outcome. Even if it were true that they do actually need each other, it feels like the incentives to defect are still so strong. Like, how would such a situation be immune to the same moloch problems? I mean, I think because there is this like clear interdependency, similar to like ourselves in our body need to coordinate. Like, you know, you can defect and that's called cancer but if you do so like you kill the entire organism but then cancer continues to exist yeah what happens right so because it doesn't it's not smart enough to see what is the long-term effects of its behavior is i actually want to take this in an interesting direction because i think like one sort of objection that i hear a lot is that global coordination and cooperation is more or less evolutionary inevitable like look at every major evolutionary transition that we have had like micro coordination yeah it's more coordination like micro molecules learned how to create the first metabolism then sort of you know RNA and lipid cell membranes created sort of the first protocell they coordinated and and and created like big clusters of cells that we call multi uh multicellular organisms we we include them right societies of multicellular organisms that managed to coordinate and cooperate won over those who didn't.

47:17Like, I mean, the most powerful states today are like, it's an amazing coordination machinery where we're all working together through rule of law, et cetera, et cetera. And the fact is that those societies out-competed those that didn't coordinate at all. So you can give sort of a false picture of, look at history, like coordination is everywhere. But I think what that doesn't take into account is survivorship bias. And to explain survivorship bias, there is this like World War II analogy. So, you know, during World War II, you had a lot of, you know, planes that went out on bombing skirmishes.

47:56And then when they come back, statisticians looked at all of the bullet holes in the airplanes to see like, okay, where do we need to put extra armor in order for the planes to survive? And it turns out that all of those bullet holes were around the wings of the planes. And then you might conclude, okay, we must put more armor in the wings, because no bullet holes were around sort of the engines. But actually, like the planes that hit the engine, they... I mean, those pilots crashed and died. So they never came back. And I think we are committing this sampling error globally, because planet Earth is just one single data point.

48:35Like we don't know what looks like in other worlds. Like there might be other planets where in fact, like multicellular life might form really easily. Like Nick Lane and other biochemists seem to think that. And I think they have strong arguments. It's basically just chemistry to create biogenesis. But then in that biogenesis, let's say thermal vents in the ocean floor, maybe viruses evolve really early and they kill all of life. Or maybe a super predator evolves really early before life can sort of, you know, properly diversify and kills everything. So there might be sort of these defectors throughout evolutionary history that kills all of life.

49:20So the question is, do we see the similarity to bullet holes in our plane, in our evolutionary history. And the fact is that we do. And I can't believe this is more, not more talked about, but like arguably five out of six major mass extinctions was caused by life itself. So essentially like bacteria or algae having a mutation where they're incentivized to just grow and multiply really, really rapidly. But that multiplication creates like runaway climate change or like acidification of oceans that almost kills all of life. And that has happened like again and again. So I actually think that life through natural selection is less stable than we think.

50:05And Bostrom sort of has this analogy about technology. We invent new technologies and it's almost like picking a ball from an urn where some balls might be white and it's clearly beneficial technologies, but others might have a dual use that can kill us. It might be AI, it might be nuclear weapons or something else. And it's actually blackball that can kill us. And I think the same thing exists in mutation through evolutionary history, in biology itself. But then isn't that just a question of timescale? Because we, yes, there are all these, we've had all these like six mass extinctions and so on.

50:44But if you zoom out far enough, life has still prevailed. Yeah. And you could argue that it's still on this trend of increasing complexity, even though it might have these sudden mass die-offs and so on. Basically, if you zoom out enough, it seems like it's a one-step-backwards-but-three-steps-forwards type of situation. But I guess your point is that we can only be here to say that in the worlds where we happen to have been lucky and survived. We don't know whether it's a true fact of life that coordination always exists or that we just happen to be extremely lucky on the one planet where life itself has managed to survive its own inner self-destructiveness.

51:30Exactly. I mean, we will always have a biased sample. Our existence of being here, asking the question. Exactly. And so I think the best we can do is sort of look at our evolutionary history, but also look at where are all of the aliens? Like, we don't seem to find aliens anywhere. And like, there's different solutions to the so-called Fermi paradox. Like, why don't we see life everywhere? I mean, if you look at population curves, they tend to go up. And cosmically, that should be the case as well. You know, it should be sort of an adaptive strategy to go out there and colonize the universe. And in fact, you could colonize the entire Milky Way galaxy within like a couple of million years, which is nothing in cosmic townscales.

52:14So we should see them everywhere. And one explanation might be, oh, biogenesis, it's super hard for life to form. Or it might be, oh, life always kills itself because of dangerous technologies. But I actually think that Moloch or like Darwinian demons and multipolar trap is just one continuous long filter. because we're constantly picking those balls through like mutation and natural selection. Right. Of just like basically a cancerous type species coming along and doing the cancer. Exactly. And I think it's actually worse than just picking the balls at random because I think we're sort of incentivized to pick darker and darker shades of gray balls.

52:56Like if you plot sort of, you know, the evolution of civilizations, you know, Oh, you had maybe some civilization back in Mesopotamia or modern day Iraq that invented iron and they could produce better swords. So they built an empire around that. You might have the Mongol Empire that realized that, oh, we can actually have archers on horses and then we will overpower everyone else and build an empire. So you sort of have these selection pressures for more and more dangerous weapons over time, from like swords to gatling guns and machine guns to nukes and now potentially like, you know, rogue AI or like bioweapons where you need to just innovate like worse and worse and more dangerous things.

53:47And there is a selection pressure to do that. I buy the logical arguments, but it's just like it's such a gloomy and seemingly hopeless situation if that were to be true. I kind of don't even want to like subscribe to it. Intuitively, it just doesn't feel like that's the case. If that was truly the case, that like basically it's inbuilt in nature, that it's like almost the odds are overwhelmingly against any one planet from actually really just persistently making it. Do you discuss solutions to that if it were actually true? Yeah, I think what gives me hope at least. So I have to admit that gives a super gloomy picture.

54:23And when I first got that idea, It was like back in high school when I played around, like I wanted to be a better Python programmer. So I built like evolutionary simulations. So agent-based simulations where you sort of had mutations, predator-prey dynamics, like virus host dynamics. And I'm like, whoa, life dies all the time in these simulations. Like why is that same thing not happening on sort of planet Earth? And then when, as I sort of deep dived, I'm like, whoa, maybe actually the same thing is happening, but we're just like extremely biased in, you know, the way we look at our evolutionary history.

55:01But what gives me hope here is the decentralization of things. Like, I think we live sort of in a system where everyone in some respect is dependent on everyone else. So like I have this example in the book, like, let's imagine you want to create a chicken sandwich from scratch. It sounds like the easiest thing ever to do a chicken sandwich. But let's say you don't, you can't buy any tools or raw materials from everyone else. Then you're basically back to stone tools. So you need to grow the grain. You need to figure out how to mill it. You need to feed the chicken. And there was actually a guy on the internet that tried to do that.

55:42And it took him six months and it costed him$1 ,500 to create a simple chicken sandwich. So it just is a testament to how extremely interconnected power actually is. So it seems like power is not interconnected, but that's because we don't have this global view of what everyone else is doing. But I think something like a reputational market could offer that global view where actually you know that if you deliver a certain component to someone, you know the reputation of that person or you know the reputation of the next company in the supply chain and so on and so forth. So I think that interconnectivity gives me a great deal of hope.

56:27What can people watching do to either find out more about reputational markets or to help build them and design them? So first of all, you can pre-order the Darwinian trap on Amazon and read the book. But then secondly, I think we need to spread awareness. I think there is this narrative where everything wrong in the world is due to a couple of bad apples. Like, okay, social media created a bunch of fake news and everything is Mark Zuckerberg's fault. But the fact of the matter is that if Facebook didn't invent the feed and sort of the pay-per-click model in a social media feed maybe myspace or someone else would have done it and they would have been the winner so it's not like we need to stop blame individuals so like don't hate the player hate the game uh as a famous you know hip-hop artist change it to don't hate the players change the game yes oh yeah i love that even more yeah let we need to all change we need to recognize that there is the game here and we need to go away from like scapegoating individuals and think about what system of incentives are creating this in the first place so you need to vote for like politicians that are not connected to this sort of simple narrative of you are good and you are evil so I think that's one thing that you know everyone can do uh but then you might be in a position of power you might be I don't know like running a pension fund for instance then you might have a great deal of power in terms of like who you invest in or who you not invest in you might run an enterprise you might have power of like your suppliers or employees and everyone you're you're connected to um you know as a politician you might even have a greater deal of power so i think like everyone has a little bit of power but i think the first thing we need to do is that we need to make sort of this concept of you know moloch multipolar trap or whatever we call it and probably we need to coordinate everyone who's talking about it and sort of you know use the same name I actually feel like as a side note it's like a little bit of a coordination failure on my part calling it like Darwinian demons because when I wrote the book like there were not so many people talking about Moloch and then like your TED talk came out which is like brilliant by the way and I feel like there's been a lot of more sort of you know popularization around that as a concept.

58:53But we need to sort of spread that concept to make people realize that, you know, this is actually the generator function behind a lot of the problems that we're seeing. Are there any like specific resources that people can go to now? Like, is there a particular group already developing these reputational markets through production markets? Like, is there a website people can go to? No, not at the moment. But my hope is by sort of you're presenting this idea and i've talked to a bunch of people who are sort of interested in developing the idea further one of the you should make like a subreddit or something oh yeah i think that that would be super cool you should maybe just make a subreddit of uh reputed r forward slash reputational markets you should just make that yeah i love that idea anyone listening christian's gonna make it go join yes i'll join it too yeah and then like let dialogue start and then maybe over time it can become a discord server or whatever but like i I think that's, you know, that's talk about decentralized solutions.

59:50Yeah. And that's a decentralized way of like, because, yeah, as you say, it needs a kind of hive mind. It needs a community to help make it. And then it would be a form of a hive mind. So just to finish up, last question. Why Burning Man? Why are you here in the first place? What is it that is so special about it? Because to me, it is a perfect example of a decentralized system. It's got the tiniest amount of centralized organization in terms of like they lay out, the organization marks out the streets. They provide a few porta-potties. And there's a medical center and such if you need it. But otherwise, it's like, there you go.

1:00:37There's the infrastructure. Now you people come in and there's a few basic principles. like I don't have to say rules but it's like don't drop trash um be you know inclusive and don't make your any don't build something that's only for a certain you know like special group of people let anyone be able to access and enjoy whatever you build and uh be and radical self expression you know don't become a monoculture like you want to be into hip-hop you can you want to uh be into metal you want edm like I mean there's a different flavors of music but it's just like different like styles of aesthetic um and yet just through this basic these these basic 10 principles and this basic little bit of infrastructure this like very decentralized thing does its thing and so much cool stuff emerges like just i've never seen something where i like i call it like the high density of what the fuck's per second yeah i have just like I'm like, what the fuck?

1:01:35There's like a guy wearing, like there's a T-Rex who will be part of the, there's a dog pushing a lawnmower. It's just endless moments of surprise. It's so fun. So why is it, what does it mean to you? Why is it a win-winny thing? Yeah. So I've been a burner for the past like 12 years. It has had like such a great influence on my life. And I think what it really highlights is that humans are like pro-social creatures. Like we want to be loved and we want to love one another and we want to do cool stuff for each other. So like in the ideal of like circumstances, like we just want to do good. And I think, but I also think there's like failure modes where Burning Man is sort of glorified.

1:02:29And what I mean by that is this situation is so unique. Like you have people that are organizing camps for like a year. Like as soon as they leave Burning Man, they think about how do I build the next thing. But there is so much resources. There is an abundance of resources pulled into this week. So you essentially have so much resources that there is zero competition for resources, with the exception of the bikes that I mentioned earlier, where you sort of have the clear coordination failure. So I think what it shows that in the ideal of circumstances where we have sort of resource abundance, we tend to cooperate.

1:03:09And I think like Burning Man sort of ties down to this sort of old debate between Hobbes and Rousseau. I mean, Rousseau sort of argued that, you know, the natural state of humans is that, you know, we just sort of want to be in a tribe and we want to be pro-social. And Hobbes argued that, you know, actually the natural state is that we will sort of kill one and each other. And that debate was renewed recently with like, you know, Steven Pinker releasing Better Angels on Nature, essentially arguing that, you know, our nature is actually not that good. I mean, we have a lot of suffering and really horrible punishments and wars and people clubbing each other down on the street in the past.

1:03:54And then someone like Noam Chomsky sort of defending Rousseau and saying like, actually, anarchy is good. We are all pro-social. But I think both are right at the same time. And it all depends on the resource constraints of the system itself. I think Rousseau is 100 % right. like if we have resource abundance, which I think to some extent we had like pre-agriculture with like, you know, smaller societies where we could like all live off the jungle and the land without any problem, without like rapid population growth. But with the advent of civilization, we went more into sort of the Hobbesian or pinker world, if you will.

1:04:35So I think Burning Man is just a testament of like, us humans are multifaceted And it all depends on the incentive structures of where humans are embedded at any point in time. Awesome. Thank you so much. Thank you. This was great. Yeah.