#35 - Allison Duettman - Cooperation, Technology and the Power of Existential Hope

0:00in a world in which, you know, we don't kill ourselves in the next hundred years, we will have emulations. I don't see a world in which that's not going to happen unless anything else really bad happens. And, you know, the space, like the virtual space is just so much raster than the physical space. I don't even think that we have a lot of conflict or like we don't have to have a lot of conflict over resources because we can just create really great rich virtual worlds with a fraction of the resources that we currently have. Hello, friends, and welcome to the Win Win Podcast. Today's episode is all about cooperation.

0:32I'm speaking to Alison Dirtman who has essentially just written the book on cooperation through the lens of what are the types of technologies and almost like philosophical changes that we need to make in terms of how we cooperate with one another in order to have a both safe but also free future. The bulk of this conversation with Alison, because her expertise lies in thinking about the future and like building viable worlds where we can have all the benefits of these incredibly powerful technologies, but without all the risks that often come with them. And so we spend a lot of time exploring the different possible mechanisms of cooperation and coordination that could be sort of in our near future through these emerging technologies like blockchain.

1:17So prepare yourself for quite a intellectual and abstract conversation. It's definitely a bit more heady than many of my other episodes. A large body of Alison's work is around this idea of cultivating what she calls existential hope. That's a large chunk of this conversation and a very, very important part that I want win-win to really focus on. So on that note, here is my conversation with Alison Dutman.

1:57Alison, thank you so much for joining and welcome to Win Win. To start with, I would love to talk about this sort of concept that you make a lot of your work sort of centers around called existential hope. So could you just to start with tell us what existential hope is and why it is a win win in your view? Yeah, well, I guess there's various different ways to approach it. First of all, I think I should say I didn't invent that concept. It's from a paper from Toby Ord and Owen Cotton Barrett at FHI. And they basically kind of made that observation that, you know, we care a lot about existential risk or at least in the little subculture that I'm a part of.

2:38And we often kind of like tend to get really kind of bogged down and thinking about all terrible outcomes and like an existential risk or a catastrophic risk is really like a risk after which the expected value of the rest of the universe or at least like you know any future after that is much diminished and we never really think about the kind of counterpart to that what would it look like to have something like a eucatastrophe which is like a word that Tolkien I guess invented and to describe the opposite of a catastrophe basically like something after which the expected value of the world of the universe would be much much higher so like an extremely good event where you can just agree like okay now we're in a much much better position for the rest of the universe to go pretty okay and so we should think about a these individual moments and events you know what are they but also just generally like if we don't kill ourselves and there is a chance that we might not then where do we want to go and not just because you know this is something that's possibly exciting to ponder, but also because if we only ever just run away from the scenarios that we don't want, it doesn't really bring us on a path to the scenarios that we do want, right?

3:45Like there's like a million different options and only ever avoiding the worst ones doesn't actually set up on a path out to the better ones. And finally, it's sometimes kind of exciting to think about what could be possible and it might actually increase the chances that we get there because if we only ever try to get each other away from a specific outcome it creates these kind of like loose-loose dynamics sometimes you know anything that this podcast is trying to avoid and where you actually have these zero-sum dynamics where you're just trying to avoid a specific thing um or slowing something down or um you know dancing around um a specific technology here and there but you never really think about okay well are there actually things that different interest groups could agree on.

4:27What would a future look like that's actually exciting from different perspectives? And could we actually get more collaboration towards these futures going, maybe even as a way to avoid the worst case scenarios? So there's various different, I guess, facets to it, but I thought the concept was amazing, super undervalued. It still is, but now there's finally products like this one out there that is trying to wake people up a bit, that more is possible, and it's not super unlikely that we get it. so going back to this idea of eucatastrophe that's like eu catastrophe um it seems it's funny because the other day i was thinking about like you know i was i just got back from hereticon where they had this thing the apocalypse ball and it was very fun and like it was kind of like a halloween themed you know people you're encouraged to come in costumes uh themed around the apocalypse and also obviously i went as uh well i had two costumes actually i went as moloch uh because And it's like, well, if anyone's to be blamed for the apocalypse, it'll be that thing.

5:24And then I also came, I had a Shoggoth costume. But I was like, I was like, well, this is fun and games and all, like dressing up as these, you know, leaning into this idea of apocalypse. But at the same time, it actually made me feel quite uncomfortable because I do think there's some truth to this idea of like what we dream up is more likely to happen. you know and if we spend so much time always visualizing or making art about catastrophe and um these dystopian these dystopian futures and I mean if you go on like any any streaming channel right now the the ratio of like dystopian to utopian art is something like I don't know I mean it's definitely worse than 10 to 1 it feels like 99 to 1 I can barely think of any like look Black Mirror the TV series right that had I think one or two out of out of the 20 or so episodes they made positive.

6:19Sanji Rupero, my favorite. Yes exactly Sanji Rupero was the only optimistic one the rest were just these horrifying examples of where like technology can go horribly wrong and so well first of all like why do you think it is that it's so much easier to like visualize dystopias than utopias? I think there's multiple different reasons. One is just, you know, it was probably evolutionary advantageous. At one point, you really try to avoid the worst case scenarios. And existential hope is not saying that we shouldn't try to do that. You can still have this mindset of like maxi-poc, like maximizing the probability of an okay outcome or like de-passimizing.

6:57But you can think that the way that we best maximize the probability of an okay outcome might still be by pointing out that there's also good outcomes so it could also just be an instrumental strategy but like apart from the fact that maybe evolutionarily it was like pretty good to always think about where the next predator is um you know it's just something that like you know keeps getting ingrained by our culture it's just you know that much that much easier to create an interesting narrative arc by which there is something that um you know creates like an extreme conflict and those are also to some extent like the kind of media stories that we get fed over and over again and one thing that um you know we can kind of like put our finger on is like this notion of like an outlier aversion or like an outlier inversion like it's basically like a concept ish that means that civilization is already doing pretty well like you know we have relatively high living standards um we have relatively good education we have relatively little violence across the board and relatively good health care and so these things are not super interesting to talk about anymore because you know they're kind of like omnipresent not everywhere but like at more and more places and so what actually kind of stands out are the exceptions so those are the things where it doesn't get right and so those suddenly seem like they're much more prevalent just because they are the things that are interesting to talk about there were a few concepts for a while like a few efforts that were trying to kind of go against that by creating like news channels that really just like told you of like how many more diseases we eradicated this year and how less war there was even though that's probably not quite as true anymore but you know it's just hard to create to create news around that i don't think it's impossible but if i look at a few of the kind of existential pieces you know that i like like the favor of the dragon tyrant for example is one you know from from nick bostrom which basically it um it compares death to a dragon that um you know a civilization always has to sacrifice and a certain amount of individuals too every year to keep going and they never question why until one day this kind of like child comes along is like oh should we actually be doing this i don't know can we fight the dragon maybe and for a while uh they don't really um get the act together and eventually they do and then now spoiler alert you know after they have actually won and fought the dragon um and really made some sacrifices the king is not happy but the king's kind of sad because he thinks about all the years in which I didn't find the dragon and in which they still were sacrificing people to the dragon the dragon signalizing death and that's to some extent a hopeful story or like it at least gets the point across that we could be doing something about death but a much better hopeful story I think um even better than that Bostrom's story was good but a better one would be think about a world in which it's absolutely normal not to die of old age like that's just the baseline of the story drop right like we just walk around and um you know unless we don't kill each other or die of like you know other causes um we just you know age like to an extent where there's always better technologies available to at least give us the option to continue to continue to live if we wanted to and then the threat that kind of comes in that creates this narrative arc of something terrible happening is actually that um death comes back so you know like suddenly it just so happens that people start dying again no one knows why and so you can create kind of like narrative arcs in extremely positive scenarios as well you just have to think about it right a bit more it's a bit harder to do and so then the whole storyline would be about okay how do you now go back to the state where it's just absolutely normal or to die just to get people that understanding of like if we had a different default understanding of civilization um you know that's much more positive than anything that we currently totally accept as a given might be absolutely atrocious.

10:48So it just takes a bit longer and it's harder to do than I think the current media scenarios that we have, but it's possible. Right, because it's tricky when you're, I just read this amazing piece by Dan Harmon, one of the creators of Rick and Morty, which is like, it talks about, he calls it like the story circle and how you need for any good story to actually hit home, whether it's a movie or a book or any quick example of a story that makes people listen and recognize as a story that they want to listen to. It has to go through this kind of like challenge arc of like the hero who then like everything is OK, but wait, it's actually something slightly off.

11:30And then they sort of go in this like descent of some way where then there's this big challenge and they have this awakening, but then the awakening is not enough and they still have to go through a big challenge, which is a climax. but then eventually they sort of return back to where they were but with but with new insights and if you almost every single movie whether it's like a rom-com or like the terminator has this kind of arc that it follows um and so you you need to have a way to tell a story of a challenge by definition as a you know a utopian world if you build it where's the challenge i guess but that's what you've just said actually is fascinating because you're saying that like you can have what would be relatively utopian compared to today and there's still going to be challenges so I guess that now gets but you know coming back to Bostrom who I interviewed for Win-Win like a few months ago and you know he just wrote this big book on utopia and like trying to wrestle with this idea of like is it possible to actually ever be like happy without some kind of challenge or like is there you know when we have every single thing provided for us where if we you know basically the idea of challenge itself would only be opt-in so I wonder do you have any thoughts on like do you think challenge is an essential part of like human flourishing or do you think we can ever become post-challenge I mean you know to some extent challenges also like they change with it's like in a video game I guess as you get stronger you know like the challenges um also like adapt to your skill level at that point and i think you know he in boston at least he mentions that like well worst case people can wirehead in a useful way but like then again we can also think about much more exciting deep utopias which just define why define wirehead quickly for people oh basically like you know you kind of like lock yourself in like um neurologically to just like experience extreme bliss at like anything that you do um or you could even like lock yourself into like experience deep meaning like basically become like an art historian uh and love it uh and just have also the faculties to be able to appreciate art in that way but he also you know then goes further and says challenges are still exciting and i think that to some extent that ties back again with this like notion of sorry that i come back to the example of longevity but it's just so prescience because um there's individual longevity and there's civilization longevity and i remember um when i was a kid i um i really i didn't want to die from the very first one of my very first thoughts but i just couldn't get over the fact that we might have to and um i couldn't really like understand why not everyone else cared about it but i think one of the reasons is that we just don't think about like we like the the most that we do or tend to do in our main lives is like consider okay how can we create meaning within these like artificial constraints of like 80 years but we rarely ever ask about like what would it mean to like lift these constraints and i think existential hope is all about like asking what would it mean to like lift the constraints that we currently think as like entirely um almost like god-given or laws of physics what would it mean to like lift those up and instead of like trying to adapt our meaning making to artificial constraints what would it mean to like lift these constraints upward and to expand them to fit the kind of like life goals that we have.

14:48So for longevity, one could say equally like, well, you know, really you have one narrative arc and your narrative arc kind of like starts when you're young and then you make some investments. What's your future self? Eventually you cash in when you're like 50, you're like at the kind of like 40, 50 at the top of your life. And then later on, well, you know, you just kind of decline. So it's this like narrative arc as well for your own life. And it's like, what would that possibly be? like afterwards it's just death is the only chance or like the the only kind of like rational outcome but i think that kind of like misinterprets what life could be you know life could not just be one story with one narrative arc but it could be like a large book with many short stories so like you know as long as you're physically capable you can have like one story story arc in which you know i do the job that i currently do in the next one um i dedicate all of my life to animals and the next one after that i only go explore the world and the next one after that i finally read all the books I haven't read in the entire time.

15:41So you can make up very many different challenges or different narrative arcs, especially if you think about mind uploading, you know, like when you land into, in worlds like Age of M or Greg Eggins and Permutation City, the sky is not even the limit here. It's literally like computational constraints that are the limit. There's a, you know, there's often a lot of pushback to the ideas of transhumanism, extropianism.

16:09I mean, even like this idea of like that we should be dreaming big like this. And I want to guess I want to try and steel man those people's perspectives. You know, I think, you know, there's a there's a naturalistic argument of like that death serves a purpose. And like it allows, you know, from an individual's perspective, yes, death sucks. But from a civilizational perspective, actually, it keeps the organism healthy because it's like old ideas die out. it gives room for like new people etc um so yeah like i guess what what's your sort of steel man of their views and then what's your response to that well i hope i can do a good job at steaming them um i it's never totally um honestly computed in my head so i'm not sure if i can do a good job with it honestly but but i'll try i guess like one of them is really well we have just dictators forever and that's more like a you know that's a societal constraint and we can work with that you know like we can we can figure that one out um the other one is often that i hear is overpopulation and i mean as i think that many people kind of wake up to right now we're probably going to have the opposite problem very soon but it's still uh somewhat in people's people's heads impressions the other one is just well will we have like an erosion of like novelty new ideas um you know like old has to make way for new like a bit the point that you just mentioned of like you know in order for us to progress that has to be the case but again i would go back to the fact that it's not going to be me the way i currently am if i live a very long life and if civilization gets very old like what i become shifts and shape shifts and and will so i think that the way that we currently imagine that future of like not just super longevity but even you know like neurotechnology etc is like very much rooted in how we currently live but like with different ways of how i can like you know morphologically change physically where i would live um the friends um that i'd make the places i would see like people just change or they would have the capacity to change a lot the main idea here is that you're in that world um not your 80 year old self but like you're a young self so you have still brain plasticity enough to just become different versions of yourself um and you know i guess the main thing is just well will there be enough space resources and capacity for those people to exist and i guess if we think super long term in a world in which you know we don't kill ourselves in the next hundred years we will have emulations um i don't see a world in which that's not going to happen unless anything else really bad happens and you know the space like virtual space is just so much faster than the physical space so i don't even think that we'll have a lot of conflict or like we don't have to have a lot of conflict over resources because we can just create really great rich virtual worlds with a fraction of the resources that we currently have so this kind of like notion of we're going to create a race to the bottom and scarcity and and we just like you know everything will be driven towards competition and we won't really have meaningful lives anymore is i think because that is to some extent uh the constraints that we have lived on under the methusian dynamics but it's not the constraint that we have to live on for a super long time at least the subsistence levels or the kind of levels of competitive equilibria in which we're all going to race towards on the very long time scales they don't have to be super low um we can like make those levels nice you know we can have enough technology nanotechnology or virtual technology to just create like abundant living environments in which people can strive i mean at the same time like you're talking about this idea of like if brain emulation becomes viable and thus actually most sort of sentient beings will be existing in that type of environment.

19:56I'm not sure I quite buy that because in order for it you know for these brain emulations to become this is talking about like Robin Hanson's style of Age of M right in order for those to become a viable option in the first place it's probably because there's some kind of economic benefit for it you know it's done because we need like there's there's some kind of economic activity that's going on that they will fulfill and I worry that given sort of the trend the current trend of the economy which is to basically find more and more efficiency at least that's one of its major optimization functions that like won't that kind of like be taken to like an extreme outcome of that so you're almost getting like trillions and trillions of these emulated like sentiences but they're they're like they're almost as simple as possible such that they just do their little their little tasks whatever their economic tasks are it doesn't sound like a very good life for those things assuming they have some kind of sentience so it's not clear to me that it would be this like rich utopian world it's like the you know if if it was born out of like an economic drive in the first place yeah well i mean i don't think we're on a deterministic path to an amazing future we don't have to do anything anymore like i totally hear you and um like all jokes aside like i think the concerns that you mentioned are concerns about like sentient risk of other minds you know we're not really good at um accommodating for other existing sentences uh that already share our planet right now like we're pretty bad that are you economic that are economically useful to us such as animals in farms right animals yeah animals they ain't doing great um and you know that's one thing and they're already here so that's not really super promising is it um and then there is a lot more research currently being done in a in a pretty serious way um at looking at alternative mind forms so that includes artificial intelligences um that we're currently building but it also could include things like many of these like brains on chips etc like basically like these chimeras between like biologically computing enabled minds.

22:08And so I think there is currently a lot of kind of work being done in a pretty serious way to try to avoid futures in which we create what could just be called one of the worst risks of all time, which are not even existential risks, but suffering risks in which these entities that we're creating suffer. And this type of suffering that is at least theoretically possible to inflict on them might be much more astronomically larger than the suffering that we already experienced just because we can like minds are adaptable right um so it's i'm not saying this is not possible at all this is definitely a future we can't be polyanish about and we have to avoid on the flip side though first of all i would say the only reason to create emulations is not just economic one i guess economics is everywhere but one would just be for digital immortality and longevity like one might just literally be you wanting to have an emulation of yourself.

23:00I guess that just has some economic value, but you're not just creating it for improved economic value, but possibly also because you won't have fun in that world. There will be economics involved in that for sure, but it's not necessarily that you want to have your emulation be maxi-economically productive in the time that they're emulated. Possibly. I don't know who you are, and I don't know what emulation you create, but knowing you a little bit, I doubt it. I think you'd be rather nice to the emulation that you create. And then secondly, you know, like, I think even the emulations that will be created to produce work and be economically viable, I just hope that we'll have a world in which we can, like, be more specific about whether or not these entities are actually sentient.

23:42And for the work that is, like, drudgerous, it's not necessarily evident that we have to create sentient emulations or, like, sentient M's for that at all. like you know we could just create like you know automated entities that are much much better at automating specific tasks that we're doing there's no reason for us to create them in a sentient way that they suffer during this i mean it's like an abhorrent thought why would we do that and then i guess for emulations that are you know created um we're eventually you know we spread out as a lot of evolutionary pressure and like you know eventually our idea even of what is sentient and uh changes there might be borg minds there might be human ai symbiotes whatever like you know this now we're like in super speculative la-la land here but like even there i would say that our idea of what counts as sentient will probably adapt um and then hopefully you know we uh will just create like um precious forces and other types of rights for these emulations um that create just an extension of the current economy in which more actually existing sentiences like us have rights not all of them not animals but like creating that kind of like new infrastructure new also allows us to create better initial conditions in which we like get more precise about sentience and really actually allow the entities that that exist to have rights and to live according to those rights and so i think this kind of like um notion of you know it's it's it's just all left to evolutionary forces like i do think we have an opportunity to shape this and that opportunity is kind of now or is at least pretty fastly approaching um we're already in some kind of grey zones at some points and yes I think we just can't be complacent now.

25:25Tell us a bit about your work at Foresight Institute. Actually tell us what Foresight Institute is first of all and then specifically what you're doing there in terms of like world building. Yeah well Foresight Institute is an org that existed since 1986 so it's been around for quite some time and I found it actually online when I was still doing my thesis like I guess that's now 11 years ago I was on AI safety and I was like looking around on the internet like trying to find other research in that area it was like relatively early there wasn't all that much on like AI philosophy in particular and then I found a few kind of like meetings that Fawcett had on related topics and it was all online and so I just like kind of stumbled into the rabbit hole of our online archive and couldn't believe that there was this kind of like organization of people that like cared a lot about the long-term future while also mostly working in science and tech being optimistic without being polyamorous about the risks of various technologies so I thought it was like just really inspiring and I cold emailed them pretty much and then for the last I guess 10 years I've been in the Bay, which is where Fossite is housed.

26:37But basically, Fossite was this kind of, it's a nonprofit, a relatively early shelling point for people that cared a lot about technological progress and what they could enable for society and life in general. And founded by Eric Drexler and Christine Peterson, based on the book Engines of Creation, which was this book on what can we build with nanotechnology and AI in the long term. and you know in the past years it has changed a lot you know it's a pretty long-standing organization but we mostly give we support science and tech that is like too early stage too niche too ambitious too interdisciplinary for a legacy institution to support and so we give grants prizes fellowships we host workshops events etc in like longevity biotechnology molecular technology neurotechnology and ai mostly those are like our big verticals i guess and in each of these we have like specific uh strands that we think are undervalued or at least that are too weird to early stage for other orgs to uh to take them on and so we focus on those and you do these kind of um world building almost like hackathons can you explain how those work yeah that's basically part of the existential hope uh project and existential hope lens is kind of like if we have these different technological verticals like longevity biotech, neurotech, AI and nanotech like as our like technical tracks in which we push technology forward like the question is kind of like okay we're racing forward but where do we want to race with these technologies and that's where the kind of like layer of existential hope kind of for us comes in as almost this like guiding north star of like thinking hard about various different positive paths for these technologies but also about possible risks and so we have this entire website existentialhope.com in which I like fanatically for the past seven years or something collected links resources podcasts videos anything really on positive futures and like put them into different libraries and it also relaunched I guess last year this world building course and world building hackathons because the idea kind of is that it's sometimes pretty difficult if you're working on a specific technology to actually see where this could go especially to see where it could intersect with other technologies and like with other social social political changes that are currently happening so it's like really difficult to actually think hard about what a future could look like there are prediction markets those are great uh there's sci-fi that's amazing um but i I think actually looking at a few different technological strands and figuring out how do they intersect and what are some positive outcomes of those and how does that match with current political and economic trends, that's difficult.

29:25And so we're just getting people together that are working on different types of technologies and different types of social issues and help them kind of update each other on what's happening and then get together to build worlds that kind of contain, like that are a bit more enriched by what's actually happening in tech in the next five to ten years and so those are the world builds and hackathons are basically then to design actually institutions that could help us get there quicker so once we're in these worlds we usually create like a day in a life and we create like a timeline of how did we get there and the timeline usually if you cast it all the way back there was eventually some kind of institutional change or some change in the way that we interact with each other.

30:10And that's the kind of like hackathon part of like figuring out other interesting institutions that we could already build that make that path a bit easier. Is there a specific example of a world that you really liked that someone designed like a day in the life of? Can you give an overview of one? Yeah, well, I will not do that justice now. So one of them I thought was interesting because there's this piece I love from Scott Alexander on atomic communitarianism and the archipelago, which basically means, okay, we are getting more and more polarized in our worldviews. How bad is that actually?

30:52Well, it's bad because the values clash a lot in unuseful ways and we try to impose our values on others. But if we had this kind of archipelago in which different islands were living, the islands are kind of signaling signifying communities in which different like-minded folks could kind of like get together and create their own almost you know autark ecosystems and then we account for externalities and for war and mimetic interference between these islands um that would not be a bad world uh so there can be a lesbianotopia christianotopia etc and then you have like free movement between them if they if you agree on the specific um rules that govern these uh different different utopias.

31:32So that's, I think, one really nice utopian vision of value diversity, actually being able to retain it. Another one that Another World took on was Eric Drexler's notion that he developed together with Mark Miller, who we wrote this book on Getting the Future on, of kind of Pareidotopia or Pareidotropism, which basically means that if we extrapolate forward and get to a world in which we have more automation through AI but also possibly through nanotechnology so like you know much more physical abundance at that point it might become actually much more exciting for people to cooperate because the gains from cooperation are just so much larger so even if you know currently we're like fighting over a very small pie often and because the stakes are so low we get really into it because we don't even have that much to gain and so we want to make sure that what we gain is uh is the most but like imagine a world in which you know even if you like you know you offer me a deal in which you get 90 of uh of like you get 90 of something that you want and i only get 10 of what i want but the 10 because of automation and um uh and uh and physical abundance that that part is still so large for me that it would really lift me up that's a world in which I'm much more likely to just accept a lot more cooperative deals and so you know that one of the worlds played on this notion of periodotopia but really mapped out different technological strands and how we get to that world including a nanotechnology strand etc so basically like they're rich in terms of the technological timeline of how we get there but also then what kind of new social systems does that enable how could we live in those Can you define exactly what you mean by Pareto-topia?

33:19Well, yeah, I guess there's, maybe I'll just define the concept and then I guess why I'm so excited about it. There are also different facets of it, I should say. But in general, basically, it's just a notion usually used in economics. And it basically means that, you know, it's a change. And by which, like, you know, if you and I, basically, it's not quite as good as win-win. like win-win it's basically like you and i um we cooperate and we both are much better off that's awesome this is the nonplus ultra definitely the gold standard but it's not all there is so basically oftentimes we are in these scenarios where like you know we just can't really make progress but like what if we could like find more of these scenarios where at least one of us gains like if you gain and i don't lose that's a pretty good outcome too it's still a positive it's still a positive sum outcome the pie you know the overall sort of value of the system has increased even though you know one party technically is neutral to it yeah but like neutral is fine you know if like if you get ahead and i'm not worth off for it that's kind of nice you know like overall the system grows and like uh and it says that the connect the system is like climbing operator preferred hills because it's not like that you and i are the only people in town you know like let's say you and i meet we cooperate you get ahead i'm happy for you then i go to the next person we cooperate i get ahead um that person is perhaps left off neutral um that overall the pie grows because there's so many like there's seven billion or more like people that we can cooperate with and especially in a world in which there's like where it's much more easier to find good cooperative partners um you know and offer them better deals etc like and all of these costs are kind of coming down right now and we can end up in a world that is much more cooperative in which people overall get better off each by their own standards just by the fact that we enable private or preferred cooperation.

35:14Of course, I'm not saying that win-win, like win-win is still great. You know, if you can do win-win, that's awesome. But win-win is not always possible. Yeah, I mean, that's one of the reasons, because the thing is, is like, you could argue that our current society under, you know, okay, yes, there's capitalism is definitely externalizing a lot of harms to the future that might be coming up, you know, they're sort of accumulating and might come back and bite us in the ass, you know, blah, blah, blah. But aside of that, you know, it's clearly like most capitalistic interactions are positive some in the sort of medium term, whereby the pie is growing.

35:51And yet, because they're not always obviously mutually win-win to the individual in that moment, you get these like feelings of envy and of people, you know, like right now there's very much an anti-billionaire sentiment. And I can totally empathize with that. And I say this in the best example of a billionaire that is, they become a billionaire because they build a brand new business that creates all this new cool stuff that actually benefits lots and lots of people in terms of their interactions. But because those people don't immediately feel the benefit, from their perspective, it's like, well, because people are so sort of stuck in this idea of like, someone else's gain must come from a loss in other words it's like zero sum mindset um that they even though the pie is growing and the system is like technically positive sum people are still insufficiently satisfied from that um and so that's why i'm like i'm trying to understand from people with this podcast of like how can we have it so that things are so obviously win-win that they're like this is great i'm happy with this system like we are trading like this this is good um do you have any like thoughts on like how like what are some like tactics that can be you know what changes could be made to um these kind of like technically purito uh optimal type interactions whereby people feel like they're actually directly gaining yeah i guess uh first i say even if it's just period of preferred you know like it doesn't have to be all the way optimal but um period of preferred that that's pretty good i think even just making people aware of these dynamics um and that you know that like zero-sum the zero-sum mindset comes from this like place of scarcity that doesn't necessarily hold in the future and making them aware of like the overall um gains of civilization i think is useful like for example i think it really like to me the whole principle um or like of Pareto preferred corporation um is interesting from like two perspectives one of them is like philosophically and one of them is almost like historically or like that it just happened to work out pretty well like philosophically my background is in philosophy and ethics and um I think it's just I don't think we will ever as a civilization all agree on one value function right and we'll be able to like all pull on one string and and like agree like this is the future let's go and like i am not personally even consistent in my own value functions meanwhile you know i have specific intuition intuition situations i like change all the time i try to get more coherent not perfect meanwhile everyone else starts from the same position right they have through their genetics through their upbringing uh through the culture they have specific types of values that they're trying to um that they're trying to go after they have different types of preferences and like it's just unlikely that they will ever like ever really converge and so we can't really look at civilization as a whole and say how do we move forward let's just pick that one thing and let's all go there instead what we need to do is kind of like put look at the system you know what's a system that we can put in place that allows for value diversity because we already have value diversity it's not going to go anywhere and if anything with technological change increasing we might get more value diversity over time because technological change leads to like more cultural change over time rapidly and leads to us becoming very very different perhaps in what we value and so we need to have like a system in place that philosophically even just from a meta-ethics perspective of like thinking about like what's the right system to put in place that accounts for the fact that people just have different preferences and i think this like notion of Pareto preference is actually awesome because it allows that you know you're only um you basically you're left to your own devices if you find nothing to cooperate with someone on or you can cooperate to at least the Pareto preference like at least the the advantage of one person um if you find something to cooperate on and so this is this notion that we kind of like introduce in the game in the future book it's like voluntary cooperation and so basically not everyone needs to cooperate with everyone else it's fine many people you know like there's just not that much of a kind of like vector overlap um and that in that case you know you should only have these consent-based interactions to actually figure out that you know people only cooperate because they actually think it's in their own interest if i don't know actually what's right and i can't look into your head of what's best for you then it's a pretty good guess that if you kind of consent to an interaction that's the interaction that you actually expect to benefit from by your own standards so that's great and so if we have the system of both you know the kind of like peaceful coexistence but also cooperation for like at least benefit of one that just appeals to me from a philosophical perspective because like if you think about ethics and how we actually live a good life and how we live a good life to each other with each other I think this notion of Pareto preference and this notion of voluntary cooperation is a philosophically appealing concept.

40:51I don't know if you want to say that. I could say another reason why it's interesting, but that appeals to me for one reason. I don't know if it appeals to everyone. Well, before we go any deeper on that, perhaps we can just zoom out a bit. Because I imagine some people are skeptical of how much of an important role cooperation has played thus far in society. You know, I think most people watching this show are aware of like how important competition has been in our evolutionary history. It's like one of like arguably the most, well, one of the most important driving forces that has sort of created, you know, made evolution.

41:30You know, the evolution thrives upon competition, the survival of the fittest, et cetera. And that creates these iterations and improvements. But cooperation is also an essential ingredient of that. So can you sort of explain how you view cooperation has driven civilization to where it's gotten to today and perhaps give some examples of some of your favorite examples of clear, obvious cooperation? I mean, I would say, first of all, yeah, competition is prevalent, but oftentimes we compete on better cooperating with each other. and so you know competition for example in one part of the economy is basically a bunch of different companies trying to figure out how to offer us better cooperative deals in which we can come to uh to an agreement so to some extent like you know competition isn't all that bad it's like the flip side uh sometimes or like it's one of the flip sides of corporation it's basically like you know if we live in a very large civilization in which like different people um get better at cooperating with others um that's the type of competition like that's good like i would say that's quite healthy you know if we're just competing at better cooperating with each other that's not not a bad strategy but i think like you know like i'm not a historian and um and while i love game theory i'm also not like a professional game theorist but like over time i think you know if you look a bit into our past and like many other thinkers and writers have pointed this out but like you know over time at least if you kind of like buy this idea that like roughly violence has like to some extent decreased like right now we're in a little bit of a um you know of a brink in that trend but like over time you know we have gone much much better at like not coercing each other to do specific things that meant that over time you know like being less coerced into situations and being less forced to do things, having more rights etc.

43:27That entire process also leads to just cooperation becoming a much much more viable strategy and becoming so much more attractive. So basically if me coursing you to do something is not okay anymore, then I have to get much better at cooperating and offering you possible deals that you'll agree to and so there's you know for example like one notion and i'm not sure if it's true but like you know maybe one way in which empathy goes supercharged is the fact that you know me thinking about what you might want um makes me much more efficient at offering you something that you might want and so over time this these kind of forces of like decreasing violence increasing like voluntary cooperation increasingly like only um um kind of like agreeing to the and to the deals and and scenarios that you find like actually useful that has kind of like shaped civilization to an extent where you know we now live in this like incredibly rich um societal infrastructure that has been shaped by these more rich ways to cooperate and at first maybe it was just you and i cooperating and then me and someone else and then you know bilateral trades so we'll get you very far and so eventually we developed all of these systems of like you know rights then on top of those we built property rights which are like you know the rights uh to do things really using specific resources.

44:45We developed a system of money, of prices, of like much, much more kind of like just complex institutions that like allow us to package these rights. But like overall, we're just getting a lot better at cooperating with each other through these incredibly abstract mechanisms that don't just allow bilateral cooperation, but like allow pretty much like a lot of people to cooperate with each other through these market abstractions that we've built up through an economy. And it's absolutely mind boggling that we've made it this far and like the complexity of the systems that we've made. But like now that we're here, I think, you know, we can get much better at cooperating.

45:22We have, you know, these like, you know, Moloch traps and multipolar traps, etc. that like we have to learn how to avoid. But like we're already at a pretty good position right now. So we can definitely do better. In your book, you talk about the sort of current hindrances to voluntary cooperation. You've got centralized power, the problem of different values, which we've already touched on a little bit. And then, as you just mentioned, these like moloch traps, which make it very hard for people to cooperate because there's always an incentive to defect in the short term. And then we've also got various new technologies on the horizon, which may actually make it harder to cooperate, too.

46:10so I'd like to kind of actually go through these four points as you know fairly briefly so can you explain first of all why centralized power too much centralized power is a problem to voluntary cooperation yeah well I guess you know to some extent it's always like almost like an offense defense that always shifts you know like we have times of more decentralization and then there's like different layers of centralization that we build up in more decentralized layers and that's how we to some extent like evolve um as a civilization but um you know in general if we think that you know one reason why we have a decline of violence and we have a decline of coercion and we have a the ability for most of us to peacefully coexist is because it is in each of our own interests to uphold this shelling point um can you define shelling point oh well it's basically um i guess like a coordination point that we all agree on without actually having to communicate on it so so can you give an example um i guess oftentimes what the canonical example is like you know if you and i wanted to meet up in new york but we would never set a date or time um you know we might meet at something like the empire state building at midnight or something just because that is in each of our heads the most likely point to converge on but we can have these points also you know in like more complex environments that just like tend to for example another example of like a shelling point is often like rivers or mountain ranges they kind of like separate different countries from each other just because it's like a natural boundary that that both kind of parties can like immediately agree on that it just exists and it would be kind of like hard and artificial or would almost feel arbitrary to instantiate it somewhere else and so I think this notion that we mostly cooperate based on consent has become almost a shelling point because it's in each of our individual interests to uphold it and so that's like it's kind of difficult to understand but like you know if if we're in a dark alley and I really want something from you and I know that that specific example I could much easier get it by, I don't know, just forcing you to give it to me.

48:25I really like that the little Pac-Man behind you and I'm just coming and like cursing you to give it to me. Well, that might be, you know, possibly useful for me in that scenario. I can probably do many, many useful things with it. But over the long term, it kind of creates this notion on this erosion of the fact that, you know, I would have to ask you for it. And so the next time, you know, like you might do that to someone else, they might do it to someone else. many people will see that now there's this erosion of like hey we ask for things and we don't just curse others to like you know give us their property or like or do something and so like this notion of like you know we cooperate through voluntary consent has emerged over a long time and it's in each of our interests to uphold it even in scenarios where i would otherwise short-term benefit by not upholding it and in order for that to matter it also matters that we're like in a relatively multipolar or decentralized world because we can hold each other in check um and and that's kind of like many of the systems that we have evolved from the constitution to like various international agreements are basically different actors that are checking on each other that they're like not going into um a scenario where one of them imposes their will on everyone else and so you know the constitution is like you know um three-way um three-way structured this way but even like you know we have the federalist systems or like systems of like different states um that are like you know also keeping a check on the on the centralized power of the kind of like overall government and and the whole reason why that's i think important is because if you didn't have that and there was one entity that would have an immense amount of power um like one centralized entity with a lot of power it's not really in their interest anymore to uphold this notion of like a consent-based cooperation because they don't really lose that much by um but affecting you know they could now just go in take something from you um without really fearing very much um very much repercussions to it and so it's important that we have these like that we have many different actors that can hold many many other different actors in check decentralization is not the only uh necessary sufficient condition for that to occur we might also have a totally uncoordinated state of anarchy we need to actually create these structures of and systems of checks and balances um but we can do that and i think it is the most it is the most resilient strategy i think to uphold a system in which cooperation is kind of like the the preferred point yeah because that that's i think you know reading through your book gaming the future which i'm going to link to um in this and i recommend everyone watching this conversation do go and read it's on sub stack it's not that long of a read um because really what it is getting at is this idea of the need of what often like Daniel Schmachtenberger and people like that call this idea of a third attractor because right now you know civilization is kind of being pulled between two forces both of which are actually not what we want as an as an end state you've got on one hand the idea as you know we we develop these more and more powerful technologies which can be either dual use or even just purely like attack biased and just dangerous so say for example you know the ability to make a very very deadly you know pathogen technology is it's getting cheaper and easier for more and more people to be able to do that and beyond a certain point you're then going to be living in a very dangerous world where like terrorist groups or just omnicidal maniacs or whoever or just curious people who are just like I wonder if I could make a really bad virus oh shit it got out um whatever you might view of covid uh you know like we're trending you know in many ways we're trending towards this kind of world uh you call it like the idea of small kills all um and so that's one category of risk as technology becomes more and more powerful and more ubiquitous so then the sort of obvious surface level solution to that would be well shit we need more and more like surveillance technology and like safeguards and like centralized centralization such that like there's like some kind of governing body to slap down anyone who's starting to do these kind of risks you know these dangerous technologies that might kill everyone but then if you build more and more of that centralized surveillance technology now you're opening up this other horrible can of worms which is we might end up in you know some hellscape tyrannical control for the rest of time where everyone like there's no privacy there's no freedom um and so people like schmachtenberger and indeed like honestly one of my main goals with this podcast is like what is an alternative to both of these states what's a third attractor that gets us off of this horrible seeming trade-off where we get we minimize both of those risks but maximize freedom and maximize the like long-term flourishing of society and one of the solutions that you pose to this um is this idea of uh essentially decentralized defense you call it multipolar active shields so can you talk us through what a multipolar active shield looks like yeah well it's it it sounds itself quite dystopian um it's just you know like the the entire idea is really that you know if we think things through you know like the both of the dynamics that you laid out like one is this kind of like risk of technological proliferation and the the small kills all term actually came from bostrom because he actually wrote a paper on the fact that we might need something like a central um kind of like world governance mechanism that has ultimate surveillance and ultimate enforcement capabilities to avoid is it in the vulnerable world hypothesis paper it's the vulnerable world hypothesis paper yeah and so basically you know we're kind of like working against that and and saying okay yes the risks are there um you know i guess just now you know like um i think in russia you saw lots of the like bioweapons labs suddenly like popping up again and like them being like um uh yeah being occupancy again so you know it's not like far-fetched scenarios we have nukes we must worry about bio weapons we must worry about like you know like other robotic weapons um automation etc so it those are real real risks on the other hand you know the the centralization risk i think that you mentioned that boston um kind of like puts forward as a solution of like you know there's like one world government like i'm almost more scared of that historically that's just something that like has not worked whenever people have tried to do something like that and it's something that we always like to do as a solution like it always comes up again over and over and this time we can do different because we have ai and like this time we can like true centralized control has never been tried yeah like now we can do it all all correct and i think the one of the notions is why it's bad is the notion of why i mentioned like these like it's unlikely that the voluntary shilling points will be upheld if we had such a thing but even if we wanted to construct it like it has a single point of failure which means that if it's not right one day um it can go horribly horribly wrong like basically world government leading to civilization of suicide uh because it's wrong one day in uh in the risks that that it wants to like annihilate it's robin hansen's i think um most likely uh reason why we have a great filter or like why we might not make it through into the long term so you have to be right every single time if you have one super powerful entity it also creates this massive um risk like a honey pot almost like for external actors to either like you know infiltrate you to corrupt you um uh to to blackmail you um it's just this one kind of ring of power that the people right which which will not necessarily attract typically those who you know are attracted to power will naturally gravitate towards this and typically those who want power are exactly the worst ones who actually deserve it yeah like i mean that that tended to be like that civilization speaking um who knows maybe we can find some absolute angels to run it but even the angels will be corrupted by devils outside so basically yes i agree so that's also a big risk and i think it's not often enough really pointed to as a risk um and so we need something in the middle uh or like a third way like not even in the middle but something totally different and i think you know our solution is like it's quite dystopian so i'm not actively advocating for it but we're basically going from the premise that like it's quite likely that we will have a lot more surveillance already like that's just the path that we're on the technological path that we're on and it's also likely that we'll have a lot more automated enforcement like we already have a lot more um drones we already have a lot more ability to automatically enforce basically whatever we decide is right um so those are just like two technological realities that i'm just not sure if we can make them go away we're not excited about them but but we might just end up with them but if we must have something like surveillance can we have surveillance and so surveillance is this notion from david brin which basically means well surveillance is like you know monitoring from above surveillance surveillance will be monitoring from below so basically this notion of like can we at least have surveillance in a multipolar decentralized world where everyone is watching each other and that sounds absolutely atrocious but maybe we don't even have to do it that way maybe we We can do it in an encrypted way where, let's say we all agree there are specific pathogens or specific viruses or whatever you might want, that they're just like not worth it or like some specific gain of function research.

58:09We all agree that that's just not worth it to do. In that case, maybe you can have like a decentralized privacy preserving monitoring infrastructure on various labs or actors that are likely to produce them. It's still quite difficult to figure out the specifics here. then if these kind of like encrypted tripwires get triggered, only then is that kind of like system revealed to all of us. And we then collectively kind of come up with this like open source defense mechanism to defend against the threat that is unleashed. So the idea is that we don't all have to monitor each other. It can maybe be this encrypted monitoring fabric that is trained on specific kind of like to flag specific risks.

58:51if someone is like you know in their lab out there like trying to do something quite specific so yeah so you basically you're saying yeah it's by necessity you would still need to have a form of surveillance but it would be bottom up in terms of Yeah, there's no centralized authority controlling it all. it's all these little multipolar nodes. But they technically have access to that information. Yeah, I mean, it makes sense to me. I still would rather not live in a world where that exists because as always, these things, there's the ability for corruptibility. And in theory, it is kind of still the death of privacy forever because it's basically saying that everywhere there will be cameras at all times.

59:36But I guess that is preferable to the, if it seems conceivably true that we would end up in a world where we're just going to all die by a pathogen or anything else anyway, like at least we get to carry on existing. Okay, so I accept the premise. At the same time, like the feasibility of building such a thing, like to start with, you're going to need some kind of centralized authority in the first place to sort of build such a system, or at least some degree of like coordination to agree upon like protocols of like, what is the minimum, you know, what is this risk threshold, which has to be surpassed in order to be like, okay, now we need to take a look at the footage or, you know, each node can take a look at the footage.

1:00:27So that, you know, it's a bit of a chicken and egg. You need to have that, some kind of centralized mechanism to build it, but then you have to turn off that centralized mechanism. And typically once you build a centralized mechanism, it never goes away, right? Do you have a solution to that? That is the million dollar question. I mean, there's a bunch of million dollar, billion dollar questions in there and a bunch of like civilization destroying questions in there in the whole thing. I want to emphasize again, I'm not a fan of this. I'm just saying, okay, if we actually live in a world in which it only takes two people to manufacture something that can kill everything, we need to think about like uncomfortable solutions and you know it's not necessarily that there's cameras everywhere um you know with the death of privacy but like it might just literally mean there is this entirely encrypted decentralized monitoring um structure that might even just like you know test what you do in the lab you know like it's probably not necessary to like film me while i do things in the lab um but like and and again it's very difficult to even monitor for that because it's literally just like you're doing something in the lab and like there isn't much of a gap between something that is life-saving and life-threatening um right so it's very difficult to monitor for in the first place but nevertheless let's just like pretend that we've solved this and we can create something where you know there might not even be cameras it's just like entirely uh encrypted um kind of like monitoring fabric that like you know like you can't just like corrupt by like you know turning over a leaf and like looking at looking at something unless very specific uh pre-agreed and like electronically or computation enshrined principles um haven't been met um now how do you even get to these principles how do you build that structure i was ever anyone ever going to be incentivized to use it yeah good question i don't know i don't think we should get there by like uh having an and like a un propped up by various enforcement mechanisms impose it on the rest of the world that's kind of like definitely trying definitely getting into the second trap of centralization and of like total um yeah i guess just total domination yeah exactly you know what we can do is create systems that just like that tend to work like sometimes it's not like it never works that you know there are there is voluntary consensus across different labs that you know there are a few things that we just are not going to do anymore you know we're um kind of creating this like you know monitoring fabric that where we can just keep each other in check a bit and like help each other monitor for these risks um there's like a fantastic work from kevin esfeld on like you know better like security in a um uh basically like some kind of like monitoring fabric too but like basically like there are a few technologies coming along if enough labs kind of like gradually um would um would adopt those it's not impossible that they can take off at least to some extent and then of course you need um you know to some extent like um not necessarily an enforcement mechanism but like a way in which uh more people are incentivized to do that so it would be great if there um would be a little bit more useful information about which labs are doing that and people actually like um changing uh their um their consumer behavior based on based on which labs are doing this um yes or no and like it's just not um like in the past we have had institutions emerge even to prevent risks and in a relatively gradual manner without them being like top down imposed and like if they take off in like a small section and people you know find those metrics appealing then it's not impossible that that they can grow and you know you have that to some extent with like many of the labs labs like to better for some better some worse agreeing you know themselves without ever having been forced to to specific safety principles and you can argue about like is it enough or is it not enough but like you know it does matter that like we care intrinsically about doing the right thing and then you can start cooperating and holding each other in check then you can start giving each other benefits for everyone else who agrees to specific to specific safety measures you know You can do that in a voluntary manner without it having to be top down in force.

1:04:49And we've had that over and over. What's an example where we have had this? Looking at nuclear weapons, isn't it the case? I remember reading somewhere that the US openly shared that their own advances with the rest of the world, including Russia and their enemies, their advances in making their systems more robust against an accidental first strike. You know, that's an example of like clearly just, again, like Pareto optimal, right? You know, an example of like clearly win-win information. It's like, this is how we figured out how to not have an accidental first strike, you know, accidental launch of nuclear weapons.