Existential Risk and the Future of Humanity: Lessons from AI, Pandemics, and Nuclear Threats (Toby Ord, Author of "The Precipice")

27:33So beings that act systematically to steer the world towards what is good or what is right in a way that maybe Maybe even if there were no humans, but there were other mammals, maybe they would morally matter, but they wouldn't be trying to move the world towards better situations. If they turned out to be destroying the ecosystem, such as through some kind of byproducts of their living, they wouldn't be able to kind of realize that that was a bad thing and to act so to correct that kind of side effect. And so maybe we're the only kinds of beings who can do things like that. So that's a focusing on what makes us cosmically unique.

28:13And maybe we're the only place, as Carl Sagan said, where the parts of the universe can come to know the laws of the universe itself. But I think that even if we're not unique in that sense, there's still a whole lot of importance which come from these other senses. There's still all of the deaths that would happen during the event. There's still all of the future lives and the meaning in those lives that will be lost. And there's still all of this failure to live up to the kind of hopes and dreams and expectations of our ancestors who gave us this rich life that we have today. Well, thank you for humoring my unlettered nihilism.

28:53But that was certainly a part of the book that got a wow exclamation mark with some of that Carl Sagan piece. And then, you know, I think you give such a beautiful personal example around this generational compact of when you had your daughter and you realize what it is that your parents gave you and ask how, you know, I'm never going to be able to repay this of you. And, you know, they sort of tell you, you know, that that's just how this goes. You can only pay it forward. It's not possible to, you know, pay it back. Yeah. I mean, that was a crazy experience. But there's this realization, and I really felt this kind of weight of not quite responsibility, but I guess gratitude, but overwhelming gratitude.

29:37And also realizing how little I did to tidy my room or help with the housework and how I felt that it was some kind of affront to even pay back like 1 % of what they did for me. Not realizing that it was only this tiny fraction. If I'd realized the full magnitude of the thing earlier, I think I would have been a bit better as a child. So I appreciated their answer. It was certainly a very helpful answer as far as I was concerned. Another kind of answer would be to say, well, we're getting old and you could visit us every day and move back to Australia. Yes, very true. Which, I don't know, if they'd asked for that, I would have certainly had to think about it because, yeah, no, I really take seriously this magnitude.

30:25And I also take seriously this magnitude of what we, you know, this kind of gratitude, yeah, to all of these other generations. That if you just look at kind of any little thing around you, like I'm looking at the computer keyboard and as well as the materials it's made of, of like, you know, like aluminum that is extremely difficult to work out how to refine that. You need high energy and so on. But also, you know, the keys as well as being made of plastic, which has its own kind of complications. But the letter forms and the development of the language and the electronics inside it and the rubber of the cable and just how this has been kind of come together from all different kinds of plants and minerals across the world and different processes.

31:12Even looking at simple things like things made of brick or wood and the improvements in carpentry and so on to get a surface as smooth as this and the developments of sandpapering and planing. And it is bewildering that you could just look at kind of any item and even just a very simple thing and realize that in a lifetime of work, you kind of wouldn't be able to create this thing from the ground up. There's just this, you know, in terms of standing on the shoulders of giants, I think it's a very useful kind of humility lesson to look around to you and really appreciate it. Yeah, that's a good exercise.

31:52It's almost a gratitude exercise, to put it glibly, but profound. With all of that said, perhaps we should talk a little bit about AI. And I was interested that you said it's not clear if that risk of an unaligned AI is higher or lower? What are some of the different factors that muddy that up a little bit? Yeah. Back in 2019, when I was putting the finishing touches on the manuscript, the cutting edge AI systems were reinforcement learning game playing systems. Think of things like AlphaGo by DeepMind or the StarCraft and Dota playing action game type systems. So these are systems that through this process of reinforcing them, giving them kind of rewards if they do something right and kind of punishments if they fail and letting them kind of explore the space of actions in a game, you can often move very quickly through the human level.

33:00from very weak play quite rapidly through the human level. I think in the case of the chess playing system with AlphaZero, that within a day or something, it moved from complete, worse than a human amateur, just random moves through the human level and to superhuman play. I don't know if it was minutes during the human zone or something like that. That can be a very rapid ascent. And then also these systems are very inhuman. They don't understand language. It's extremely unclear how you would explain anything to do with morality or what people value. So if you had some industrial system, suppose you use these techniques to train something to run a company or to produce a factory that produced a lot of widgets and trying to maximize the number of widgets it could produce or something.

33:52It might just do all kinds of damage in the course of doing these actions because it kind of wouldn't understand that the world around it was important in some intrinsic way. But now we've got these systems that are very different to that. Here are like four key differences. So one is that these large language models are trained based on human data, on a huge amount of human data. So about 10 trillion words of human data. and they have read pretty much every book and every paper so that they know a lot of moral philosophy. If you ask them about the details of population ethics or various abstruse things, they can give pretty good answers to them all.

34:36And as Stuart Russell pointed out, pretty much every work of fiction is just in every page. There's some human judging some other human's behavior and whether it was found inadequate or not and why. And so they've witnessed a whole lot of these kinds of statements so that they would understand if the AI were to take certain kinds of actions, whether humans would judge it as good or bad, or perhaps whether it would be ambiguous. And there's just this vast wealth of information to have access to about that so that the information is in there, which is good. Also, if you train on human data and the pre-training phase of these AI systems, the first phase is this next token prediction.

35:18So you look at a whole lot of words and you try to say, what's the next word that's going to come up? And that's ultimately a form of imitation learning, where it's learning to kind of pick the kinds of words a human would pick next. And that fundamentally pulls you towards the human level, the human range of behaviors, rather than doing some kind of bizarre set of behaviors that are totally different to us. So unlike things that might kind of just zip through the human level and then overpower us very suddenly, this is something that had very rapid improvements, but pulling towards the human level, which is nice.

35:52So it might at least spend some time there before we apply additional techniques and allow it to kind of take off again beyond the human range. And then also these things are fundamentally not agents. So a raw large language model is just one of these next token prediction things. Once we do additional what's called post-training, so additional stages of training, such as reinforcement learning from human feedback, where we get them to produce some different answers, and then we mark the answers and tell them which one we liked and which one we didn't. That helps teach them how to follow instructions and things like that.

36:29And at that point, they are in some sense are like some kind of minor agents because they're not just pure imitators at that point. But there's still much more like simple one-step things that don't do a whole lot of planning. They're not planning kind of 18 moves ahead or something like that. So you've got these systems that are more like oracles than they are like agents. They're more like just things that you ask it a question, it gives you the answer. and so they could be much safer they can avoid these types of cases where there's a subtle misalignment and then it goes off on its own you know and and amasses a lot of power and potentially works out how to get humans out of the way and then even then also the ai systems are generally very inscrutable it's very hard to understand what they're thinking but there have been quite good developments in interpretability of the the neural net weights and on top of that when we build reasoning systems out of LLMs, by default, they reason in natural language, like in English.

37:34You can kind of see what they're thinking. So they're just like, you know, a whole lot of ways in which the technology is less scary. It doesn't fundamentally start with goals. It doesn't start with long-term planning. It knows a lot about human morality, even if it's not necessarily influenced by it. And so the step is then to how to turn that knowledge into something that guides it. But that's a way in which the technology that we've been given that was somewhat difficult for anyone to anticipate five or 10 years ago, has turned out to make alignment a bit easier as a problem compared to what it could have been.

38:14That is really interesting. I've obviously, as a tech writer and someone involved in the venture capital ecosystem followed a lot of this, but I think the way that you just framed that up so crisply was really clear. It's this sort of reinforcement learning style where you have really unpredictable behavior that doesn't have the context of the world around it and isn't born in a human mind in some way. And then the rise of LLMs has more of this context and more of that human style of thinking. This episode is brought to you by Brex. Fred Adler, the influential venture capitalist of the 1970s, was known for displaying decorative pillows in his office that featured a signature business philosophy.

39:00Corporate happiness is positive cash flow. In today's post-SERP environment, Adler's wisdom feels particularly relevant as founders need to make every dollar work harder. That's exactly what Brex delivers. Their modern finance platform was built specifically for startups like yours and designed to help extend your runway when capital efficiency matters most. With Brex, you get global corporate cards with up to 20x higher credit limits and no personal guarantee required. Their banking solution has no minimums and no transaction fees, while letting you earn high yield from day one with same-day liquidity.

39:38Best of all, Brex knows you were born to build, not juggle spreadsheets and finance tools. Their AI-powered platform brings cards, banking, expense management, and travel all in one place. It's simple, scalable, and designed to get you back to what you do best, building. More than 30 ,000 companies, including one in three U.S. venture-backed startups, trust Brex to help make every dollar count toward their mission. Join them at brex.com slash Mario. you know. Given that sort of evolution over the past few years, where do you see the sort of biggest risks remaining? Is it that there's, you know, some new paradigm that shifts things back in a frightening direction?

40:24You know, I could also imagine that simply extrapolating from this point could create significant issues, maybe not existential, I don't know. But yeah, what should we be paying most attention to? Yeah. So I think it's a good question. And even in the case of alignment risk, it's not clear that it's gone down. So I said the technology is in some sense better, but maybe the world around the technology is worse. So we'd had a certain amount of racing between different AI labs, even back five to 10 years ago. But that really went up, notch once Microsoft and Google got involved. And now we're in a situation where it's not just racing between small labs of specialists, but rather between trillion dollar companies, the largest companies in the world.

41:21And nation states. And nation states as well. Yeah. So this geopolitical race between the US and China is also very worrying. And so the pressures from these races and also the general commercial pressure that comes from, I guess, the capitalist system applied to this race are very fierce. And so it's ultimately, if you look at, say, DeepMind and OpenAI, they were both created, I think, by idealists or at least by people in their more idealistic moments. And there was a bit more of a chance that there'll be room for that idealism rather than these additional kind of pressures on them to race. So that's an example of where that one has also gotten worse in some way, even though it's got better in other ways.

42:11But then, yeah, back when I was writing the book, I really felt that there were a number of different kinds of risk from AI, but the chief one was this risk of AI takeover or alignment risk. So you could think of that as one AI system or a small number of AI systems that are fairly similar to each other, maybe different copies of the same model. Realizing that to achieve their somewhat inhuman and misaligned goal, that they needed to wrest power away from humans who would stop them if they saw them attempting to achieve this goal. And that they would try to accumulate power and eventually overpower us and take control of our future from us in a fairly abrupt way that would involve violating our rights.

42:57Okay. But another possibility that's arisen, I was aware of these possibilities already, but they seem smaller. And now I feel that these are all roughly equal in size or threat. So another possibility is what gets called gradual disempowerment. And I think of this as a case of, suppose the AI systems never violate our rights, that we maintain sufficient control over them, that they don't break the law, but they're more successful than us over time at doing all the kinds of jobs that we do, which often uses the definition of artificial general intelligence. If that's the case, we should expect that over time, even if we get richer through trade with these systems, that a higher and higher share of the wealth eventually accumulates in AI hands.

43:49So ultimately, they're more effective at doing what we do, and they out-compete us, and their wealth grows faster than ours. And then at some point in time, they have half the wealth, and then a generation later, they have 99 % of the wealth. And with wealth goes power. They could use that wealth to influence political elections, even if they can't themselves run for election. And we just might expect, just as if you're a minority in a country where some other subset of people, even if your wealth is increasing, if others are increasing much faster and the share of control of resources is all flowing into their hands, you might expect it to be a bad outcome for you and a bad and permanent outcome perhaps.

44:33So that's a gradual disempowerment story. And then there's also other ones such as what's sometimes called AI coups or AI dictatorship, where an individual, it could be the head of a country, or it could be, say, the head of an AI lab, or it could even be someone, say, who's not the head of the country, say, someone else in the White House or in the AI lab who tries to seize control of the AI system and have it kind of secretly, uses it to try to plot, you know, to take over. Ultimately, if these AI systems eventually do become capable enough that they could seize control of the future by themselves, for themselves, then they'd also be capable enough that if they were what's called intent aligned, so that they could do what humans actually want them to do if asked, then that a human could ask them to take over on their behalf.

45:22That's another kind of concern. Normally, there would be safeguards against an AI system doing these things, but maybe the human, if they run the AI lab or they work there, maybe they're in a position to remove those safeguards. Or if they're the leader of a country, they could command that they be removed. So that's another kind of concern. And then a fourth kind of concern is the building of various forms of weapons of mass destruction using AI-assisted help. So in particular, the concern there is that AI systems could kind of democratize newer and kind of more powerful destructive abilities.

46:00So to kind of empower many individuals, perhaps people just with an undergraduate biology degree to create some world-destroying virus or something like that. It doesn't have to be biology. It could be in other disciplines if new capabilities appear. So I think that those four different areas are all concerning to me. And I'm also troubled by the fact that I don't know which is the most concerning. and I'm troubled by the fact that a lot of the things you could do to help with one of them make the others worse. So some of them, for example, the issue is concentration of power and other ones, like the weapons of mass destruction, the issue is the dispersal of this power.

46:42So open weights, for example, helps with one of these issues and hurts others. And so I feel really at sea for a lot of these levers that people argue about. I really don't know which way to pull the lever, whether it's actually helping or not for the overall situation. That's reassuring in the very small sense to me and disconcerting on the broader scale because you clearly think about this at a much deeper level and much more than I do. But I do find myself really struggling with figuring out what levers can we actually pull here that unilaterally help. And on one dimension, the race dynamic is something I keep struggling with figuring out how we get around that.

47:27Because if we sort of assume that the major actors of these AI labs are morally normal, which we could probably debate that. And there's probably actually significant variance, I would suggest. But just the sheer incentive structure and mindset that you get into when you are in these race dynamics, like really feels like it leads to a lack of circumspection, lack of sort of caution. Are there good examples historically of taking moral action in a race dynamic? You know, there are some perhaps from from the nuclear race, but that was quite incomplete as you as you share in the book. Yeah. I mean, I would say the mainly bad lessons, lessons about bad behavior rather than good behavior.

48:17I mean, one example there was the Manhattan Project that I think is particularly salient is that if you ask, why were all of these people, like these academics who studied physics, working on building a super weapon that would be the most horrible weapon humans have ever built? Yes. And the answer for most of them was that they were concerned about Hitler's Germany having unrivaled access to those weapons and holding the world to nuclear blackmail. So they worked as quickly as they could to develop such weapons. But when Hitler died, when he killed himself, and then shortly after the war in Europe was over and Germany surrendered, they didn't stop.

49:04Only one of the atomic scientists actually stopped at that point. And you'd think that if there was one reason that was so powerful that it led you to make the worst thing that humans have ever made, and you hadn't yet finished making that thing, and then that reason disappears, you might hope that you would have time for re-evaluation. And there were some reasons why America used the nuclear weapons against the Japanese. but they were certainly much weaker than those other reasons. And I don't think that the atomic scientists would have signed onto the project with the idea of shortening the war in the Pacific being the objective, at least not many of them.

49:46So one of the things we learned there was a group of people doing something that had serious problems for reasons that they'd convinced themselves were important but then just no longer being sensitive to those reasons disappearing. And there's a whole lot of cases like this. I mean, another example of, I guess, a smart person talking themselves into a bad argument. I can't remember who it was, but the chemical weapons for the Germans in World War I.

50:14He has these written documents saying that he thought it could shorten the war. And that even though it could kill a lot of people, the bullets were also killing a lot of people. and that if the war was shortened, then ultimately fewer people would be harmed by it, both soldiers and also the civilian populations. And so he thought that ultimately, while the weapon itself would be brutal, that this would fundamentally be a good thing. And the big thing, well, a big thing he didn't realize, even on his own logic, was that Germany lost the war. And if you were helping the side that lost the war, that means that your technology was lengthening the war, not shortening it.

50:54And so the idea that he was smart enough to develop all of these chemical techniques and things and to run this argument, but not smart enough to realize the argument only works if your side wins the war and that there was like a 50-50 chance that your side would lose the war, in which case it doesn't work. Yes. You know, I think there's a lot of these things where we see smart people, in that case, maybe not someone who is particularly idealistic, but it shows that it's quite easy to talk yourself into various things. And I think that there's a lot of that going on at the moment, particularly when it comes to racing.

51:29Yeah, you talk about the unilateralist curse in the book, which is sort of the idea that all of us may judge these different actions that we could potentially take to, let's say there are 20 people who might release the next AI model. whoever is essentially most optimistic and sort of has the most Panglossian view of what might happen, they're the ones who release it. And then if it actually causes damage, everyone sort of reaps the damage of that or incurs the damage of that. And it feels like there is some of that happening in this case. Exactly. So just like how in an auction, it goes to the highest bidder.

52:11And so if all the bidders, if there are many different bidders and they all have somewhat noisy and inaccurate estimations of the value of some object, the person with the rosiest impression of the value of the object will be the one who wins the auction. So this is called the winner's curse, that sometimes you can systematically end up... The very fact that you won probably means you overpaid. There's a very similar issue here where, yeah, if a whole of people could all unilaterally make some action happen. Maybe it released the genome of smallpox or something like that. And a lot of different people are trying to think about the benefits and the drawbacks.

52:49And there are serious benefits of doing that kind of research. The same with, say, gain-of-function research, where you try to make a virus more transmissible or more deadly in order to understand what mutations are needed so that you can be on the lookout for those mutations occurring naturally. There is an argument, a rationale there. But it's not obvious if that rationale wins out versus the harms of doing that. But if you just allow anyone to do it, then it will always be the people who end up with the most inaccurately positive estimation of this action who will make it happen. And so this is an interesting form of coordination problem.

53:30And there are ways around it. For example, you could get together with the other group of people who could all unilaterally make something happen. So say all the other people who could be releasing a model like this. And you could say, let's have a vote. And if you just take the vote, it ends up reflecting the median estimate, the middle estimate, if you do a majority vote. And so that actually almost completely resolves the problem, if you can do it. In the case of a model release, it could run into antitrust issues if there's an issue of collusion to slow down a technology. So unfortunately, I think antitrust law could well be quite bad for us when it comes to these situations.

54:10There are appropriate mechanisms whereby companies can get together if they create a third body, a standards body, which is open to new members and not just a kind of fixed list of members. Then you get that body to recommend a course of action and then you follow that action or something. There is an official way to do it. I worry that the extra effort and steps involved in doing that slows everything down. It slows these remedial actions down while the other actions are being sped up by the pace of investment, need to make a profit, and so on. Given that you're sort of using the example of antitrust, do you think if we were able to resolve some of this race dynamic within the US, would that be sufficient?

54:55because the sort of thinking there would be that the most advanced models and most advanced capabilities are US-centric right now. And so by sort of removing this dynamic, we're reducing the existential risk from an aligned AI. Or is the fact that China is clearly taking this very seriously, even if they do seem to lag in some considerable places and have been cut out of the semiconductor supply chain in really meaningful ways. Does that sort of fundamentally, just by controlling the US, invite a different race dynamic where we're sort of actually hamstringing Western powers and leaving the possibility of unaligned AI in a much less democratic environment?

55:41Yeah. I mean, this is another thing that's related to the unilateralist curse, that if there's some issue that seems at some level problematic, let's say taking some new step with a technology is at least at some level troubling, then if all of the people who are concerned about such things all stand aside and say they're not going to do it, then the person who has the fewest ethical qualms will be the person to step across the threshold and maybe will get the first version of something will be a more dangerous version of it. So that is a general concern. I think that it's also one of those, it's exactly the kind of thing that people in the position of the atomic scientists could use to rationalize racing.

56:24Totally. So while it is not a bad idea, it convinces different people different amounts. And then you kind of worry that the type of people who are erroneously convinced by it will be the people who take the actions. I think it's a real challenge. So as you say, there's kind of racing, roughly speaking at two levels at the moment. There's this kind of company racing between different companies, most of whom are at least headquartered in the US. And then there's a kind of racing between the US and China. So the US is clearly the one that triggered the race between the US and China. And partly it triggered it because on the rationale that eventually the race was going to happen and we want to win it if it does happen.

57:06And so if we start the race, we've got more chance we'll be winning the race. And there's a logic to that. But it still is a case of, you know, if you think races are bad, and the racing behavior is kind of like defecting in a prison's dilemma, and you think that unfortunately, the structure of the world is such that probably both parties are going to defect. So therefore, we should defect first. Maybe, maybe that's the right thing to do. But it's still kind of, you know, still jerk behavior at some level. And so I think, ultimately we succumb to some existential risk with AI. If you kind of imagine the different powers that be or all the different people involved kind of being called up to the pearly gates, meeting St.

57:47Peter and trying to explain their behaviors, it's a little kind of thought experiment. You'd have to get into some Gabe theory. It wouldn't sound great, I think, if you said fundamentally, we knew it was really risky, but we thought people would do it, humans would do it even though they shouldn't and we thought there could be a race so we started the race and did you do you attempt to have a treaty to like not race no we didn't attempt that we thought it wouldn't work so we'd started the race and then we raced and we got there first and we built the thing and the thing killed us all i just think it'd be pretty difficult to really you know if you add a kind of like imagined your honor yeah at the end of these statements They don't sound great to me.

58:26And I think that as it happens, given that China is behind in this technology, I think that they have additional reasons to not want to race. So if the US were to think, as I think they should, that we would rather there be no race, and China would prefer that even more because the race that they would be is a race that I'd likely lose as well. then I think China could be in quite a position to be happy with some kind of verification conditions or things around kind of avoiding a race. So I feel that there's been very little thought going into not racing. And I think that if you pretty much just had those two countries agree, they could potentially police their own spheres of influence to say, okay, no one here is building things towards superintelligence.

59:17No one over there is doing it. And then have verification on each other's AI industries. Such that no one cheats it, essentially. Yeah. For example, by saying, okay, well, we'll send 30 of our people over to be part of your AI companies, and you can send 30 of yours to be part of ours or something to make sure that we're not doing it. I think that there are ways to do this. In the Cold War, I think there's a lot of lessons from the Cold War. Ultimately, there was a situation where the US and the USSR were very much adversaries of each other, if anyone is. And yet, they actually treated each other as adversaries in what I think is a somewhat grown up way that we're not doing these days.

59:58I think that the US is treating China as an enemy, not an adversary. And they just kind of want to get China. Whereas with the USSR, I think there really was a realization that there are a bunch of things that were in their common interest. Not everything, but there were some things. So for example, the Non-Proliferation Treaty was in their common interest. If the US has nuclear missiles and the USSR does as well, they don't really want all the other countries to have nuclear missiles. It's in their common interest that this would be an elite kind of exclusive club. And it was also, I think, in most of the world's interest.

1:00:31And so they helped to broker and police this non-proliferation treaty. And similarly, their arms reduction treaties that they had, because they both would have rather that they had a tenth as many missiles as they actually had. And they managed to do that. And there were very smart people working out very novel verification techniques. There were these ideas of, once spy planes became possible, of allowing the Russians to fly spy planes over the US airfields while they soared their bombers in half and they pulled the halves apart from each other with tractors. And those bombers are still there on the airfields, sawn in half as a costly demonstration of destroying their military capacity.

1:01:13Even without inviting inspectors over, they worked out ways to credibly show that they were reducing their arms. And so, there were really people thinking outside the box is what I'm saying. And there's a lot of very smart people today thinking about these issues. A lot of them are going to work at AI companies and build these technologies. But there's also a lot of kind of complaining that we have of like, oh, verification's too hard. There's no way we can verify a kind of deal to not race. And I feel that the amount of, let's say, full-time equivalents spent on thinking about verifying a race between China and the US on AI, maybe there's like three full-time equivalent people working on that or something in the whole world, if that.

1:02:01And probably almost nothing has been explored. So I think that there's a lot of just giving up, oh, that'd be way too hard without really trying to work out, are they really clever technological or sociological techniques? That's really interesting. I want to touch on one of the other areas briefly that you talk about in your book, which was the extremely prescient and immediately prescient part, which was pandemic risk. You mentioned gain-of-function research and sort of thought through the trade-offs there. What have been the lessons of COVID that you take from that kind of research in general and how risky it is?

1:02:40And more broadly, did COVID make us more prepared in some ways for what comes next? Or have we really not taken almost anything from it? So one thing it did was that when I was writing the book, I was certainly concerned that people were just not going to take threats to humanity's entire future seriously. That there was a feeling that, I don't know, that there was a time when these things were possible and that that time was in the past. Maybe in the case of pandemics, a feeling that, oh, sure, there was this 1918 flu that I've heard of and there was the Black Death, I guess, but that was medieval.

1:03:15And technology has advanced so much since then. And we do so much in our hospitals. We understand the germ theory of disease and all of these things. And so we're doing much better at this. And so it's not a risk. And COVID certainly reminded us that we're still at risk and that for all of our technologies, in some ways, we're just as vulnerable as we were before. So when it comes to dealing with individual infections and trying to save someone's life in the hospital, we're much better at that. But overall, when it comes to the spread of disease, because the airplanes that we've invented and things, you know, the cities that we've built and the public transport systems and so on are so efficient at just spreading this disease around.

1:04:02A lot of our technologies make it worse at a similar pace to the way that our protective technologies make it better. and it's not obvious at all that that we're actually more protected overall when it comes to to pandemics so that's you know it's helped give us some of that kind of reminder and also it also was this feeling I think that you know when it started right there was this thing where you're like hang on what you know is it going to be the case that everyone in my country is going to be at home in their house like and the streets are just empty for days but you know days I mean months.

1:04:38Is it a school's going to shut down? And there was this feeling of like, is that, should that even happen? And in general, I think, you know, for people who'd lived through, say the Blitz in London, you know, that really shaped their view and they felt, oh, well, we're not even allowed to have lights on at night or something. And they've been aware that the space of ways of living, you know, could be quite different in the space of kind of big things that could happen could be quite big. Ultimately, COVID was, you know, was probably this biggest world event since World War II. I feel like there's this kind of level of what's the biggest change to life that you can imagine is set by the biggest one you've lived through so far.

1:05:17And so at least it's helped raise the ceiling on what we can imagine could be required of us and how different life could be. So that's been somewhat useful for kind of letting people imagine more. I mean, also, I guess, in terms of political trends and political populism as a rise of this movement and so on. If you lived through the 90s and the 2000s, it had really felt like we were in some really stable era of some sort. And then we've come out the end of that. And that took quite a while to learn those lessons and to realize that, at least for someone of my age, that the only world I'd known as an adult could be coming to an end as a geopolitical order or something like that.

1:06:02It's helpful to take that wider view and COVID has helped us do that. So in some ways, it's helped us kind of realize that for other threats as well. It was thought by almost everyone that there would be what's called a panic neglect cycle, where having been hit hard with something, that you overreact and that the governments would kind of would do more than they should. Either they'd spend more money or they would kind of like inhibit freedoms too much or something like that to react against this thing. And then that would slowly fade away until it was too little. But that hasn't happened. We actually didn't get the panic phase of that.

1:06:35And by the time everyone had reopened after COVID, I think people were just so sick of it that they didn't even want to talk about funding these bills to spend even a tiny fraction of the amount of money that we were cost or the amount of damage that was caused by COVID to prevent future such things happening. So that was a really missed opportunity. And I was surprised. I would have thought that you'd get this panic. And then the question was how to soften the panic into the appropriate level of response, but then how to maintain that appropriate level as well. And yet, no, we never even got the appropriate level.

1:07:12And I also feel that it had some good features like the mRNA vaccines getting kind of a test out and kind of flexing our muscles on this new technique of very fast vaccine development was good. And I guess I don't know overall how it's really helped our preparedness. I think that, again, you can kind of tease out like four or so of the biggest factors that either made it better or worse, or perhaps in some cases have not made it better or worse, but revealed underlying structural weaknesses. For example, it didn't take all that long before there was a lot of acrimony between different political groups or between different countries about who is at fault and who should be handing over the vaccines and various things like this.

1:07:58I guess we learned that we can band together for a while, but a while might be a six-month period. If the crisis carries on for years, that that really does start to fragment. Yeah, but I think that's true that it's, you know, your point on the way in which this totally shifted our frame of reference from this can never happen, that would be a, you know, as fanciful as a zombie apocalypse to, you know, actually, we all lived through it. And also, you know, Operation Warp Speed with, you know, the development of vaccines is maybe a good example of a beneficial race dynamic and where those things can work favorably.

1:08:41Yeah. I mean, it's even, you know, I think the Operation Warp Spade was a successful government, you know, slash private, you know, cooperation and so forth. Yeah. You mentioned sort of the political landscape and the institutions that played their parts in COVID. One of the things that I've been thinking about in reflecting on the book and some of these questions is how compatible are democratic institutions with addressing with existential risk? You know, there's sort of this time span of discretion where your senators are thinking on, you know, I think they got six years. I actually can't remember now.

1:09:21Six year cycles, your presidents are thinking on maybe two and a half because then they start thinking about, you know, the next election and so on and so forth. no one is really incentivized to think about maybe even 10 years. That would actually be a really farsighted person. How can we sort of address these existential problems with such a short-term mindset from the institutional level? Yeah. So it's a real challenge. It's not necessarily, you could say, a challenge of democracy in the sense that if you didn't have democracy, if you had some kind of autocracy, it's not clear that makes it better at all.

1:10:00if the autocrat happened to care about this issue, maybe it would make it better. But if they don't happen to care about it, then they wouldn't really be able to get the signal coming from the people. I'm not sure that it's democracy itself that's to blame for it.

1:10:16But some things happened to the democracies that we have in Western countries, at least, to really shorten the timescales that they focus on. I'm not exactly sure what's going on there or why that happened because it does seem like the democracies, if you go back 50 or 100 years, it does seem that they were able to achieve larger projects and projects that operate on longer timescales than they currently can conceptualize. So maybe it's to do with the kind of complex relationship between the democratic institutions and the media and the success of certain forms of media and kind of punishing them for not preparing an answer to the question they asked on Monday or something.

1:10:59And so everyone's thinking in terms of the news cycle, not even the election cycle. I don't know what it is, but it does seem like the time span, the time horizon for politicians has gotten shorter. There's also, though, separately from whether it would be better to not be a democracy, there's a question of, is there a kind of failing still of the current democratic system to perhaps live up to the principles behind democracy when it comes to this issue? And I think that there may well be. So when we think about the most important decisions that policymakers are making in our time, they may well be decisions such as what they're doing about climate and what they're doing about, say, technologies like AI.

1:11:41For example, deciding actively to not regulate and prevent regulation seems to be an active decision to do something negative there. But these are decisions that if you make them in one way versus another way, may well have implications that just echo through the generations. Having kind of like, you know, making people's lives significantly worse for many generations to come or perhaps not even getting a chance to exist at all. uh and so i think that that there's this issue that most of the stakeholders or like the you know or the people who are affected by it don't have representation um in uh in that decision because they're people of future generations and so and in some cases we can also predictably work out which way they would vote you know they would vote for doing more action against climate change Yes.

1:12:31For example. And so it's not even just that there's this kind of unknowability, so we couldn't possibly factor it in. And so there's a kind of constituency or stakeholder group that is just not getting the franchise. Yeah, that's really interesting. As somewhat similar to questions about people of different races or sexes or landholders versus non-landholders and other types of issues in democracy of extending the franchise. And so I think that there are potentially things that a democracy could do about that. For example, in the UK system where we have the House of Commons, which is a bit like the House of Representatives in the US, we also have the House of Lords, which is an appointed body.

1:13:13Some of whom are appointed by the Prime Minister, some of whom are appointed as the kind of good and great of society, former heads of the Royal Society and things like that. And it would be possible to appoint a group of lords to represent future generations. Or you could do it in other ways. You could have a citizens assembly, for example, representing future generations. And I think that experimenting with tweaks to the current understanding of democracy to deal with the fact that now so many of our choices, our most important choices, could be affecting all of these stakeholders who don't have representation.

1:13:47You know, the idea of kind of no taxation without representation, you know, kind of foundational to the American ideas of democracy. And yet, you know, maybe we need to take it seriously in this case of future generations and to work out, are there ways to appropriately understand their interests and then also to force some kind of representatives to represent those interests? Because obviously the people themselves who live in the future can't vote in this thing. It doesn't mean they can't be represented. That's a really fascinating idea. Yeah. Having these sort of stewards of the future as part of the governance process.

1:14:20Or even if it seems to be too impractical to at least set it as an ideal to say, oh, if only we could do this, that would be great. Unfortunately, we can't or something. But it seems at the moment, people don't even think about it or don't realize that maybe there are things we can do about it. Yeah, there's sort of the implication that people suspect or sort of hope that politicians are thinking about the future. But I think the reality is that that is, as you said, has sort of long since gone. Yeah, so that's a nice way to at least consider it. And we like to sort of end episodes with a few sort of more abstract thought experiment type of questions.

1:14:57So with that in mind, if you had unlimited resources and no operational constraints, what experiment would you like to run? I guess I think I have no idea. Oh, wow. Great. OK, that's a apologies. No, no. This is a sign that you're you you wouldn't want to go off on a wild experiment without thinking it through, which makes sense, given your background. I guess that's part of it. There's a famous comment from Carl Sagan when he was asked about, are you really sure about nuclear winter? How could you know? The experiments never happened. And he said, it's unfortunate that we have no spare Earth to destroy in some kind of experimental laboratory in order to determine the exact reliability of this theory.

1:15:47So if we had unlimited resources. We could have those bare Earths to destroy in order to find out what these risks actually are. But unfortunately, that sounds like the kind of experiment that's probably ethically a no-go regardless. Yes, that's right. That makes sense. Well, let's try this one. If you had the power to assign a book to everyone on Earth to read and understand, what book might you choose? I might choose Practical Ethics by Peter Singer as a book that's a really good kind of no-nonsense approach to thinking about ethics, It really helps show how to kind of reason about all kinds of issues with an open mind, not assuming that our current kind of ethical intuitions are the be all of it and all.

1:16:27And, you know, I think it's a good example of that kind of thought. Is there a sort of a particular framework or sort of lesson that you particularly reflect on from that book from Singer? I imagine he's a major influence of yours. Yeah, he is. Yeah, probably just really what I said then of the fact that he just kind of models this kind of really open inquiry into moral matters. Derek Parfitt mentions that ultimately there's been only really a couple of generations in which people have made what he calls non-religious ethics their life's work. So if you look at the whole history of philosophy, almost all discussion about ethics had been within a religious context where the set of possible answers always had to kind of align to what had already been thought thousands of years earlier.

1:17:21But people trying to, separately from that, to try to understand these questions, there'd only been a couple of people in the whole history of philosophy. And then there'd only been a very short time since it became more widespread. bread. And so he saw that as a reason to be very optimistic about the possibility that we'll make some substantial moral progress. Fascinating. What tradition or practice from either another culture or time period do you think we should widely adopt? I think I don't know enough about particular time periods. This is the problem about being an academic and wanting to not just make something up.

1:17:58But I do feel that something that involves more gratitude to our ancestors, I think would be particularly important. I know that there are many cultures that have more gratitude to our ancestors than Western cultures, which have almost none. But I'm not sure exactly who we should be copying it from. But I would love to see more of that. Yeah, I agree. That's a great one. Well, Toby, it's been such a pleasure. And yeah, I can't thank you enough for your work and your writing and for your time today. Oh, thank you. It was a wonderful conversation.

1:18:57See you next time as we continue to explore the future.