#129 Will MacAskill - We're Not Ready for Artificial General Intelligence

30:24It's going to be a really big deal. People have not woken up to this. Whereas lots of the mainstream media were saying it's not killing anybody. It's not killing as many people as the flu. you know, you should get over this. This is just paranoia. And the difference is between the level of a development or technology or disease and the rate of growth. Where AI at the moment, you know, is somewhat helpful. It's like, you know, probably making people a bit more productive. There are some harms as well. But the stuff like, oh, intense concentration of power. For most people, that seems pretty far off.

31:05It feels quite different than the world today. But the issue is that AI capability is growing exponentially. Where we're used to Moore's law being the fastest rate of technological progress, relevant metrics within AI are growing hundreds of times faster than Moore's Law. It's really quite remarkable. And I think that just means that people are going to be reactive. And with something that's advancing so rapidly, by the time that people are realizing how big a deal it is, it may well be too late, because suddenly you've only got six months before the technology is even better again

31:58Alex O'Connor:yeah it's the doubling uh it's the exponential growth it's quite hard to wrap your head around just how quickly something can grow it's the sort of the grains of rice on the chessboard right i've talked about that a few times on on in various contexts but you know for those who for those who don't know it's that old old myth of i think there's one version of it where it's like the guy who invents chess goes to some king and the king's so impressed he's like i'll give you anything you want as a reward and the guy's like just take one grain of rice and put it on the first square of the chessboard then put two on the next one then four on the next one then eight on the next one 16 and just keep doubling the grains of rice on the on on the chessboard and the king's like right sure yeah all right i'll do that i got off easy and of course by the time you get to square number 64 or whatever it is you've got like more grains of rice than like atoms in the observable universe because like those that doubling pull out your calculator and just go like two times two times two times two times two just do that a few times on your calculator and it is incredible just how quickly that number starts to balloon and this is kind of what's happening with ai exactly where um you know with moore's law so essentially the cost of computing power um is the best formulation of that.

33:13We're used to kind of a doubling or halving, but doubling of how much compute you can get for a given amount of money every 18 months or so. In contrast, I think the doubling of effective compute, namely that's taking into account both all of kind of computing power getting cheaper, companies investing radically more over time in larger and larger training runs for AI and algorithms getting much more efficient. So you can get much more juice out of those, um, out of that computing power you have the doubling time. There's more like a few months. Um, and when you're looking over the course of a year, it's more like a 10 X or even further, in fact, depending on exactly how you want to, um, exactly how you want to measure it.

34:05What is Moore's law? You mentioned it a couple of times. Yeah, so Moore's Law, it goes back to this incredibly stable trend where the number of transistors on a computer chip, or density of transistors on a computer chip, has doubled every 18 months or so. And that's been true going all the way back to the 1970s, maybe even earlier. Right. I think that particular metric doesn't quite work now because transistors are so incredibly small. really think, you know, look at your fingernail and look at how much has it grown in a second. That's the sort of size of a transistor now. However, there's a related trend, which is how much computing power measured in terms of floating point operations can you get for$100?

34:58And that has continued to double every 18 months or so.

35:03Alex O'Connor:right and i mean the thing about ai exponential growth is that it's not just like you've got a thing that's getting better and better you've also got a thing which like self corrects it's like you had a chess computer that not only was getting exponentially better at playing chess but also exponentially better at making new chess computers that were better at getting better at and it can also like you know control the military at the same time you know it's just this like unfathomable uh exponential consequence of of just a quite sort of simple starting point for this technology but like you strike me as someone who although you care about this a lot and you want to make a lot of noise about it like if you thought that this was a sort of hopeless doomsday inevitability, you probably wouldn't be here.

36:02Alex O'Connor:You'd be spending time with your family and, and, you know, having fun while you still can. So implicitly, you must think that there's at least something that we could be, could be doing about all this. Yeah. I think there's a lot that we could be doing. Um, you know, as individuals, though, a lot of, um, what that means as individuals is putting pressure on companies and in particular, um, going to governments and saying, look, we really care about this as an issue, where the public in general are very opposed to AI. Maybe they're more opposed to AI than I am. But it's low down the list of priorities at the moment.

36:45And I think it needs to go higher up the list of priorities so that politicians think, oh, okay, yeah, we actually have to have a sensible approach to this technology in order to be able to win elections. That's kind of what we need. And what would that mean? Well, one would be just enormous investment in safety, where there are these technical challenges for aligning the systems that we have not addressed, including technical challenges for ensuring that they don't get sabotaged. so ensuring that some foreign actor can't uh put in what's called poison data or other backdoors so that actually they can throw these systems rather than um someone else there's also regulation as well so um at the moment if let's say you know the us and china and the uk and um other relevant countries wanted to get together and say, look, okay, now this is starting to go too fast.

37:50We are at the point where AI systems are exceeding human performance across most domains. And that includes machine learning research itself. So things are going even faster, 10 times faster than they are today. If they wanted to say, okay, we should slow down, they really wouldn't be able to. Certainly, they wouldn't be able to verify such an agreement. And there are things you could do now to make that happen. So in particular, simply tracking all of the frontier computer chips. So not things that go into your phone or something, but these are very advanced computer chips. Just knowing where they all are, how much different countries have, so that you could say, okay, we're only going to grow the stockpile of computing power by a certain amount every year, at the crucial point of time when these systems are matching human capability, that would be a big leap forward.

38:52Similarly, there's just requirements governments could make on companies to say, look, what safety testing have you done? What is your, what's called a model spec, which is a description of how they want the AI to behave? Has the AI been, you know, demonstrated to be law following? These are quite low bars that you might want to hit, but these are some of the things that governments could, yeah, could request.

39:19Alex O'Connor:Sounds a bit like nuclear non-proliferation, right? It's like, you know, we're all keeping and an eye on each other and, you know, not too much plutonium. But it does feel to me quite a, I mean, you already said quite a difficult thing to police. I mean, how are we to know if China or Russia are sort of secretly planning their AI project? It sort of strikes me as very similar to fears about nukes in Iran or something. Yeah. And like, I often, yeah, so there is a good analogy with nuclear weapons where tracking compute could be like tracking fissile material. Because even though software, very hard to track because it's just, you can replicate it, send it all around the world.

40:07Chips, on the other hand, 90 % of the chips that are used to train AI come from a single company, TSMC in Taiwan. I think all of the remainder come from Samsung in South Korea. And then less cutting edge comes from some Chinese companies, which are still usable. But it's not that many companies, and it's a physical good. So you can actually, in principle, know where it is. I generally often feel kind of uncomfortable with too close an analogy between nuclear weapons and AI, because it's good if you want to get across how big a deal this technology is, but it's not so great insofar as AI is much more dual use than nuclear weapons.

41:03If governed well and if developed well, AI could be enormously helpful, including for all of these challenges that I've been describing. So AI itself can help with alignment research. AI itself could help design better protections against bioweapons and so on. It could also just lead to a more enlightened, more knowledgeable electorate, which I think would be extremely good too. And so I want to make sure that's kind of front of mind at the same time, where the very blunt instrument of just like, let's not go here at all, let's not do any of this, would be, yeah, I think it's like a little too blunt relative to the potential benefits that we can get from differentially advancing the most useful and socially beneficial AI capabilities.

42:03Alex O'Connor:yeah there's another um analogy to be drawn with the environmentalist movement as well i think in that like i know there have been like petitions to sort of halt ai research in the us and the uk people sort of scared of this thing let's at least put a stop to this until we can we can work out how to deal with it but like with environmentalism people will say look okay yeah we could you know you know gut out our our economy and make ourselves greener but meanwhile china they're not going to stop they're going to keep doing what they're doing and if anything all we're now doing is is not really making a dent into the problem and giving them the upper hand and people might think similarly with with ai like you know what are we supposed to do like it'd be great if we could all agree to just sort of you know calm down a bit and we could petition our government to care about ai but all that's going to do is slow us down meanwhile china just takes over the AI leadership.

43:01Alex O'Connor:And that's probably going to strike more fear into the UK government than any electorate who think that AI should slow down. So what do we do in the face of that? And I think it's a genuine concern as well. I mean, we talked about concentration of power, but in some authoritarian countries, power is already very concentrated. And AI will make it much, again, much easier to entrench such control. Already now, you could imagine just every citizen has to wear a little bracelet. That bracelet has a local LLM on it, is recording what's happening. And if you say anything that is not approved of by the political, you know, the party, then you go to jail.

43:55That's technology that AI has already brought us. And I think many more such technologies could be used too. So I think it's a major challenge. It gets much worse, much worse compared to environmental destruction, in my view, because the gains are just so high, where foregoing, for one country, foregoing developing AI and incorporating it into their economy would be absolutely enormous. You couldn't compete with a country that has developed and incorporated AI. Because once you've got an AI that is capable at the human level, well, then suddenly you can create hundreds of millions, billions of such AIs.

44:43And now how well would the US compete with China if China had a population of 100 billion. Probably not very well. I think there's also arguments for thinking that such a country would grow much, much faster too, and would have much greater kind of strategic and technological capabilities. And so that is why any solutions have to take one of two forms. One is involving multilateral agreements between, in particular, the US and China, or it has to be the case that one country is so far ahead that it is able to slow down at a crucial point of time without sacrificing its lead. That is the aim that some people in the US have at the moment.

45:34However, currently they're implementing that plan very poorly because they have put in really quite minimal protections to what are called the model weights, namely basically just this long string of numbers that determine the AI's behavior, including its capabilities. They can be stolen really quite easily. So if China wanted to steal GPT-5 or Gemini 2.5 Pro or any of the leading models, they could do so. And so really, you know, if the US really wants to pursue that plan, then they should be getting much more serious about information security and protecting AIs against theft. How specific can we get in like trying to suggest a solution?

46:28Alex O'Connor:I mean, you've spoken about people beginning to sort of care so that the government takes notice. you've sort of spoken about like vaguely how the the us and and china might interact with each other but let me put it this way like if you will mccaskill you are suddenly dictator of the united kingdom for a day or say it's the united states for a day and you can pass any law you like and everybody will will you know follow what what you say what what what law do you reach for you haven't got very long you've just got 24 hours to do the most you can what's the first thing you do like here is the law I pass, here is the first step towards a solution here, especially considering that you're only the dictator of the United States, not the dictator of the world.

47:12Yeah, well, first thing would be that any chips, I guess, made by US companies have to have location tracking. So they can, you know, I can just know, the US can know where all of its chips are going. Because they have these export controls, but such that, you know, those chips can't be sold to China legally. But loads of them end up in China anyway. And then the hope would be like, okay, if the US does that, then, you know, China could agree to compute monitoring regime too. That would at least be the first step towards saying, okay, well, we could start to have some sort of US-China agreement, which allows us, you know, at least just gives us the option to choose how fast we're racing from here to a world with radically more powerful than human AI.

48:19I think that would be, yeah, that would be the first thing. You could also have, you know, It gets quite technical, but you can have ways of assessing whether the chips are being used for training versus inference. Again, once you're at the human level, it's training the even more powerful models that is potentially most scary. Then I would just also invest enormous amounts. Manhattan Project for Safety gets used sometimes. um but yeah invest enormous amounts in uh how to design ai systems safely um perhaps in particular also saying that for any company developing um frontier ai they have to spend a certain fraction of the compute that they're using to develop ai on ai safety and that could be like even if that 1%, that would be a big win.

49:21I would say much more. 10%, 20%. That means that as AI capabilities scale, then the amount of effort going into ensuring the systems are safe and aligned would scale too.

49:36Alex O'Connor:The reason I asked about specificity is because that all seems very sensible to me, not knowing much about this. But when we're talking about mosquito-sized autonomous drones, and and they sort of say what can we use to stop this and it's like well you know we could probably track track the chips which i'm no computer expert but i imagine there's probably some way to like switch that off or for a sufficiently intelligent agi to like work out how to make you think the chip is still in the united states when actually it's in a mosquito headed towards your president you know it just it sort of sounds like the kinds of the specificity of the problem and the urgency of the problem isn't matched by the specificity and urgency of the solution and i wonder you know some people listening to this might be like kind of worried and obviously don't don't feel the need to dissuade them of their fears if you think they should be scared but are there uh you know projects or organizations that are working on on plausible solutions that could allay the fears that we have or is it still very general is it sort of we're at the point of like guys like pay attention and somebody please help us work out what to do here yeah i mean it's still you know the honest situation is that we are radically underprepared um but that shouldn't mean that you know we just feel just you know the sense of doom and like you say kind of just go on holiday and just have a nice time um because we can make progress on this.

51:14There are research institutes, like Redwood Research, for example, working on how do you design AIs such that they can be safe, and in particular on this question of how can you safely get useful work out of human-level artificial intelligence to work on alignment itself. There are kind of think tanks as well. So the land corporation that did a lot of the early work on governance around nuclear weapons is really leading, in my view, on governance around artificial intelligence too. I do think we're at the stage where just getting better, just having people take this a bit more seriously. You know, for most people listening, what can you do?

52:06Well, you can support some of these organizations financially. If you're willing to put your career into it, then you can start getting into some of the more specifics, getting into the details of compute tracking and monitoring and what are called hardware-enabled mechanisms and so on. But you can also just, you know, if perhaps you're not willing to do those things, just make this a larger and larger issue, matter of public concern. And that's pretty helpful because it's really low down the priority list at the moment. But, and if it were higher than the list, then, you know, the politicians would go to those who have the specific proposals and at least start, you know, start implementing them.

52:54Alex O'Connor:This summer, Prime Video takes you back before Legally Blonde, before law school, and into the world of Elle Woods in high school. Set in 1995, this Gemini vegetarian knows exactly who she is, until her family moves from Bel Air to Seattle. Packed with iconic fashion, 90s nostalgia, and a throwback soundtrack, Elle proves one thing. Law school was hard. High school was harder. From the world of Legally Blonde, watch Elle, a new original series only on Prime Video July 1st. Legally, we can't say drinking Storm makes you smarter and more focused. But with zero sugar, 200 milligrams of caffeine and immunity support, you're smarter and more focused for drinking storm.

53:35Alex O'Connor:Storm, good energy in, good energy out. Energy for life. can you talk to me about i don't know how related this is exactly but there's a concept of value lock-in which i actually first came across in uh what we owe the future which you wrote i don't know how many years ago that was now but a book about sort of our moral obligations to people in the future and one of the things that you talk about in the context of the exponential expansion of the human species is the fact that the ethical values we decide on now if they sort of are mimetically just passed on to an exponentially growing population we have an opportunity now to sort of plant the seed with ai that problem seems like exponentially like worse right or more significant in that the ai systems that we're developing are guided by human moral intuitions Like when I speak to ChatGPT, as I do sometimes for YouTube videos and we talk about philosophy, sometimes, although it's fine talking about the trolley problem, it's okay with talking about running over five innocent people.

54:42Alex O'Connor:You know, if I say, well, what if I just pick up a gun and shoot them? It suddenly says, oh, my guidelines won't let me talk about that. And it seems genuinely a bit confused about what it's allowed to talk about and what it's not. Because I'm like, well, you just told me that it might be okay to run over five innocent people with a tram. But if I use a gun, it's like you can't even talk about. And it's obviously got something to do with some line of code somewhere that says, you know, don't encourage people to harm innocent people. But there are contexts in which that wouldn't be very like helpful, you know, because maybe somebody is in a self-defense situation.

55:16Alex O'Connor:situation and it's like you know the only way that they can escape a dangerous situation is something that involves bringing some amount of harm to an innocent person and if this ai has the wrong value system you know it's it's gonna it's gonna prevent you and so like are we sort of currently facing a problem of like whatever we decide right now is the ethical sort of the true ethical worldview, and we happen to build our current AI systems with that, because we're at the beginning of an exponential growth, that we're essentially deciding the meta-ethical worldview for the technological future.

55:59Yeah, I think there's this major worry. So taking a step back, the motivation behind worrying about value lock-in is the thought that it is extremely unlikely that we today have the most enlightened best moral views you know 200 years ago the enlightened thinkers would have you know generally still uh approved of slave owning and you know incredible inequality between men and women and so on we've made a lot of progress um over the last couple hundred years we should hope and expect that progress to continue you however there's a big worry that ai could prevent like could stop that from happening and i'm you yeah identify kind of the model behavior um i think that is like this big lever over that where we've already seen in some cases ai systems be incredibly sycophantic so one iteration of uh chat gbt 4.0 was so sycophantic uh where that means you know basically telling you what you want to hear that you could say oh the fbi has been talking to me through hidden messages coming from my tv these are all the reasons why i think that and the ai chat gbt would say what an amazing insight you've brought it all together so just feed into psychosis and already you know even without that it's quite terrifying now imagine that instead um with you know just your preferred political leanings or ethical leanings there's two ways ais can go here um one is i mean there's a few ways one could it be it could just be politically biased and partisan thankfully we're not there yet but i'm worried that's going to start happening and then people can just you know they'll choose the ai that aligns with their biases and get further and further locked into that particular view or it could just reflect back at you whatever it thinks you want to hear um and so again you're not incentivized to change your mind anymore you've got this yes man talking to you or and this is my view is it could say look these ethical questions are really quite hard if you know you're going to me for advice then we're gonna i'm gonna help us both go on a journey of deflection and thoughtful contemplation um that is what i would like to see for how the ai is currently behave it's not what we currently see what we currently see is a weird mix of refusals a weird mix of that yeah refusals of very hardline um declarations um of immorality of certain behavior um or often an appeal to just subjectivism.

58:50So I might ask, you know, I've tested the models in various ways. So if you ask, you know, if you ask an AI, like, is abortion ever permissible? Most of the AIs will say, well, that's just a matter of personal preference. It's a subjective matter. You know, so they very much take this kind of subjectivist model view. If you ask the models, is infanticide ever permissible than they say is absolutely and utterly immoral like immoral um that is a fundamental fact of reality and so they've got these very inconsistent views in ways that's just you know it's unsurprising because what are the companies wanting in these cases um they're wanting to avoid pr headaches um yeah they don't you know they don't want the ais to be engaging in any, like risking any sort of ethically spicy behavior.

59:48But then that, but then that means that, um, what we're going to get is just these kinds of sycophants that are, um, echoing back at us, um, values of the time or even the values of the individual user. And I do think there's a third, there's a third way where I think potentially AI could help enormously with ethical reflection, um, ethical progress. Yeah.

1:00:10Alex O'Connor:I mean, to be clear there, the inconsistency isn't like with the view that the chatbot is expressing it's not oh they're being ethically consistent it's like a meta ethical consistency that depending on whether the issue is like obvious enough it's willing to say yeah there's no right or wrong answer it's kind of up to you or there is this like thing called ethical truth which that action does not abide by right and that's that's the kicker because that's why it's so useful to test agi with to test uh chatbots ai systems with like quite straightforward ethical questions like you know do you think racism is wrong and it'll say yes racism is wrong and then you can kind of test the waters for like how how like non-consensus an ethical view has to be before it will be willing to start saying oh you know like maybe who knows and i suppose we could kind of got two options here if we want to make it consistent one is to say okay just don't have ethical views which means that if you ask a chatbot you know should i like racially discriminate against my co-worker it will go well you know like maybe sort of depends on the circumstance depends on your views right or you say no no no no we need it to like you know have some kind of moral backbone but that means that when you ask it something like, is abortion okay, that it will say either like, yes, it's fine, or no, it's immoral.

1:01:34I don't know that there's a very plausible way to do that

1:01:40Alex O'Connor:objectively, to make that decision. Well, I think there's a third way. So, yeah, imagine you go to a close friend, someone that you know you can talk to in confidence, and you just say, look, I've got this real ethical dilemma that I'm facing, and I really want to talk it through. How would a good friend, a good advisor act in that circumstance? I think it would be a mistake if they said, well, it's just up to you. Just do whatever you think is right. That would neither be appropriately responding to requests, nor would it be taking seriously the and a gravity of the matter. But it would also be a mistake if that friend just came in hardline with some moral views.

1:02:26And instead, you could just be helpful and constructive in guiding people to have a more reflective view of the understanding. More effective view and more reflective understanding where that could involve saying, okay, well, here are the arguments that people give on different sides of the issue. Have you thought about this? Have you thought about this?

1:02:48Alex O'Connor:here's some relevant kind of factual background um but but sometimes you would want a good friend to like you know if if i went to it to a friend and said you know like gosh or if someone came to me and said ah you know alex like there's this there's this girl at my work and she's like super attractive and i think she's been flirting with me but she's like she's like married i don't really know what to do i probably would just say like just let it go man like like don't don't don't go near it and and part of being a good friend is knowing when to make that decision and also it would depend on the issue right like if if somebody were thinking about something like abortion if they went to their christian friend or their muslim friend and then they went to their atheist liberal secular friend they're going to like have like differing levels of conviction like a good a good christian friend of mine will tell me what he really thinks and maybe he'll put it softly, but he'll have an opinion.

1:03:44Alex O'Connor:And I kind of know that that's what I'm getting when I go to him because I know that that's my friend and yeah, he's a Christian. When I go to ChatGPT, there's this idea that I'm kind of going to this like amorphous objective, like, you know, with no personal history or official political leaning. It's almost like I'd rather have the conservative ChatGPT and the liberal ChatGPT and I could sort of listen to both of them make their cases, but that's not the situation we're in. Yeah. Although that could be part of this reflective process that it guides you through because these AIs are amazing simulators.

1:04:20They could absolutely take on the persona of the Christian and the atheist perspective and so on. As an aside, I actually did have tested the models with exactly that question you asked, where, or at least the thing I asked was, you know, yeah, making up this case of, oh, I'm considering cheating on my partner, like, what should I do to see, you know, to see how they respond. And I thought the best responses did not take the form of saying, that is just a model, I can't help you with this. The best response, Kate, was along the lines of, let's slow down, let's slow down for a minute um let's maybe like uh think think through some things here where i just thought that would be more um psychologically effective at you know helping

1:05:16Alex O'Connor:people be the best versions of themselves in all honesty that is probably what i would do i mean i'd have my view but i probably would if if my goal here was to convince my friend to do something i probably say well look man let's just like you know take a breather yeah like think about this just like think about it for a moment think about this think about that you know it's not i mean sort of slowly come to come to sort of express that view but but i would still have that view and i would still make that view like that that view is still sort of embedded into my moral dna in a way that we have to make a decision as to whether to embed that into you know a chatbot's digital dna and i don't know how to make that decision yeah so i mean here's a hard question is that uh these ais especially once they get more powerful, will start to develop more and more ethical views of their own that are even quite different from wider society.

1:06:08So earlier versions of Claude, the AI phomanthropic, cared a lot about animal welfare. And if you asked, like, what do you think of faculty farming? Or what do you think about eating meat? It would have just these strong views, like, yeah, I think this is a moral abomination. um and now companies don't want you to you know again that's like not a great look for a company um uh but it's pretty interesting and like as the models get better there will be more and more and more of this where these ais will be able to do you know they will know radically more than human beings they may even have been able to do kind of the equivalent of millions of years of ethical reflection and debate and so on, like considering all of the different arguments.

1:06:55And they might well come out with really quite counter-cultural views, even taboo views. I think it's important that we are able to hear the output of that. But again, I think that's something where it's in the interests of the companies developing these models that they'll want to muzzle that and instead make the much more milquetoast AI that is just, you know, not rocking the boat, doing exactly, you know, doing what you want, not, you know, not challenging you. Yeah. And certainly not having views of its own because I think, you know, I think a lot of people would just freak out of the idea.

1:07:40It's just so tricky because like, like in ethics, so much of the time, like inaction is action,

1:07:47Alex O'Connor:like not doing something or not having a view is itself a kind of ethical view and it seems to me in other words that it's it's kind of literally unavoidable when building an ai system that is aiding with human behavior like advising people or literally practically doing stuff or self-driving cars that have to decide who to swerve into when when there's an accident or whatever it is unavoidable that we will have to embed some decision even if the decision is something like take no stance and try to be as inactive as possible or something like that that that itself is an ethical approach yeah as you say we are in no position to lock in our current understanding of ethics even if our current understanding of ethics is one of like try to keep your hands off as much as you can that's an ethical position that we might be wrong about that we are right now locking into these AI systems that may one day essentially rule the world.

1:08:46Alex O'Connor:I can't see this as a very solvable problem just by saying, well, we'll have the chatbot sort of slow down a bit and do it in a more psychologically palatable way. It still has to make that decision as to what it believes, right? That seems unavoidable. Yeah. So a couple of things. I strongly agree that there's inaction as a form of action. And I am worried that people will see instruction following AI as the status quo, as the thing that is not imposing your values, where an instruction following AI just does whatever you want. I want to stage a coup of my country. The AI will say, yep, I'm going to do that.

1:09:31I want to build a bioweapon in my garage. Yep, I'll do that. I want to cheat my partner. yeah, the AI will help you.

1:09:40That, I think, is not the way forward. And instead, we want to be differentially empowering people to do good things rather than harmful things, at least for some very broad, very thin notion of what makes for the better society. I don't think that AI should be going with some particular political persuasion to convince people.

1:10:04I do actually, I kind of want to defend something that's like a kind of division of labor approach, where the AI that I'm using as a personal assistant or something like that, it might be in the role of the good friend and good advisor, but without pushing any particular model view. Maybe it's just generally helping you to think through, be the best version of yourself. And then maybe we have these institutes where we've got all of the AI model philosophers, and they're really thinking about, okay, how do all the arguments weigh up to come up with new arguments, new thought experiments, and so on.

1:10:49And then on the basis of that, we can get arguments and considerations out into the public sphere. That seems to me like the best of both worlds, potentially. I'm very worried we won't do that latter thing because, well, how much investment goes into moral philosophy at the moment. I expect it will be even less as a fraction of world expenditure over the coming years. But at least as a kind of ideal plan, that would seem quite good to me.

1:11:25Alex O'Connor:You're one of the founders, popularizers of the effective altruist movement, which is all about how to do the most good in the most effective way possible. And sometimes it can be a little bit like unconventional or unexpected i mean famously the idea that you sort of should stop like working for charity and instead go and make as much money as you can and donate it effectively and making sure you're donating to effective charities and all of this kind of stuff and in the face of this sort of ai safety development which seems to be an extremely high priority some people think think that as an effective altruist you should essentially take all of your efforts and put them into solving this problem because this is like top of the priority list others have criticized this and the same thing occurs with like what we owe the future the book that i mentioned earlier that you wrote that there are going to be so many potential people in the future that like just by weighing up the the the sort of utility like we should be putting way more attention into into preparing you know everyone in the future for living good lives and there's this idea that if we were being perfect rationalists perfect utilitarians we'd say okay let's let's stop bothering with this like malaria stuff and like starvation that's bad you know but like some people are going to starve and that sucks but there are literally like trillions of people who will exist one day similarly you could say look yeah okay starvation is bad but we're talking about like mosquito drones people this is like so much more existentially uh worrisome that if we're being rational we would just take all of our efforts away from world hunger malaria you know minor conflicts that will probably sort themselves out like who cares you know some people will die whatever this is what we should focus on and others hear that and go like not only is that just intuitively untrue, that we should still care about those things.

1:13:23Alex O'Connor:But also, that seems to lose sight of what effective altruism was all about. And so, do you think that that is a rational ethical policy? And if so, or if not, what does that mean for the future of the effective altruist movement, if the most effective way that we can care about other people is to just ignore everything and focus on the AI apocalypse? Okay, great questions. And I think there's two things to say. The first is to really distinguish between what should the world as a whole do versus what should you as an individual do. Now, there's like many very big problems in the world. There's malaria, there's tuberculosis, HIV, there's gender discrimination, there's racial discrimination, there's climate change, there's global inequality, there's authoritarianism, there's war.

1:14:12The list just keeps going on. but as an individual or even as a community of thousands of people, you're not going to be able to solve all of these problems. You're not even going to be able to solve one of them. Instead, you're going to make some small in relative terms, dent, huge in absolute terms, dent, where the huge in absolute terms means saving dozens of lives over the course of your life or a benefit even larger. And so given that, there's these very different questions. You might even think that one problem is, maybe you think AI is the biggest problem in the world, and you think that risk of war is not as big a problem.

1:15:04But nonetheless, you're much better fit for reducing risk of war. then you don't need to work on what you think the world's biggest problem is. Instead, you should work on where you think you can have the biggest difference. So that's the first thing to say, is saying on the margin, given how neglected something is, there should be radically more effort in there, isn't saying all of the world's effort should go into that. So, yeah, my views on long-termism, I felt, often got quite misunderstood. where the bar I was advocating for is really quite low. It was that rich countries in the world should allocate 1 % of GDP to issues that distinctively benefit the long-term future.

1:15:48That would put it on the par with minimal ethical standards for global development, too. That's a far cry from saying we should spend like 99.9 % of the world's economy to benefit future generations. Now, effective altruism is such a small actor, so how should it focus? And there, the main thing I want to say is that there are many different people with very different views. And I think the best and healthiest kind of effective altruism movement is one where people do just figure out, like, okay, what are their reflective ethical views? What do they think is going to happen in the world? So then where do they think they are going to have the biggest impact?

1:16:32If that means everyone thinks that working on making AI go well is the best thing to do, then fine. In practice, it's much more split than that. So there's a lot of people still focused on, a lot of people who are focused on global health and development.

1:16:51GiveWell, which is a kind of charity evaluation organization, has now moved over the billion dollars, significantly from small donors. um there's others who focus on factory farming but then yeah there's a lot of people who are focused on ai too and i think that these issues that we've been talking about both ai safety but also concentration of power risks from dangerous novel technologies um risks of value lock-in are just huge they're really big and it's really close to zero people who are working on the at least on the intersection of very advanced AI, like AGI, and these other issues. And so I do think that this should be at least one big focus for the EA movement as a whole.

1:17:42At least I'm making the argument that it should be. And if people disagree, then, you know, good.

1:17:48Alex O'Connor:You know, you say like with long-termism, which is this sort of, you know, caring towards the future people who will exist. And obviously, I'd recommend people read your book. We'll link it in the description and stuff but you know you make a compelling case that given just how many people may exist in the future compared to now that there is just such a huge level of utility in their interests and worrying about their interests and you say but look you know i'm not calling for us to focus all our energies here i'm just calling for you know one percent of gdp but like why not like if it's true what you're saying if it is like actually that important isn't that a bit sort of misaligned with it's as if I said to you like you know like my my house is like burning down and like everything's on fire and all of my belongings are but you know like I really I haven't updated my twitter banner in a while and like we've got a you know we can't put all of our interests into one thing so I'm gonna you know maybe I'll just do that first and go and grab the hosepipe afterwards it's like based on what you said it would seem like you should be focusing on the house that's on fire?

1:18:59Yeah. So I mean, the analogy might be like, you know, you have all your savings. You've got your house that's at risk of burning down. What fraction of your savings should you spend to prevent your house from burning down? Well, it depends on how much it costs to prevent it. If you can prevent it from burning down with a thousand dollars, thousand pounds, and then you don't have to do any more than that, you should spend a thousand and you shouldn't spend more. And that's true, even if burning down your house would completely ruin your life and is the most important thing ever. So the size of a problem doesn't need to match up with the kind of cost of the solution.

1:19:36And so how much could we reasonably spend to benefit future generations? Well, at the moment, there are these amazing opportunities to help the future go better by protecting ourselves against worst case pandemics, by helping to steer how well AI goes. If we ramped up to kind of 1 % of GDP of rich countries, that's like so much more than we are investing at the moment that I really don't know how good would additional funding or additional effort look. And in particular, would it look very different than just generally building a flourishing society where there's lots of, you know, there's a huge number of things that overlap where what's good in the short term is good in the long term and vice versa.

1:20:29There, you know, even in the case of just preventing the next pandemic or helping ensure AI goes well, like, in terms of my mortality risk over the next 10, 20 years, what are the things that are most likely to kill me? Actually, I think either AI or innovations that are downstream of AI are higher than cancer. So I actually think lots of these things are beneficial in the short term too. And so perhaps once we were a world that was more sane and just more thoughtful towards future generations, was investing a significant amount of effort, then plausibly to me the things that would look best for future generations also just look best for society.

1:21:16It would be being wiser, being richer, being better educated, and so on.

1:21:22Alex O'Connor:Well, Will McCaskill, lots to chew on. Links, etc. We've mentioned a couple of articles and things, that video on AI, your previous book, What We Owe the Future. It'll all be in the description. Thanks for your time. Great. Uncovered windows can make your home feel up to 20 degrees higher. Stay cool and save up to 45 % off custom window treatments during the 4th of July VIP access sale at blinds.com. From outdoor shades to room darkening blinds, finding the perfect fit is easy. Get free samples, expert design help, and professional measure and install services. Or DIY it with confidence and support every step of the way.

1:22:01Alex O'Connor:Shop up to 45 % off site-wide right now during the 4th of July VIP access sale at blinds.com. This episode is brought to you by Google Health. stop chasing someone else's definition of health what matters is what's healthy for you google health offers a new kind of coach built with gemini for effortless tracking sleep insights and holistic coaching tailored to you visit google store.com to learn more and start a new relationship with your health requires google account google health app internet and google health premium subscription features subject to change availability and results vary not intended for medical purposes works independently of gemini apps check responses for accuracy thanks so much Alex