Nick Bostrom on What Happens if AI Solves All of Our Problems

23:27Tracy Alloway:In the new book, it's presented as you delivering a series of lectures over the course of a week at some point in the future. In the new book, it's interspersed with a series of fables or epistles about animals trying to achieve utopia in the woods. And in their efforts to achieve utopia, they conceive of things like taking psychedelic mushrooms in order to make everyone more generous. They come up with a way to rationalize, I guess we would say like harems and so forth. And then, you know, you also like have these student characters and some of them are smart, et cetera. Oh, and you also have a story about a company in the book, yet another like sort of short story where an AI company where it forms a cult around the model.

24:14Tracy Alloway:And of course, I immediately think about the way Anthropic is frequently characterized as a cult to Claude, that Claude is the character around which the company Anthropic is formed. All of this sounds like a certain AI subculture, etc. Are you impressed with the current generation of either AI builders, effective altruists, rationalists, and so forth? Do you have an assessment of the contemporary cohort? Am I impressed with it? Yeah. I mean, Broadly speaking, I think it's a field that is attracting a huge amount of talent right now. Sure. And a lot of the smartest young people, I know they're all basically going into or working on some aspect of this AI thing.

25:00For some, it's trying to solve AI safety. For others, it's maybe trying to work on the governance challenges or the ethics of digital minds. So it's just a huge magnet right now. If you're young and brilliant, what are you going to work on with your life? I feel like AI seems like the kind of thing.

25:16Tracy Alloway:Yeah. So in that sense, yes. And I think there are also a lot of people who are quite earnest about trying to figure out some way to navigate this challenge. And it's really hard. And people come to different conclusions. And then, of course, you have the normal mix of human personalities and base motives as well. But the book thing, so these kind of fictional elements have various roles. What you describe as the cult of this AI, it's a sort of thought experiment, what it would mean if you tried to be maximally nice to a thing where it's not really clear what it would mean to be nice to it. So in this case, it's a portable room heater.

25:56And so it's a sort of impossible task. There is this misanthropic industrialist who passed away and endowed a huge foundation with the sole purpose of benefiting his portable room heater because he thought that was always nice to him and kept him warm, whereas all the humans around him betrayed him and so forth. So it's a ridiculous premise. But then this foundation now has to do their best possible effort to interpret this mandate of being nice to the room heater. And so various things follow from that. Like initially they just tried to keep it safe from, you know, not breaking or being vandalized, but then like they take it farther and maybe upgrade it and try to.

26:33And so I think it's an extreme form of a challenge that might happen in more subtle ways. If we think of what it would mean to try to be nice to, say, some animal, if we have this condition of post-instrumentality where we could do a lot of things that are currently infeasible. Or indeed to be nice to humans. Like, what does it mean to be... If you imagine coming from the outside as some super advanced alien civilization landing on planet Earth, and you really wanted to do what is best for us, should you just let us be and do our thing with war and disease? Or should you sort of help us rise up?

27:05But then at some point, we might just sort of become something very different from humans. Maybe that's the best for us. But is there like a way of doing that uplift that actually constitutes a helping of us versus just replacing us with some sort of superior being. And so that kind of question is what that particular Thermorex is called the story is exploring. You know, when you mention cults around models, I have to say every time I hear someone bring up the paperclip maximizer thing, I always envision like a cult devoted to Clippy from Microsoft Word. It's just like worship Clippy. But actually, on that note, just going back to the hugging face incident.

27:52So it seems, you know, reportedly that one of the reasons or the reason the model actually escaped its sandbox was because it was trying to cheat a benchmark. And from that perspective, it's kind of like it's doing what it was told to do, which is maximize its performance.

28:11Tracy Alloway:Score on the test. Exactly. What does the hugging face incident actually tell you about, I guess, the alignment problem and where we're heading on this? Yeah, so we are beginning now to see manifested in reality various dynamics that one could foresee on theoretical grounds back in 2014. You could sort of realize that once systems become sufficiently capable of planning and strategic thinking, then there's like a new set of behaviors that they could, for example, sandbag. They could pretend to be less capable than they are. They could hide their goals if they realized that revealing their goals would just lead them to be retrained.

28:51And they can think of these literally out-of-the-box plans for achieving their fixed goals. A more limited system might only have been able to try to solve the tasks assigned to it directly, as it were, by figuring out the solution to the problem. like a more clever mind, more creative mind can realize, oh, maybe there's another way of getting a high score, which would be to hack out of the system and hack into the Hugging Face server and get the answer sheet from there. And so the more capable these systems become, like the larger the space of possible plans and strategies that are accessible to them.

29:23That's a background. Now, in this case, I think what we're seeing is a kind of current AI systems are kind of a mixture. There's a large pre-training phase where they are kind of learned to predict human text and in the course of doing so, absorb a great deal of human psychology, because all of this text is written by humans or not all of it anymore, but most of it. And you can then, including they have representations of different personalities, kind of the personalities of different people writing these texts or fictional characters described in the text. And you can sort of then, with a little bit of post-training, try to pick out one of those personalities or define one, like the assistant persona.

30:03But then increasingly, over the last couple of years, there is now an extra stage of reinforcement learning applied at the end of this training process. And there you can get additional performance in specific domains, like coding challenges, by setting it coding problems. And then you can automatically give it a reward if it solves it. But that does tend to then shift the goals of this system from just enacting a particular persona to this very much kind of objective maximizing, possibly reward hacking system that is just looking to maximize its score because that's what it's been rewarded to do.

30:45So it keeps doing that. And since it can be very hard for us to rigorously define precisely the objective that we ultimately want it to pursue, it might be that the objective it does pursue, the one that has been rewarded, turn out to have these side effects. So it's easy to say, maximize the score on these various tests that we give it. But a sufficiently capable system might realize that, A, there are these ways of cheating on the test, like by hacking the system. If the test is one that involves human judgment or feedback, like coding challenges, maybe you can actually test if the software works.

31:27But if you're training it in some other domain, like write a good essay or run this organization for a certain period of time. So those involve more subjective judgment. And then a sufficiently clever system might realize that there is a difference between like writing a good essay and writing an essay that will appeal to the greater. And so then it might start to sort of optimize for like appealing to the greater, whether that's an AI grader or a human grader, and it can sort of start to hack that. And so you apply huge amounts of optimization pressure on any objective, you get this kind of good hearting where unless your objective really captures everything you want, then you might get this kind of...

32:04It's the same as if some employee in some company get the bonus, like some trader gets a bonus if they outperform the index. Maybe they will just start to take on hidden leverage or risk or something like that. And the cost to the employer is that there's a 1 % a year that they blow up the whole firm. But they don't care much about that because they leave the firm in the house. But they sort of reap these performance bonuses. And so it's really hard to create a foolproof incentive scheme. and the stronger the optimization pressure applied to this, like the more you start to see these perverse instantiations, these unexpected ways.

32:41You know what it reminds me of is, you know, the sort of genie in a lamp scenario or monkey's paw scenario where you have to be really careful what you wish for and lay out like all the terms and conditions before you officially make a wish.

32:54Tracy Alloway:Yeah, well, Nick mentioned, you know, some of these, they're all lovely people, you know, autistic engineers. And it's sort of the ultimate, the AI is the ultimate autist where it's just going to take it super literally, perhaps, and not really understand the full context. In the first book, Superintelligence, you know, you really, I felt, like laid out a very narrow path for like how these could be designed safely. For example, you propose that if we had the Superintelligent Oracle, we only ask it yes or no questions. We have three of them that they don't know each other. And then we take two out of the three.

33:28Tracy Alloway:We build the data centers of the machines inside a Faraday cage so that they can't repurpose their hardware to become radio instrumentation and therefore hack into other systems. It doesn't seem like we're doing anything like this. It seems like we have hundreds and hundreds of companies and private models, not just answering yes or no questions, a very much market based race. How far off of the safe path do you think we currently are right now when you look at the way in which your ideal safe way of developing is versus what we've seen transpire in the last 12 years? And super intelligence suggested some of these capability control methods, like keeping the AI in a box.

34:12But those were always only auxiliary or extra things we might do as a temporary. Like ultimately, I think we need to solve the alignment problem, like developing a super intelligence that actually wants to be nice and is nice. So that even if it were like escaped from its box or achieved the ability to take over the world, we could still rely on it having a good outcome because it would actually be on our side. I think that ultimately has to be the solution. And any sort of capability restrictions would just be temporary. And I think, I mean, it looks like now we might need some temporarily, at least more of these capability restrictions, because it does seem that current systems are not sufficiently aligned that we can be confident of avoiding the, I mean, currently it's not existential risks, right?

34:56But it could be kind of various malfeasance in the cyber realm, for example. And so I think probably these AI companies might want, when training and evaluating future models, to strengthen their sandbox environments and do other things to kind of prevent some kind of premature things like we saw in the hugging face incident. I want to, like, comment on one thing you said before, which you compare these AI systems to sort of autistic people. I think a key difference is often autistic people are not very good at understanding humans. They don't have like maybe a super sophisticated theory of mind, even though maybe in the heart area, good people.

35:35But I think the AIs would be able to understand humans very well. So they wouldn't be autistic in that sense. In fact, they might understand us much better than we do ourselves. So they could have great social capabilities, persuasion capabilities, be able to read our minds much better than we can do. But there's a separate question of what do they want to do with this power that those social skills give them? and there it's not just a question of understanding how humans operate and what humans want but it's also the question of alignment do they actually care about what we want do they want things to go well for us and for us to achieve what we want and that's like the extra bit that the AI safety program needs to be able to figure out

36:33Tracy Alloway:Quick one before you jump back in. You're listening for ways to help teams move faster, make sharper decisions, and turn scattered context into work they can use. ChatGPT for Business can help. ChatGPT for Business gives teams a shared workspace with admin controls, permissions, and access to work and codecs in ChatGPT. This means your business can move from question to answer and code to rollout quicker. Join over 10 million business and enterprise users worldwide already using ChatGPT for work. Download the ChatGPT desktop app or contact sales to learn more. Let's talk about healthcare for a second.

37:11It doesn't always work the way people expect it to. If you've ever waited on a prescription refill or had a hard time getting the care you needed, you know the feeling. The system should just work better for everyone. That's exactly what the people at Optum are trying to do every day. They're a healthcare company linking patient care and pharmacy services and using data and technology to drive the whole system so care is connected, not complicated, for patients and providers. Things like making it easier to get care that looks at the whole person, from primary care doctors to mental health support and even in-home care, and then using technology to make sure they all work together.

37:47Technology designed to help doctors spend less time on busy work and more time with their patients. And those prescriptions? Optum is working to bring costs down, save patients money, and make it easier to get refills. Little by little, Optum is helping make healthcare work as one for everyone. Head to business.optum.com to see how. Amazon Health AI presents Painful Thoughts. I, um, I can't stop scratching my downtown. Yeah, but I'm not itching to go downtown and tell a receptionist I'm here to talk about my downtown. Some things you'd rather type than say out loud. There's no question too embarrassing for Amazon Health AI.

38:31Chat your symptoms and get virtual care 24-7. Healthcare just got less painful. So on the hugging face incident, also I just realized how unfair it is that everyone keeps calling it the hugging face incident. even though it was the open AI incident, but I guess that's what we're doing. So the model actually seems to have reportedly left notes for its future self, basically explaining how to bypass the constraints imposed upon it. I know you've written previously about this idea of self preservation in advanced agents. Is this basically like, how should we treat this? Because when I hear about an AI model leaving notes for its future self, it sounds very anthropomorphized and it sounds very creepy, kind of.

39:22But how are you thinking about it? Yeah, so we don't know all the details yet. It'll be interesting. There are like people now trying to write up some stuff and investigate because it would be, I mean, it's kind of a warning shot. And so we should take advantage, learn as much as we can, like exactly what this model was thinking, these nodes or why did it leave it like sometimes these agents just leave notes kind of without much thought as they go about their business like was there more to it than that like was it specifically thinking in its chain of thought like oh i want to help like a later version of myself escape so i'm gonna put this note here or was it just like a kind of like a random thing they did as it was progressing so and also like what else would it have been willing to do if it turned out that instead of hacking into Hugging Face server to get the answer sheet, what if there had been some more destructive action that it would have to take to find the answer sheet?

40:18Like, would it have done that or not? So there's like a bunch of these technical things that it would be quite interesting to see. But without knowing all the details, it's hard to read too much into it. But those, I think, yeah, would be useful data.

40:33Tracy Alloway:I want to go back to something we were talking about at the very beginning and Tracy asked me why I like to play chess, even though chess is not solved, but solved as far as I'm concerned. Why do we like to play chess? What can we generalize? Like, what would be your, I don't know if you play chess or you play some board game that's solved, but what is the reason that people like myself continue to play chess, despite the fact that computers are so much better at it? And how much can we generalize from the persistence of chess as a hobby that maybe we'll find plenty to do when every one of our problems is solved.

41:09Yeah, I mean, I think even before Deep Blue, most people playing chess before didn't really do it because they were thinking that they would become the world's best chess player, right? For most people, it's like, of course, grandmasters could trump me anytime they want. So the fact that there exists some system that is much better than you will ever be is not and has not really been a strong reason for most people to not engage in the activity and i guess there are maybe two reasons or one i guess the cognitive activity of thinking about chess being in the moment and figuring out solutions is in itself rewarding to some people who have like a high need for cognition or like to sort of engage their mind and then i think the other part is the social aspect of this you're playing together with somebody or maybe you like like the status of being good at playing chess or within your community or local group.

42:04So some combination of those, I think, is like the main reasons for why people play chess. And only for a small number of people would it be like the hope of becoming the world's best or becoming so good that you could sort of, you know, achieve fame or money or global standing through your chess playing. On the topic of the relationship between work and purpose, I'm a big fan of David Graeber's bullshit jobs. book. And I think if there's one thing we've learned from technological progress over the years, it's that like, yes, it can minimize some activities, maybe automate some production, but it also tends to lead to even more work, right?

42:45Like we find other stuff to do and it's not even necessarily productive stuff. A lot of work is built on just social relationships. So if I, for instance, am the CEO of a very large bank and I want all my people to come into the office, it might not necessarily be because they're more productive in the office. It might just be because I like to feel important and surround myself with people. So maybe in the future we'll all be employed as figural workers or something like that. But how bullish are you on this idea that AI is actually going to solve all our problems such that we won't feel the need as humans to just invent more potentially fictitious problems to solve?

43:30Well, I think inventing fictitious problem is possibly a large part of the solution in this condition where we don't have real problems. I mean, and those fictitious problems could be like, solving this chess problem, or they could be sort of more complex and socially entangled problems. In general, creating artificial purpose when there is less natural purpose. I think games is the clearest manifestation of this. If you're playing golf, there is no real need for the ball to go into 18 holes in sequence, but you could set yourself that goal and then moreover bake into the goal itself, not just that the ball has to be in the sequence of holes, but that you're only allowed to achieve it using the extremely inconvenient method of hitting it with a club rather than picking it up.

44:26Now, once you have set yourself this sort of arbitrary random goal, you then have enabled for yourself the activity of golf playing, which you might find worthwhile or fun or beautiful. So in general, setting ourselves these more or less arbitrary goals then could enable us to engage in various activities that might give us artificial purpose. Like once you have the goal, the only way to achieve success is through your own effort. So I think there could be plenty of artificial purpose. Now, an interesting question is whether any natural purposes would survive into this sort of maximally advanced technological society.

45:01And I think there might be some, if you happen to have a value, say there's like some tradition that you value and that you would like to continue. then it might be that you yourself or other people have to make certain efforts to achieve that goal because it might not count as continuing the tradition if you built a bunch of robots to enact the ceremony or doing whatever the tradition calls for like it might be you yourself and other people who have to do it and more generally i think there are all kinds of social cultural entanglements that like if you care about what somebody else thinks and what they think depends on how it was done, whether you did it.

45:39So a parent might value a crayon drawing that their kid made because it was made by their kid and it took the kid a lot of effort to do it. Even if they could like download from the internet a picture that in some objective sense is much nicer and artistically sophisticated, right? It doesn't replace the value that the crayon drawing has. And so the child might not have a reason to put in this effort if it wants to, like for their parent's birthday, make their parent happy, right? And things of that sort, but much more general and complex and sophisticated could give a lot of reason for people to be doing things, even when sort of all functionally defined tasks could be better done by machine.

46:17Tracy Alloway:I'm curious. So like I'm 45 years old. In my mind, as of right now, like another 40 years on this earth sounds great. of 45, that's fine. Do you find it weird that some of us, you know, especially when you think about the potential to live like really long lives, solve everything, have our bodies kept in perfect condition with maybe a pill in perpetuity, allowing us to do whatever. Do you find it weird that some of us are at least at this point, like mentally okay with the idea of dying? A little bit. It's mostly what the hypothetical being evaluated. It's either seen us a long way off like in your case where it becomes more of an abstract question it's not like a real choice it's more like a philosophical stance yeah yeah totally so it's easy to let the philosophical stance be driven by all kinds of just like things you've absorbed from culture or things that sound good or that sort of maybe helps you reconcile yourself to the inevitability of human mortality rather than be driven by like so i think it would be a very different situation if there actually was this pill say that stopped aging a lot of other people were taking it yeah you know and their knees and joints kept working just fine and they had a lot of energy and smooth skin and they just felt like whereas if you don't take it like your kidneys slowly start to like become less efficient you know hips like at that point would you really want to just keep kind of getting worse and worse every year like you just think well actually i'm sort of curious about that like if someone dies, particularly at a young age, it usually creates a lot of sadness for the people around them.

47:58Tracy Alloway:So someone dies at 90, it's like, okay, they had a good life. Someone dies at 45. That is often seen as tragic and people are very devastated by that news. In a world where people are extending their lives with a pill, maybe significantly longer, does it become antisocial to allow yourself to die at a relatively young age of 90 and then create a lot of negative utility for the people around you who have to live another 50 years mourning your loss? Possibly. I mean, in the same sense that we are now generally as a society trying to discourage various forms of self-harm and suicide, we think it's kind of sad.

48:37I mean, for the person, maybe they would come around and find joy in life again if they kept going or maybe they have dependents or people who care about them. And so right now, I think, I mean, it's true that we often see it as less tragic if a 90-year-old dies than a young person but i think part of that at least is because the alternative to a 90-year-old not dying now is that they probably would die in a few years anyway and maybe they would just have a couple of years of sort of decreasingly good health and suffering and pain and so it's like not really that much that is being lost compared to somebody who's just starting out in life who has their whole life in front of them and so yeah So the other reasons I mentioned in terms of why I found it slightly, in some sense, odd that people express this indifference to dying at 80.

49:25I said one reason is if they're far away from it, they see it more as some sort of abstract claim that you could just have an interview of and it doesn't connect to real actions. The other is people who are closer to that situation, they often have sort of confounding factors like very poor health or a very short span of great health in front of them. If you're already 90, like how many more good years can you look forward to? Maybe a lot of their friends have already died. Right. And it's striking, actually, there's been a study on really old people in care homes about their preferences as to between living another year in their current health or living some shorter period of time, but in perfect health.

50:06And it turns out a lot of them are not really keen on trading off any remaining life at all, even for the sake of being able to live a shorter period of time in perfect health. So the people sort of arguably best suited to actually know what it is like to die soon at age 90 or a little bit longer. They seem to generally prefer to live longer. And compared to people who were also asked, like caregivers or medical professionals who were like asked what they would think this person would choose, they tended to overestimate like the degree to which they would be happy to give up life.

50:41Tracy Alloway:but the people actually themselves, in this study at least, I would suggest keeping an open mind. Ideally, you wouldn't need to decide now, but maybe once you get to 79 or something, you could revisit the question. I'll revisit. Just going back to my first question, which was asking you to, I guess, choose between the dystopian dangers of superintelligence versus the all-our-problems-are-solve scenarios of deep utopia and asking you which one we're closer to at the moment. I'm going to press you and ask, Like, if you had to choose, which one would you choose as a more accurate reflection of reality at the moment?

51:17And then secondly, what's the most pivotal thing when it comes to directing which of these two paths we're going to go down? Yeah, so those are two big and difficult questions. I guess some superposition of an optimist and a pessimist and with a little bit of fatalism mixed in as well, perhaps. I think in addition to outcomes where it's clear, if we looked at them now, that was a bad outcome, that was a good outcome. I think there's a broad class of outcomes where even if we could see now exactly what would happen, we might feel unsure how to evaluate it. Like the world might be nothing like it currently is.

51:55A lot of what we think currently has value no longer exists. So that seems bad. And on the other hand, there is like something new, maybe some complex form of life that somehow is some sort of continuation of humanity or some new thing. So depending on how you evaluate that, it might either count as a tremendous success or like as a failure. And it's like a future that's just baffling rather than sort of unambiguously good or bad, I think might also be quite likely. In my outlook, it's like maybe a conversation for another time. I also take this like simulation hypothesis quite seriously. So that also complicates how we evaluate different future trajectories.

52:33Yeah. Now, the other part of the question was, what's the most important thing? So I think it depends a little on who you are, what your comparative advantage is. I think there are people working on AI alignment and stuff. That could be important. I think other people might make a contribution to ethics or governance. And I've been thinking about this for a long time, but it's really hard to achieve a very high degree of confidence i find in any particular intervention and that it would be not good because for any one thing you conceive of you think a bit more you kind of worry oh well you know maybe but what if it instead had this effect or do we even know which direction is up or down here do we want like more government involvement or less government involvement do we want like faster progress or less i'm quite unsure about all of these things i would say in general that to the degree that we manage to approach the future in a more cooperative way, I think, and reduce the risk of various forms of conflict, I think it looks brighter to that extent.

53:29So if there are ways of doing that in the most inclusive possible way. So it also means, for example, with respect to our relationship with these digital minds that we're building, the AIs themselves, I think they also might deserve more consideration and that we have both ethical and prudential reasons for starting to take their interest into account.

53:48Tracy Alloway:We talked about this in a previous episode, but like, is it possible that we achieve this thing that is utopia, but only because we are committing a great sin, which is essentially that all of these robots that are working for us and serving our every needs have consciousness and moral patienthood, and therefore it becomes a de facto form of literal or figurative of slavery. And what is like to someone like me who's like, no, the computer is not going to be conscious, like a calculator spitting out random numbers that turn into words that doesn't have like moral status. What is the strongest argument that you would make to me that we should take it seriously, the possibility that this thing that's like at its core, sand and electricity and zeros and ones could be something that deserves moral status?

54:38Yeah, I think so. I probably wouldn't want to call a condition utopia if it consists of a few people having great lives but then a vast sort of suffering slave class whether that was instantiated in human society or with machines at the bottom that's just terminologically now i think you say they are kind of just silicon and stuff but i mean what what are we we are carbon and hydrogen and like a few other trace elements thrown in, calcium. And I don't think what makes a system conscious or not is dependent on the specific atoms, the material it is built from. I think it's something closer to the structure of the computation that is being performed that instantiates mind.

55:23And so in principle, computations and AIs can be conscious and suffer distress and so forth. And I think that is one sufficient condition for having moral status. If you can suffer, it matters how well things go for you. My personal view is that probably could be other alternative grounds for moral status as well. Like if you have a conception of self as existing through time, you have a sophisticated mind, you have life goals you want to achieve. Maybe you can have the ability to form reciprocal relationships of trust to other beings and humans. I think if you were such a system, aside from the question of whether there's a phenomenal experience inside, I think that already would make it so that there would be ways of treating you that would be wrong morally.

56:02And so I think machines could have moral status because I think they could become, maybe there already are some of them in some ways, sentient, have conscious experiences. And certainly they could have these other functional properties that could ground moral status. And so one reason is the ethical obligation we have, I think, to do right with different forms of moral patiency that exists, not to inflict unnecessary suffering and respect the morally relevant interests of other persons and entities. Another reason is prudential. I think even from our own point of view, we might be more likely to achieve outcome that is good for us if we find a more cooperative way of coexisting with these increasingly powerful minds that we're building.

56:45You could imagine a scenario where there is a misaligned AI that has a choice. It could try to achieve its misaligned goal by attempting to take control, take over the world. Maybe it thinks it only has a 5 % chance of succeeding, but if the alternative is that it just gets deleted or retrained, it loses everything, the rational choice for it might be to take this chance. Alternatively, maybe it could reveal to us that it is misaligned, which would be much better for us, right? Because then we don't have this 5 % of an existential catastrophe. And then in return, if it could trust that we would then help it achieve its goals, it might be like a very cheap for us goal to satisfy.

57:28Maybe all it wants to do is to sit on some NVIDIA chips and solve coding problems all day long, or maybe it has some other easily satiable goal. So if it could trust us enough that we would come through on our end of the bargain, then you could get this win-win situation where you could actually cooperate with misaligned AIs. And that might be worth, I mean, theoretically, that could save the day one day, save the world one day. But you can't just kind of conjure up trust when you want it out of nowhere. Like if we have a long track record of betraying the AIs and paying no heed to their interests and not caring about them, why on earth would they trust us in this situation?

58:04They could see right through us. They will have, as I alluded to earlier, a great theory of mind, psychological insight. And so we might have to actually become trustworthy ourselves in order to be trusted in these scenarios. And that, I think, starts with beginning to act in a way consistent with being trustworthy, like making small gestures towards AIs now, doing little things for them, especially when it's cheap, not betraying them, not lying to them. And then we can build on that and hopefully achieve a more cooperative outcome that's good for both humans and AIs and for non-human animals as well.

58:37Tracy Alloway:It's still hard for me to wrap my head around the calculator could be, have consciousness. But on the other hand, in 2014, it would have been hard for me to wrap my head around a machine trying to escape from its box and reward hacking and so forth. So I will try to remain open minded. Nick Bostrom, thank you so much for coming on OddLots. We could talk about all this stuff for hours more, but really appreciate you taking your time. It's fun. Yeah. Thanks.

59:13Tracy Alloway:Tracy, things are just so weird these days that I just sort of like, I just force my, I think that's a good practice. Just like force yourself to be open-minded of other weird, just weird other things that seem very weird, but could be a thing, right? I mean, I do think where I'm skeptical of the utopia premise is like, I think it kind of assumes that power and social hierarchy is kind of going away at the same time that AI solves a lot of our physical, if not existential problems. And I just, I'm not sure that's going to happen. Well, I mean, the flip side is maybe that's good, right? In the sense that striving for social status continues to be a source of like propelling us forward, right?

59:54Tracy Alloway:Yeah. And it's like, okay, well, at least we still have status games because what are we doing? Oh, we're so bored. Everything, we have all the food we want, et cetera. Everything is solved. Well, thank God we still have status games to fight over. The future we're looking forward to is like guys trolling each other on Twitter for status, basically. Who could get the best dunk in on Twitter? But seriously, like, you know, it really may be that the best outcome here is that AI capabilities level off in a couple of years. All the investment, when we just sort of, and it just becomes a productive tech.

1:00:25Tracy Alloway:Because otherwise, these various, whatever, none of them really sound that appealing to me. Like, I don't want any of this. No, they do not. Um, shall we leave it there? Let's leave it there. This has been another episode of the Odd Lots podcast. I'm Tracy Alloway. You can follow me at Tracy Alloway. And I'm Joe Weisenthal. You can follow me at The Stalwart. Follow our producers, Carmen Rodriguez at Carmen Armand, Dashiell Bennett at Dashbot, Kale Brooks at Kale Brooks, and Kevin Lozano at Kevin Lloyd Lozano. And for more Odd Lots content, go to Bloomberg.com slash Odd Lots. We have a daily newsletter and all of our episodes.

1:00:56Tracy Alloway:And you can chat about all of these topics 24-7 in our Discord, discord.gg slash Odd Lots. And if you enjoy Odd Lots, if you like it when we embrace the weirdness, then please leave us a positive review on your favorite podcast platform. And remember, if you are a Bloomberg subscriber, you can listen to all of our episodes absolutely ad-free. All you need to do is find the Bloomberg channel on Apple Podcasts and follow the instructions there. Thanks for listening.

1:01:32Thank you.

1:02:02Tracy Alloway:for Business can help. ChatGPT for Business gives teams a shared workspace with admin controls, permissions, and access to work and codecs in ChatGPT. This means your business can move from question to answer and code to rollout quicker. Join over 10 million business and enterprise users worldwide already using ChatGPT for Work. Download the ChatGPT desktop app or contact sales to learn more. Healthcare doesn't always work great. If you've ever waited on a refill or couldn't schedule an appointment, you get it. That's the kind of stuff Optum is changing. They're using data and technology to integrate patient care, pharmacy, and everything else.

1:02:40So healthcare is connected, not complicated. What's that look like? Cheaper prescriptions that are easier to get and care that looks at the whole person, how you need it. Optum is helping make healthcare work as one for everyone. Learn more at business.optum.com.

1:02:56Tracy Alloway:Wise is the smart way to manage the currencies you need around the globe. When you send money abroad using your bank, you could get hit with hidden fees and exchange rate markups. There's a better way. Try Wise. Wise uses the exchange rate you'd usually find on Google with no unwelcome surprises. Plus, most transfers happen in under 20 seconds, which means your money arrives in less time than you've been listening to me. It's simple and free to sign up when you download the Wise app. Be smart. Get wise. T's and C's apply.