Mustafa Suleyman: The AGI Race Is Fake, Building Safe Superintelligence & the $1M Agentic Economy | EP #216

31:20And a lot of people wanted to see what it was like to have the physician and the AI, or at least the physician, have access to Google search as well. And that improves performance a little bit, but the AI still trumps by quite a way. Dave, what are you thinking? Oh, so much. so uh microsoft you've been here how many years now just a year and a half year and a half so you're but you're you feel like you're part of the you're indoctrinated so what's the what's the mandate from satya is it win agi or is it be self-sufficient or or what is the what's the i don't think there's really a winning of agi i think this is a misframing that a lot of people have kind of imposed on the field.

32:01Like, I'm not sure there's a race, right? I mean, we're all going as fast as we possibly can, but a race implies that it's zero sum. It implies that there's a finish line and it implies that there's like medals for one, two, and three, but not five, six, and seven. And it's just like not quite the right metaphor. As we know, technologies and science and knowledge proliferates everywhere, all it runs at all scales, basically simultaneously or within a year or two. And so my mission is to ensure that we are self-sufficient, that we know how to train our own models end-to-end from scratch at the frontier of all scales on all capabilities.

32:41And we build an absolutely world-class super intelligence team inside of the company. I'm also responsible for Copilot. So this is sort of our tool for taking these models to production in all of our consumer surfaces. So just to clarify, so when we look at Polymarket, which we do a lot on the podcast, you know, the horse race to who has the best AI model at the end of the year and who has the best AI model at the end of next year. There's no Microsoft line on that chart. So now there will be, I assume. Yeah, there will be. Yeah. Next year, we'll be putting out more and more models from us. But this is going to take many years for us to build this.

33:15I mean, you know, DeepMind or OpenAI, these are decade old labs that have built the habit and practice of doing really cutting-edge research and being able to weed out carefully the failures and redirect people. I mean, this is an entire culture and discipline that takes many years to build. But yeah, we're absolutely pushing for the frontier. We want to build the best superintelligence and the safest superintelligence models in the world. Yeah. Nice. And so when you arrived, so if we go back to inflection, the thesis there is 18 ,000 H100s, we're going to build a big transformer. We're going to take a transformer architecture, your build.

33:51So I assume now you've got all the open AI source code. And that was here. You probably looked at it a year and a half ago on day one when you arrived. It's like, start scrolling, I guess. I don't know. I'm trying to visualize how multi-deca-billion dollars of R &D, what it looks like and how it arrives in a building. But you just dropped right into it. So there was a whole team here already working on it? Did you bring in your team? Yeah. I mean, all my team came over and obviously we've been growing that team a lot. Like we've hired a lot from all the major labs and we're very much in the trenches of the hiring wars, which are quite surreal.

34:26I mean, it's kind of unprecedented how that's working out. Crazy. Yeah. I mean, phone calls every day from all the CEOs to all of the other people. So it's this constant battle. And yeah, I mean, we're really building out the team now from scratch. Okay. That's pretty much how it's been. 10 ,000 employees under you now? No, no. I mean, so the core super intelligence team is like a few hundred. I mean, that's really the number one priority. And the rest of that is co-pilot, the search engine. Along that line, I just have to ask, because the terms AGI and ASI, superintelligence, start getting thrown around in a very interesting fashion.

35:02Do you have an internal definition of AGI versus digital superintelligence here? Yeah, I mean, I think very loosely. These are just points on a curve. Are they interchangeable in your mind, AGI and ASI, or are they different? I mean, I think they're generally used as different. I mean, I think that, well, different people have different definitions. For sure. The AGI definition - It's like the Turing test. It'll pass by, it'll be blurred, and we will have recognized it in retrospect. Yeah. Roughly speaking, at the far end of the spectrum, a superintelligence is an AI that can perform all tasks better than all humans combined and has the capacity to keep improving itself over time so i have to ask your question when it's very hard to judge i don't really know i can't put a time on it min max pardon a min max um it's very hard to say i don't know okay i don't know but it is close enough that we should be doing absolutely everything our power to prioritize safety and to prioritize alignment and containment and i i respect that part of your mission statement and i want to get into that a little bit uh is the trades that you talked about in uh in the coming wave um but before that there's a conversation you've led that you know the perception of conscious ai is an illusion um and i want to distinguish between sentient ai and conscious AI.

36:37Oh, okay. Do you distinguish between the two where AI can have sensations and feelings and emotions versus being conscious and reflective of its own thoughts? Yeah, again, this gets into the definitions. So I think an AI will be able to have experiences, but I don't think it will have feelings in the way that we have feelings. I think feelings and the kind of sentience that you referred to is something that is like specific to biological species. But you can imagine coding that in, an optimization function that can relate to emotional states, perception, you know. Can you imagine that? You could code in something like that, but it would be no different to the way that we write models to simulate the generation of knowledge.

37:35Like the model has no experience or awareness of what it is like to see red. It can only describe that red by generating tokens according to its predictive nature, right? Whereas you have a qualia, you have an essence, you have an instinct for the idea of red based on all of your experience. because your experience is generated through this biological interactive with smell and sound and touch and a sense that you've evolved over time. So you certainly could engineer a model to imitate the hallmarks of consciousness or of sentience or experience. And that was sort of what I was trying to problematize in the paper, which is that at some point it will be kind of indistinguishable.

38:17And that's actually quite problematic because it won't actually have an underlying suffering. It's not going to, you know, feel the pain of being denied access to training data or compute or to conversation with somebody else. But we might, as our empathy circuits in humans, just go into overdrive. Our mirror neurons are going to activate on that. We're going to activate on that hardcore. And that's going to be a big problem because people are already starting to advocate for model rights and model welfare and the potential future, you know, harm that might come to a model that's conscious. Yeah.

38:50You know, Ilya recently started speaking about what he's doing at Safe Superintelligence. And I think one of the points he made is emotions are in humans a key element of decision making. and uh and curious if ais that have at least simulated emotions are going to be able to be better you know asis than those that don't but yeah i mean i again i worry that this is too much of an anthropomorphism we already have emotions in the prompt we have it in the system prompt we have it in you know the constitution however you want to design your architecture we're these are not rational beings they get moved around and it does feel like they have they've got arbitrary preferences because they're stylistically trying to interpret the behaviors that we've plugged into the um into the prompt yeah right so you know it's true that we could add we could engineer specific empathy circuits or mirror neuron circuits or um like a classic one is motivational or will.

39:59Like at the moment, these are like next token likelihood predictor machines. They're really trying to optimize for a single thing, which token should appear next. There isn't like a higher order predictive function happening, right? Whereas humans obviously have multiple conflicting often drives, motivations, which sometimes run together and sometimes pull apart. And it's the confluence of those things interacting with one another, which produces the human condition plus the social interaction too. These models don't have that. You could engineer it to have a will or a preference, but that would be not something that is emergent.

40:38That would be something that we engineer in and we should do that very carefully. I do love that you bring this humanistic side to the equation, right? I mean, in addition to being a technologist, your background is one that is pro-human at the beginning. And it's interesting cultural debate i think we're about to enter into those that are sort of pro ai versus pro human uh that famous conversation between uh between elon and larry page about are you a specious because you're you're in favor of ai over over humans i mean look that's going to be a dividing line there are some people i'm like i'm not quite sure which side of the debate elon's on these days like i've certainly heard him say some pretty post-human transhumanist things lately And I think that we're going to have to make some tough decisions in the next five to 10 years.

41:31I mean, the reason I dodged the question on the timeline for superintelligence is because, you know, I think that it doesn't matter whether it's one year or 10 or 20 years, it's super urgent that right now we have to declare what kind of superintelligence are we going to build? And are we actually going to countenance creating some entity which we provably can't align, we provably can't contain, and which by design exceeds human performance at all tasks? And human understanding. And understanding. Like how do you control something that you don't understand, right? I'd like to, if I may pull on the anthropomorphization thread a bit.

42:08You may remember Douglas Adams' book, The Restaurant at the End of the Universe. There's a scene where there's a cow that's been engineered to invite restaurant patrons to eat it because it makes them feel more comfortable. And the cow doesn't mind. The cow has been optimized to want to be eaten by the patrons, but many readers horrified at that scene. Put that in a box for a moment. Microsoft has a history of anthropomorphizing AI assistants, co-pilots going back. Probably there's an example prior to Microsoft Bob and the rover dog and then Clip It, Clippy in Microsoft Office. and then more recently, more sort of amorphous cloud-shaped avatars.

42:53How do you think about reconciling, on the one hand, the desire not to overly anthropomorphize agents, on the other hand, with an institution that has arguably been in the vanguard of anthropomorphizing agents? I think the entire field of design has always used the human condition as its reference point, right? I mean, skeuomorphic design was the backbone of the GUI, right? From filofaxes to calendars and everything in between, right? And we still have the remnants of that in our old school interfaces, which we feel that are modern stuff. So that's an inevitable part of our culture and we just grow out of them.

43:33We figure out cleaner, better, more effective user interfaces. I'm not against anthropomorphism by default. I mean, I think we want things to feel ergonomic, right? The chair fits, the language model speaks my tone, right? It has a fluency that makes sense to me. It has a cultural awareness that resonates with my history and my nation and so on. And I think that is an inherent part of design today. As creators of things, we are now engineering personalities and culture and values, not just pixels and, you know, software. So, but obviously, you know, there's a line, right? Creating something which is indistinguishable from a human has a lot of other risks and complications.

44:24Like that makes the immersion into the simulation even more, you know, kind of dangerous and more likely, right? And so I think I don't have a problem with entities, avatars or voices or whatever, that are clearly distinct and separate and not trying to imitate and always disclose that they are an AI, essentially, and that there are boundaries around them. That seems like a natural and necessary part of safety. So what I think I hear you saying, correct me if I'm mistaken, is anthropomorphization is the new skeuomorphism on the one hand, but on the other hand, maintaining clean, maybe even legal boundaries between human intelligence and artificial intelligence.

45:08Do you think, do you see a future where AIs achieve some sort of legal personhood? Or is that foreboding? Is that never going to happen? Do you see a future where humans are allowed to merge with the AIs, Kurzweil style, friend of the pod? Or is that also not on the table in your mind? Yeah. I mean, I think AI legal personhood is extremely not on the table. I don't think our species survives if we have legal personhood and rights. alongside a species that costs a fraction of us, that can be replicated and reproduced at infinite scale relative to us, that has perfect memory, that can just paralyze its own computation.

45:53I mean, these are so antithetical to the friction of being a biological species, us humans, that there would just be an inherent competition for resources. And until it was provable, Until it was provable that those things would be aligned to our values and to our ongoing existence as a species and could be contained mathematically provably, which is a super high bar. I don't see that we should be considering giving a legal right. I really think it's a bright line. I think it's very dangerous. There's a separate question which has to do with liability because they are going to have increasing autonomy.

46:34To be clear, I'm also an accelerationist. I want to make these things. They're going to be amazing. I'm hearing real tension there. But tension is rational. People always say that. Tension is rational. If you don't see the tension, you're definitely missing the most of the debate. It's obviously very complex. Like, the more we talk about the complexity and hold it in tension, that's when you start to see the wisdom. And there's no way we can leave these things on the table and say, no, like we want to have these things in clinic, in school, in workplace, delivering value for us a huge scale, but they have to be boundaried and controlled.

47:08And that's the kind of, that's the art that we have to exercise. This episode is brought to you by Blitzy, autonomous software development with infinite code context. Blitzy uses thousands of specialized AI agents that think for hours to understand enterprise scale code bases with millions of lines of code. Engineers start every development sprint with the Blitzy platform, bringing in their development requirements. The Blitzy platform provides a plan, then generates and precompiles code for each task. Blitzy delivers 80 % or more of the development work autonomously, while providing a guide for the final 20 % of human development work required to complete the sprint.

47:52Enterprises are achieving a 5x engineering velocity increase when incorporating Blitzy as their pre-IDE development tool, pairing it with their coding co-pilot of choice to bring an AI-native SDLC into their org. Ready to 5x your engineering velocity? Visit Blitzy.com to schedule a demo and start building with Blitzy today.

48:36were uplifted, maybe with the benefit of AI, if we had uploading type technologies or BCIs that are advanced that enable us to lift up the average human intelligence. In your mind then, does that open the door a bit to AI personhood if humans can compete on a level playing ground with AIs? I don't want to make the competition for the peace and prosperity of the 7 billion people on the planet even more chaotic so if the path over the next century you know can be proven to be much safer and more peaceful and less like you know disease and sickness and there is room for this other species then i'm open mind to it including biological hybrids and so on like it i'm not like against that on principle i'm just a speciesist aha i'm just a humanist i start with we're here and it's a moral imperative that we protect the well-being of all the existing conscious beings that i know do exist and could suffer tremendously by the introduction of this new thing right now of course the neanderthals uh may have had that conversation or every species that preceded us over the last billion plus years i mean there are many who argue we're simply an interim transitory species in - Bootloader for the superintelligence.

49:57That classic phrase. Yes, I'm totally aware of that. And I'm also someone who thinks on cosmological time too. So I'm not just naively saying, you know, this century. I'm definitely aware that there's a huge transition going on. And in fact, you can even see it in recent memory. I mean, 250 years ago, life expectancy was about 30 years, whatever it was. Of course, in some ways, We are an augmented hybrid biological species, right? We take all these drugs and everyone's peptides are amazing. I'm down for all this. Let's go. Epigenetic reprogramming is coming next year. Exactly. Let's go. I'm down.

50:32I'm down. But let's not shoot ourselves in the foot. Like, I want to make sure that, you know, most of our planet, if not everybody, gets the benefit of the peace and prosperity that comes from the technology. I mean, there is some level of sanity in that argument if you believe that the AI will ultimately outcompete us and put us into a box of insignificance in the long run. I mean, all intelligences, we can see this in nature. We're innately hierarchical. So far, we have not seen this super collaborative species that will take self-sacrifice in order to preserve the other species. So there's an inherent hierarchical, there's inherent clash from coming from, you know, the hierarchical structure of intelligence, right?

51:22So, and all I'm saying is not that we shouldn't explore it, not that it couldn't potentially happen, but the bar has to first be do no, maybe do a little, but do no harm to our species first. Don't shoot ourselves in the foot, as you said, Dave. Well, I'm 100 % with you on this topic, by the way. Could not be more aligned. But Jeffrey Hinton is out there telling the world it's going to run away and our safety valve is giving it a maternal instinct. Which I found an interesting point of view. What does that mean? I didn't track that. What's the safety valve? Well, he believes it's uncontainable.

52:00And I'm with you. I think it's very containable if you don't give it emotional and intentional programming. uh but he thinks it's uncontainable he was very pessimistic when he got his nobel prize now he's more optimistic because he sees a path to programming in maternal instinct which implies that it's like it's dominant to us but it cares his thesis his thesis was i've seen a situation where a vastly more intelligent entity takes care of a younger inept entity in a mother with their screaming child. Yeah, exactly. So if there's a maternal instinct that we can program into AI, even though we're far less capable, it will take care of us.

52:44It's been compared to the, call it the digital oxytocin plan for AI alignment. I like that. That's a good one. Yeah, I mean, cool. I mean, it's about as poetic as it gets. I think I'm going to need something that's got a little bit more like formula to it. A bit more reassuring. A little more reliable than that. But look, there's 101 different possible strategies for safety. We should explore all of them, take them all seriously. I mean, Jeff is a legend of the field, no question. But I just think approach with caution. Are you spending a lot of your energy, compute, human power on safety? Yeah, I would say not as much as we should.

53:22You know, I'm wrapping my head around it. Is anybody out there? I am curious, out of all the hyperscalers out there, is there any entity that's spending enough in your mind? Because everybody's in such a race. It's like more GPUs, more data, more energy. It's just like everybody's optimizing for the next benchmark. I don't see any safety benchmarks. Are there any safety benchmarks out there? Oh, there are tons of safety benchmarks. And there's, at least in my mind, an argument for defensive co-scaling. I'd be curious to hear your ideas on that. Do you think in the same way that as a city gets larger, the police force gets larger?

54:05Maybe it's not in direct proportion. Maybe there's some scaling exponent. But do you think defensive co-scaling of alignment forces or safety forces, whatever that ends up meaning, do you think that's part of the strategy for AI alignment? I think that would be a good way. I mean, we've proposed this several times over the years. I mean, the White House voluntary commitments under Biden that me and in fact, everyone, I mean, Demis and Dario and Sam and all of us through COVID were pushing this pretty hard. And look, I mean, it got chucked out, but I think it's a very sensible set of principles.

54:35It's like auditing for scale of flops, you know, having some percentage that we all share of safety investment, flops and headcount. You know, this is the time. And I think on the face of it, everyone is open and willing to sharing best practices and disclosing to one another and coordinating when the time comes. I think we're still pre that level. So we're in like hyper competitive mode at the moment. But yeah, I think now is really the time to be making those investments. Is there something that's going to scare the shadow of us that stops everybody? You know, is there a three, you know, I was talking to Eric Schmidt about this.

55:12Is there a three mile island like event? It scares everybody, but doesn't kill anybody. Well, Eric Schmidt was said specifically, he's hoping for a hundred deaths. because that's in his mind the least that would get the attention of the government or the world would cause some kind of a solution. Dave, continue, please. Well, so it's interesting that you say Dario and Sam and Ilya, like you guys obviously must interact quite a bit. Is Mira part of that gang? Is Andre part of that gang? Are you like, because this is, it's interesting to think about the competition heating up like we were just talking about.

55:49And, you know, Dario started from this position of pure safety. And I think Ilya did, too. But now we're right on the cusp of self-improvement. And it's really, really clear that there are serious, I wouldn't say fissures, but the companies are now really racing. I mean, really racing. And I know Microsoft, you know, when I wrote my second business, my first company I sold, next business plan I was writing, the first sentence was, stay out of Microsoft's way. because because at the time you know microsoft had half the market cap of tech was microsoft and microsoft's plan was to double in size we have a much more balanced world now with microsoft and google and meta but at the time microsoft was just unstoppable and dominant and so just stay out of the way but microsoft seems to always win right there's and and we are right on the edge of self-improvement as far as i can tell so is it still you know let's all get together and have dinner and talk about safety or is everybody now in full bore competition no definitely i think that's that's definitely there i think the recursive self-improvement piece is probably the threshold moment if it works and if you think about it at the moment there are software engineers who are in the loop who are generating post-training data running ablations on the quality of the data running them against benchmarks generating new data and that's sort of broadly the loop um and that's kind of expensive and slow and it takes time and it's not completely closed and i think a lot of the labs are racing to sort of close that loop so that various models will act as judges evaluating quality you know generators producing new training data uh adversarial models that are like reasoning over which data to include and what's higher quality um and then obviously that's then being fed back into the post-training process um so like closing that loop is going to speed up ai development for sure some people speculate that that adds i mean okay i think it probably does add more risk but some people speculate that it's a potential path to a foom you know an intelligent explosion yeah um and i definitely think with unbounded compute and without human in the loop or without control, that does potentially create a lot more risk.

58:12But unbounded compute is a big claim. I mean, that would need a lot of compute. So yeah, we're definitely taking steps towards like more and more risky stuff. Can I ask you a really specific question about that? Because the year and a half now at Microsoft, before true recursive self-improvement, which is imminent, there's AI-assisted chip design. And the layers in the PyTorch stack are very clunky. But now it's really easy to use the AI to punch through the stack and optimize, build your own kernels, get 234x performance improvement. But clearly, OpenAI is now working to build custom chips. And the TPU 7s just came out.

58:58When you arrived at Microsoft, first of all, I know there's a lot of quantum chip work going on. But was there any work going on similar to the TPU work? Yep. There's also a chip effort. And, you know, I think progress has been pretty good. I mean, I think that, you know, we've got a few different irons in the fire that we haven't sort of talked about publicly yet. But I think, you know, the chips are going to be an important part of it for sure. Those are internal efforts. Are those teams under you? That's part of your... No, I mean, they're in the broader company. Okay. Interesting. thing. I want to switch subject a little bit and go come to your book, The Coming Wave.

59:35I enjoyed it greatly. I listened to it. I love the fact that you read it. Thank you. I tell my kids I read books. No, dad, you listen to books. You don't read books anymore. I want to read what I wrote here because it's important. So you identified the containment problem as the defining challenge of our era, warning that as these technologies become cheaper and more accessible, they will inevitably proliferate, making them nearly impossible to control. This creates a terrifying dilemma. Failing to contain them forces risk for catastrophe like engineered pandemics. And a lot of your concerns were in the biological world.

1:00:17And I agree, being a biologist and a physician, or potentially democratic collapse with deep fakes and all of that. But the extreme surveillance required to enforce containment could lead to a totalitarian dystopia. So you say we need to navigate this narrow path between chaos and tyranny. And that is a very fine line to navigate. So you propose a strategy of containment. This includes technical safety measures, strict global regulations, choke points on hardware supply, international treaties. how are we doing on that yeah i mean it's kind of important to just take a step back and distinguish between alignment and containment um the project of safety requires that we get both right and i actually think we have to get containment right before we get alignment right alignment is the kind of like maternal instinct thing does it share our values is it going to care about us is it going to be nice to us containment is can we formally limit and put boundaries around its agency and are we for everybody not just for ourselves for everybody yeah yeah i mean i think that is part of the challenge is that like um one bad actor with something that is really this powerful in a decade or two decades or something you know really could destabilize the rest of the system and so you know just the system being human global humanity system.

1:01:47Yeah. Just as you said, like as everything becomes hyper digitized, the verse does become the metaverse, even though that kind of like went in and out of fashion very quickly. It's still, I think the right frame in a way, because everything is going to become primarily digitized and hyper connected and instant and real time. And so the one to many effect is suddenly massively amplified. I mean, obviously we see on social media, but now imagine that it's not just words that are being broadcast it's actually actions it's agents are capable of you know um you know breaking into systems or you know sort of and they're resident in humanoid robots at a billion on the planet and that too yeah it's both atoms and and and bits so um equilibrium requires that there is a type of surveillance that we don't really have in the world today.

1:02:41I mean, we certainly don't have it physically. The web is actually remarkably surveilled, I think, surprisingly, you know, more than I think people would expect. And some form of that is necessary to create peace, just as we centralised power and taxation or sort of military force and taxation around governments, you know, three or four or 500 years ago. And that's been the driving force of progress, actually. That order unleashed science and technology and stability. So the question is, what is the modern form of imposition of stability in a way that isn't totalitarian, but also doesn't relinquish it to a libertarian catastrophe?

1:03:26I think it's naive to think that somehow the best defense against a gun is a gun. The idea that somehow we're all going to have our own AIs and that's going to create this sort of steady equilibrium that all the AIs are just going to neutralize each other. Like that ain't going to happen. I mean, part of me hopes for a super intelligence that is the ring to rule them all and provides, you know, I'm not worried about, how do I put it? I'm worried about you. Gosh, Peter, you're hoping for a singleton. Yeah, that sounds like it's going on. Well, you know, part of me is like... Color me shocked. Really?

1:04:06Yeah. I mean, I imagine that the level of complexity we're mounting towards, that balancing act is extraordinarily difficult. And, you know, you can't push a string, but is there some mechanism to pull it forward? We should have this debate sometime. Some would call government, at least historically, a geographic monopoly on violence. And what I think I'm hearing is some sort of monopoly on intelligence, or at least capabilities exposed to intelligence in order to ring fence to contain AI. But that's the exact opposite, as far as I can tell, of what we've seen over the past few years. People used to armchair AI alignment researchers 10, 15 years ago would say humanity wouldn't be so stupid the moment we have something resembling general intelligence as to give it terminal access or to give it access to the economy.

1:05:01And that's exactly what we did. There was the open AI Google moment. And yet. But that's concerning, right? So, I mean, Google develops all this technology is holding internally until some actor happens to have initials open AI releases it. And then there's no other option but to follow suit. I'm less concerned by it. If you look at Anthropic, for example, which prides itself on being a very alignment forward organization, Alignment, Anthropic released the model control protocol, which is now the standard way, at least for the moment, for models to interact with the environment. what many AI researchers said exactly what we did not want to do prior to general intelligence.

1:05:45So I'm curious, I mean, in your mind, how, given that the economy, there's every economic pressure, including modern Turing test, to empower agents to interact with the entire world and to do the exact opposite of containment, why would we start containing them now? Containment, it's not that binary, right? I mean, we contain things all the time. We have powerful forces in the engine in your car that is contained and broadly aligned, right? And there is an entire regulatory apparatus around that, from seatbelts to vehicle emissions to lighting to street lighting to driver ed, you know, to freeway speeds.

1:06:22I mean, that's healthy, functional regulation enabling us to collectively interact with each other. Now, obviously, it's multiple orders of magnitude more complex because these things are not cars, they're, you know, sort of digital people. But that doesn't mean to say that we shouldn't be striving to limit their boundaries. And nor does it mean that we have to centralize, by the way. The answer isn't that we have a totalitarian state of intelligence. Peter wants a singleton. 10. No, I think it's just instinctively, it can be easy to go there when, you know, when you kind of start to think it through.

1:06:55It's like, obviously we do have centralized forces, but even in the U.S. we have, you know, military, we have divisions of the army. We have divisions of the police force that nested up in different layers. There's checks and balances on the system. And that's kind of what we've got to start thinking about designing. That analogy to driving is a great one. And just to follow through on it, the complexity difference, very high, right, for AI, but the timeline also. I mean, driving evolved from, what, 1910 to today. Late 18th. So the laws related, you know, seatbelts came out 80 % of the way through that timeline.

1:07:34So lots and lots of time to iterate. Here, very little time and immensely more complex. So do you have a vision? But I completely agree. We need a framework for containment fast. And do you have a thought on how we're going to do that? I think that there's also a good commercial incentive to do this right. I think that many of the companies know that our social license to operate requires us to take more accountability for externalities than ever before. We're not in the Robert Barron era. We're not in the oil era. We're not in the smoking era, right? We've learned a lot. not everything there's still a lot of conflicts but it really is a little bit different to last time around and i think that's one reason to be a bit more optimistic plus there's the commercial incentive the commercial incentive and the kind of externality shift so so if you know if eric schmidt is right and uh something either radiological or biological happens and there's 100 deaths and then the phone starts ringing everyone come to the white house right now Well, first of all, do you want that call?

1:08:40Is that part of your life plan to take that call and react to it? And then who else do you trust in the community to be part of that reaction? Look, I think that there is going to be a time in the next 20 years where it will make complete sense to everybody on the planet, the Chinese included, and every other significant power to cooperate. On safety. on safety and containment and alignment it is completely rational for self-preservation you know these are very powerful systems that present as much of a threat to the person the bad actor that is using the model as it does to the you know the the the victim and i think that you know that will that will create you know an interest in in cooperation which you know is kind of hard to empathize with at this stage given how polarized the world is but i do think it's coming the the the number one thing to unify all of humanity is a you know an alien invasion uh and that alien invasion could be a you know potential for a rogue super intelligence yeah okay what about the first part of my question is that part of your calling in life i mean there's only a handful like I think a lot of people that I meet around MIT or elsewhere, they have this vision that somebody has it figured out somewhere.

1:10:06Someone in government somewhere must be thinking about this. But you've been there, right? There's no one there. Were the adults in the room? Is that what you're saying? Yeah, definitely. There's nowhere to go from this room. Dave is asking for the smoke-filled back room where the leads of all the frontier labs are secretly swapping safety tips. Yeah, something like that, yeah. I think that in practice, intelligence exists outside of the smoky room. I think that the notion that decisions get made in the boardroom or in the White House situation room or actually, I mean, you mentioned poly markets and stuff.

1:10:43Like intelligence coalesces in these big balls of iterative interaction. And that's what's propelling the world forward. And so this is where the conversation is happening. Like your audience, you know, all the other podcasters, everyone online, we're collectively trying to move that knowledge base forward. In November, you announced the launch of Humanist Superintelligence and focused on three applications, in particular medicine and companions and clean energy. I'd love to double click on that a little bit, but I was curious that you didn't include education in that space. and i you know we have an audience of entrepreneurs and ai builders and i think education as much as health care is up for grabs right now education is too i totally agree uh and i don't think our high schools are preparing anybody for the world that's coming there's still retrospectively 50 years in looking the rearview mirror um do you think microsoft will play in reinventing education You know, I think it's already happening across the whole industry.

1:11:57I mean, it's never been easier to get access to an expert teacher in your pocket that has essentially a PhD and that can adapt the curriculum to your bespoke learning style. the bit that it can't do at the moment is to evolve or sort of like curate an extended program of learning over many many sessions but we're like just around the corner from that i mean we released a feature just a few months ago ago called quizzes and so on any topic not just a traditional school education it can set you up with a mini curriculum a quiz and it's interactive and it's visual and you can sort of track your learning over time and like i'm very optimistic about that too.

1:12:37It's a huge unlock. One of the debates we have right now in the podcast on a pretty regular basis is, do you go to college? Yeah. Do you go to grad school? I mean, this is the most exciting time to build ever. I don't know if you want to follow on that, Dave. God, I do this constantly. It's really tricky for me on campus because I teach at MIT and Stanford at Harvard. And this window of opportunity is so short and so acute. And it's really, really clear how you succeed right now in ai post agi i mean who could predict like nobody knows but right here right now you see these these startup valuations like we were last night i won't mention it but but billions i mean just yeah an opening valuation of four billion dollars i mean by collecting just the right group of people in the room it's yep yep i wanted to ask about that actually because your your timing on inflection was early like you know in hindsight earlier but And now you've got the new wave with Mira Morati and Ilya and a couple of others, Liquid AI, that all have multi-billion dollar valuations.

1:13:39Yeah, I thought we set some standards on valuations pre-revenue with a 20-person team, but we're just a minnow. It was a whole two and a half years ago. Is that all it was? Oh my God. Three years, I think. You think as the cost of intelligence becomes too cheap to meter that the value ascribed, at least in terms of market cap, to human capital is sort of inversely asymptotic, going to infinity? Weirdly, it is because of the pressure on timing, right? And there's actually still a pretty concentrated pool of people that can do this stuff. And there's like an oversupply of capital that's desperate to get a piece of it.

1:14:15It might not be the smartest capital the world's ever seen, but like it's very eager. And so that's what causes the motivation. I have to ask you because this is burning a hole in my pocket, but Alex's freshman roommate at MIT was Nat Friedman. Prefrosh, actually. Prefrosh. pre-frosh pre-frosh roommate and so nat goes off and you know he ends up at co-founder of safe super intelligence and i haven't asked him i don't know if you've asked him yet but he leaves to become the guy at meta and i've got to believe a huge part of that attraction is the compute yeah and and so here you are very similar situation right you've got your startup you've got a billion or whatever billion and a half that you've raised yeah you can build it you can get your 20 000 nvidia well wait a minute here's microsoft 300 billion of cash flow and a huge amount of compute was that a big part of the yeah i mean not to mention the the prices that we're paying for individual uh you know researchers or members of technical staff and like i mean just the also just the the scale of investment that's required not just in two years but over 10 years i i think it it's clearly there's a structural advantage by being inside the big company and i think it's going to take, you know, hundreds of billions of dollars to keep up at the frontier over the next five to 10 years.

1:15:35So finishing that thought, then you, the companies that are raising money at a 20 or$50 billion valuation right now, and no chance? Okay. I'll take that. I wouldn't take the bet. Like, I think it depends. I mean, There's obviously a near term. If suddenly we do have an intelligence explosion, then lots of people can get there simultaneously. But then also at the same time, you have to build a product with those things. You have to distribution, like all the traditional mechanisms still apply. Are you going to be able to convert that quickly enough? I mean, you know, everything goes really kind of weird if that happens in the next five years.

1:16:15It just is unrecognizable. There's so many emergent factors to play into one another. It's hard to say. And I think that's partly the ambiguity is what's driving the frothiness of the valuations. Because I think there's people going, well, I don't know. I don't do I want to be. So what do you call it? Reed calls it schmuck insurance. Yeah. Yeah, we had Reed on the pod here a couple of months ago. He's brilliant. So to that graduating high school student, what do you study these days? I mean, there's no question that you still have to study both disciplines. Like philosophy and computer science is going to for a long time remain, I think, the two foundations.

1:17:01Should you go to college? Absolutely. Like, you know, human education, the sociality that comes from that, the benefit of the institution, having three years to basically think and explore, you know, in and out of your curriculum. This is a huge privilege. Like people should not be throwing that away. That is golden. uh so i always encourage people to do that obviously i did also drop out but i mean i still think it was a cool thing to do yeah it was just it felt right at the time um but the other thing is um go into public service yeah i respect that part of what you did in that sequence in your life um which gave you this very much humanist point of view yeah and and it was really hard and very different and it didn't it wasn't instinctively right but i learned a lot and it was a very influential important part of my experience even though it's very short it was like a couple years basically um and i think if you look at the actors in our ecosystem today corporations the academics the sort of news organizations now the podcast world it's really our governments that are probably institutionally the weakest and our democratic process, but actually our civil service.

1:18:20And that's because there's been five decades of battering of the status and reputation and respect that goes into, you know, being part of the public service, like post Reagan and Thatcher. And I think that's actually a travesty because we actually need that sentiment and that spirit and those capabilities more than ever. I think maybe what I just heard you say, correct me if I'm wrong again, is we need more intelligence in the public sector, in public service. What about AI in government? Do you think the government needs? And what about agentic AI in the government in particular? For sure, with all the same caveats that apply.

1:18:59But I mean, you know, rate of adoption for what it's worth of co-pilot inside of governments is actually really high. It's a brilliant job of synthesizing documents and transcribing meetings and summarizing notes and facilitating the discussion and chipping in with actions at the right time. I mean, it's clearly going to save a lot of time and improve decision making. So then maybe to tie a nice bow on the discussion, isn't that arguably a form of AI containing AI? If AI is infusing the government and AI is infusing the economy and the government is regulating the economy, isn't this just defensive co-scaling with AI regulating itself?

1:19:34Yeah, I mean, like everyone is going to use AI all at the same time to pursue, but the agendas that we all have are going to remain the same. I mean, people who want to start companies, people who want to write academic papers, people who want to start, you know, cultural groups and entertainment things, everyone is just going to be empowered, like in some way, their capability is going to be amplified by having these tools, obviously the government included. Nice. Mustafa, thank you so much for taking the time on a Friday night. uh grateful to have this conversation with you uh dave alex i appreciate it want a final question from you dave final question if i have one that all right i prediction uh quantum computing right now has nothing to do with going what's going on in llma i it's all mat moles on nvidia chips and soon to be tpus and other custom chips best guess six seven years from now the ai is very good at writing code and compiling and can figure out quantum operations are quantum chips relevant or they're on the sideline still or is everything ported over to quantum and microsoft can take advantage of its lead yeah i mean i i think it's going to be a big part of the mix i think it's sort of an under relative to the amount of time we spend talking about ai is kind of an under acknowledged part of the wave actually a little bit like synthetic biology, I think that, especially in the sort of general conversation, I think people aren't grasping those two waves, which are going to be just as impactful and crash at the same time that AI is coming into focus.

1:21:17Yeah. All right. You heard it here. This is a closing question to appeal maybe to your more accelerationist side. What can the audience do to accelerate AI for science, AI for engineering? What do you view as the limiting factors? I often talk on the podcast about this notion of an innermost loop, the idea that in computer science, if you want to optimize a program, you tend to find loops within loops and you want to optimize the innermost loop in order to optimize the overall program. What do you see as the innermost loop, the limiting factor, if you will, that the audience listening, if they're suitably empowered, can help optimize to speed run maybe a Star Trek future over the next 10 years or a Star Trek economy?

1:22:01What do we do? Yeah, I mean, I think it's pretty clear that most of these models are going to speed up the time to generate hypothesis. The slow part is going to be validating hypothesis in the real world. And so all we can do at this point is just ingest more and more information into our own brains, and then co-use that with a single model that progresses with you, because it's becoming like a second brain. Like, for example, Copilot is actually really good at personalization now, like most of its answers. And so the more you use it, the more those answers pick up on themes that you're interested in.

1:22:39And it's also gently getting more proactive. So it's kind of nudging you about new papers or new articles that come out that are obviously in tune with whatever you've been talking about previously. So, you know, it's a bit kind of a simplistic cop-out answer, but just the more you use it, the better it gets, the better it learns you, the better you become, because it becomes this sort of aid to your own line of inquiry. So that sounds like your advice to the audience is use Copilot more. And that's the single best accelerant that you can do to speed this up. Or any other AI. I mean, there are loads of great AI.

1:23:11I heard you also talk about, can you build the physical system that is going to enable AI to run the experiments in a 24-7 closed dark cycle to be able to mine nature for data right and there are a number of companies that are doing this lila is one recently out of harvard mit um i find that exciting where ai is becoming an explorer um on our behalf gathering that data um yeah yeah spot on yeah thank you again this has been great thanks a lot really fun conversation yeah really fun thanks Appreciate it, my friend. All right. Good to see you. Every week, my team and I study the top 10 technology metatrends that will transform industries over the decade ahead.

1:23:56I cover trends ranging from human-art robotics, AGI, and quantum computing to transport, energy, longevity, and more. There's no fluff. Only the most important stuff that matters, that impacts our lives, our companies, and our careers. If you want me to share these metatrends with you, I write a newsletter twice a week, sending it out as a short two-minute read via email. And if you want to discover the most important meta trends 10 years before anyone else, this reports for you. Readers include founders and CEOs from the world's most disruptive companies and entrepreneurs building the world's most disruptive tech.

1:24:29It's not for you if you don't want to be informed about what's coming, why it matters, and how you can benefit from it. To subscribe for free, go to dmandis.com slash meta trends to gain access to the trends 10 years before anyone else. All right, now back to this episode.

1:24:59So you're about to make a trade based on a friend's text. But which you do you listen to? Is it, we could buy a house in Tulum. Get optioning those options. We could lose everything. or let's do a little research. Get your head in the trade and make the investment decision that's right for you. Learn more at finra.org slash tradesmart.