Dwarkesh and Noah Smith on AGI and the Economy

20:39I think the economy will be incredibly different from what it was in the past. I'm not saying this is like the modal world. There's a couple of reasons why this might not end up happening. One is even if your labor is not worth that much, the property you own is potentially worth a lot, right? that if you own that S &P 500 and there's been explosive growth, you're like a multi -multimillionaire or the land you have is like worth a lot. AI can make such a good use of that land to build the space probes, assuming that our system of property rights continues into this regime. And second, so I mean, in many cases it's hard to describe how much economic growth there has been over very long periods of time.

21:15For example, if you're comparing the basket of goods that we can produce as an economy today versus like 500 years ago. It's not clear how you compare, we have antibiotics today. I wouldn't wanna go back 500 years for any amount of money because they don't have antibiotics and I might die and it'll just suck. So there's actually like no amount of money to live in 1500 that will rather have than live today. And so if we have those quality of goods for normal people, just like you can live forever, you have like euphoria, drugs, whatever. These are things we can imagine now, hopefully you'll be even more compelling than that.

21:45Then it's easy to imagine like, okay, it makes sense why this stuff is worth way more than the stuff that the world economy can pursue for you and for normal people today. And so I guess I'm just thinking about, this is a thing that economy has really struggled with in the early 20th century. It's this idea that we had this capacity to expand production, expand production, expand production. And then the thing is that companies competed their profits to zero and the profits crashed and nobody wanted to expand production anymore because they weren't making any profit. We're seeing this happen again in China right now with overproduction.

22:10We're seeing BYD having to take loans from its suppliers just to stay financially float even though it's the best car company in the world because the Chinese government has paid a million other car companies to compete with BYD. And so you overproduce, so you have this overproduction. So the solution was to expand consumption. This is the solution people are recommending for China now to expand consumption so that you can reflow the profit margins of all these companies and have them continuous companies. And so the idea is if AI is producing all this stuff, but it's overproducing services, it's overproducing whatever AI can produce.

22:39And the profits from this go negative, that makes the GDP contribution go to zero and basically open AI and Anthropic and XAI and whatever will just be sitting there saying, why am I doing this again? Why am I, no one's buying this shit. And so at that point, it seems like there will be corporate pressure on the government to do something to redistribute purchasing power so that they don't compete their profits to negative. And so they have some reason to create more economic activities so they can take a slice of it, which is essentially what happened in the early 20th century. Yeah, I disagree with this.

23:10I think, I want that saying this will happen. I'm saying like, that would be the analogous thing. I would prefer to be the case that even as a libertarian, I would prefer for significant amounts of redistribution in this world, because the libertarian argument doesn't make sense if there's no way you could physically pick yourself up by the bootstraps. Like your labor is not worth anything, or your labor is worth less than subsistence calories or whatever, which is a more relevant thing. But I don't think this is analogous to the situation in China. I think what's happening in China is more due to the fact that you have the system of financial oppression, which redistributes money and also currency manipulation, which basically redistributes ordinary people's money to basically producing one EV maker in every single province.

23:47So it is the market distortion that the government is creating that causes this overproduction. We can go into what the analogous thing in the AI cash looks like, but I think if there isn't some market distortion, I just think people will use AI where it has the highest rate of return. If it's not space colonization, there will be longevity, drugs, or whatever. I'm just asking why would I invest all this money into AI producing stuff? Why would I just invest the massive hundreds and billions of trillions and whatever of dollars into producing stuff for people who are all gonna be out of a job and won't be able to buy this.

24:15But again, I don't think you'll be producing it for them. I think you'll be producing it for whoever does have... There's stuff in the world, somebody will have stuff. Maybe it's the A .I .s, maybe it's Sam Altman. You're producing it for whoever has the capability to buy your stuff. And will they want A .I. And I'm just saying, A .I. can do so many things. Least of which is like colonizing the galaxy. People are willing to pay a lot of stuff. Right, I'm just trying to... It's trying to... I'm just trying to get this straight on my head of what this economy looks like. And I'm seeing a picture of the trillions of dollars those needed to build out all these data centers will be done not for profit, not to make money from a consumer economy for the creators of the AI, but to satisfy the whims of a few robot lords to colonize the galaxy.

24:50I think you're making two different points and they're getting wrapped into a one. I'm saying, yes, important word. So there's one about, do you expect to do the case that the robot overlord world happens? And I'm saying, no, actually, even without redistrib - First of all, I expect redistributions to happen. I hope it happens, but even if it doesn't, And I don't think it'll happen because people like corporations want to be registered to an app. And I think it'll be good to happen for independent reasons. But I don't buy this argument that the corporations will be like, we need somebody to buy our AI.

25:18Therefore, we need to get the money to the organ to consumer. You believe broad -based asset ownership will create a whole lot of broad -based consumer demand even in the absence of labor income. Honestly, I don't have like a super strong opinion, but I think that's like plausible. But independent of that, I'm like, okay, even if that demand doesn't exist. Just like the things you can do with a new frontier of technology, As long as one person wants it, there's so much room to do things. Space colonization is an obvious example. They like - That's not money. Right. There's obvious demand for the things that you would be able to produce, right?

25:47Like one of the things that you can produce is colonize the galaxy. Right, exactly. So, but the question is, like, I can see a paperclip maximizing a autonomous intelligence is colonizing the galaxy. But in terms of - That's a lot of growth. That is. But in terms of - And so, by the way, I would like to say that I am a paperclip maximizer. I am the real paperclip maximizer. I want to maximize rabbits. in the gaps. I want to turn the entire galaxy into Fluffy Rabbit. That's my goal. And so my goal with AGI is to enlist the AGI's to helping this goal. But then to align them towards rabbit. Anyway, get this down in front of the opening.

26:18I know. I mean, like the social welfare function is Fluffy -ness. But I guess my point here is, as long as AI still doesn't have property rights and it's humans making all the economic decisions be it Sam Altman and Elon Musk or you know you and me, Then at that point, that really matters for what gets done because if we're talking about the money needed to build all these massive data centers, which currently it's a lot of money. It's a ton of money required to build these data centers. And that money need will not go away. We can't just say, oh, cost goes to zero because we can say unit cost goes to zero, but total cost doesn't go to zero.

26:54Nor has it. It has increased. The total spend on data centers has increased. And I think everyone expects it to increase for the fiscal future. The question is, is that money being spent because AI companies expect to read benefits from consumers like you and me? Or to what extent is it that? And to what extent is it, Sam Altman feels like doing some crazy stuff and Sam Altman's just godlike richer than everybody else. And so Sam Altman is actually consuming when he builds those data centers. He is building those data centers so that he can indulge his godlike limbs. I think that more plausible than either a single godlike person is able to direct the whole economy or Like there's this broad -based consumer every person.

27:36I have these are extrins. Yes. I think more plausible is like AIs will be integrated through all the firms in the economy a firm can have property firms will be like largely run by AIs even though there's nominally a human border directors and it might not be nominal Right like maybe the AIs are aligned and like genuinely give the border directors an accurate summary of what's happening but like day -to -day they're being run by AI's. And firms can have property rights, firms can demand things. So it's all you have as a board of directors in AI. Yeah. Okay. I mean, in the ideal world. Okay, so then what we're basically looking at is the labor share of income goes to zero, or something approaching that.

28:12Depends on how to find the AI. Right, the annual income. And capital income is distributed high -line evenly. It's more distributed much more evenly than labor income, but it's still distributed reasonably broadly. Like, I have capital income. You have capital income. So at that point, we have just an extremely unequal society where owners get everything and then workers get nothing. And so we have to figure out what to do about that. Yeah, 100%. Piccadie is killing himself somewhere. Yeah. Piccadie's been wrong about everything. Yeah. So let's hope he's wrong again. I mean, he'd be happy. He'd be like, see, I was right because we're an economist being right is the most important thing.

28:42Exactly. I mean, the hopeful case here is the way our society currently treats retirees and old people who are not generating any economic value anymore. And if you just look at the percent of your paycheck that's basically being transferred to old people. It's like, I don't know, 25 % or something. And you're willing to do this because they have a lot of political power, they've used that political power in order to lock in these advantages. They're not like, so we're robbing you, we're like, I'm gonna go to Costa Rica instead. You're like, okay, I'd pay this money, I'd pay this concession, I'll do it.

29:14And hopefully humans can be in a similar position to this nasi -bi -i economy that old people today have in today's economy. All right. What do humans do? Let's say they get some money, They have enough to live. How do they spend their time? Is it art? Religion, poetry, drugs, it's the final job. Yeah, we're out of the end of the career here. Or is it real the last man of history? Wait, so here's an idea. How about sovereign will fund? Okay, so sovereign will fund we tax Sam Altman and Elon Musk. We use Sam as a metaphor here. He's a threat of the firm. We tax him. We tax Mark. And so then we use their money.

29:49Only the friends of the show will be touched. We use that money to buy shares in the things that those people have. So they get their money back because we're buying the shares back from them. Okay, so it's okay. And then we hire them. Because then what we do is we hire a number of firms, including A16z and pay them two and 20 or whatever, to manage the investment of AI stuff on behalf of the humans, but then the humans become broad -based sort of index fund shareholders or shareholders and whatever. You guys choose to invest in and then you take a cut. and this could be the future economy. This is what my PhD advisor, Miles Kimbles, has suggested this is what the socialist Matt Brooding has suggested, and this is what Alaska actually does with oil.

30:29Capitalist to like it, socialist like it, Alaska likes it. I think sovereign law funds generally have a badge fact record, there's some exceptions that have managed to use their wealth while like Norway or Alaska, but there's just like these political economy problems that come up when there's this tight connection between the investment, which should theoretically be just highest rate return and politicians. So I don't have like, have a strong alternative. Ideally, you just let the market decide how the investment should happen, and then you can just take a tax, but then exactly where does that tax happen?

30:57I haven't thought it through, but. Are you dubious of this? Yeah, I wouldn't want the government influencing where that investment happens, but I want the government taking a significant share of the returns of that investment. Yeah, are you dubious of the trope that labor provides meaning? And if people don't have a clear sense for labor, then it will be very difficult for them to obtain alternative sources of meaning, or is that kind of a capitalist sort of stroked that isn't a so -it -cher? My suspicion is that humans have just adapted to so much. Like, I have a cultural revolution, industrial revolution, the growth of states, like once in a while, like a communist or a fascist or a gibberish team will come around or something.

31:34Like, the idea that being free and having millions of dollars is the thing that finally gets us. Yeah. I'm just suspicious of. But by the way, do we not disagree about the thing? I'm saying, once we get AGI, humans will not have high -paying jobs. Do we disagree about this? I think humans may have high -paying jobs because of comparative advantage. The key here is if there's some AI -specific resource constraint that doesn't apply to humans, then comparative advantage ball takes over and then humans get high -paying jobs even though AI would be better at any specific thing than humans. Because there's some sort of aggregate constraint.

32:09I mean, the example I always use, of course, is Mark Andreessen, who is the fastest typist I have ever seen in my life and yet does not do his own typing. And so because there's a Mark Andreessen specific aggregate constraint on Mark Andreessen's, there is only one of him who so he hasn't taken all the secretary's typist jobs, but because he has better things to do. And so if there's some sort of AI -specific research and strength that hits, then humans could have. Now, I'm not saying there will be, and I'm not saying there won't be. I'm saying, I don't know if there is. Yeah. The reason I find that implausible is that I think that will be true in the short term, because right now there's 10 million Asian 100 equivalents in the world.

32:44And a couple years there might be 100 million, like, Asian 100 has the same amount of flux as a human brain. So theoretically, they're like as good as a brain if you have the right algorithm. So there's like a lower population of AIs, even if you got AGI right now than humans. But the key difference is that in the long run, you can just keep increasing this supply of compute or robots. And so if it is the case, so if an 800 costs a couple thousand dollars a year to run, but the value of an extra year of intellectual work is still like a hundred thousand dollars. So you're like, look, we've saturated all the 800s, and we're gonna pay a human hundred thousand dollars because there's still so much intellectual work to do.

33:20In that world, the return on buying another 800, like an 800 cost for $40 ,000, just like in a year that 800 will pay you over 200 % return, right? So you'll just keep expanding that supply of compute until basically the 800 plus depreciation plus running cost is the same as an extra year of labor. And in that world, that's like much lower than human subsistence. So compared to advantages, totally consistent with human wages being below subsistence. It is, but that comes from the common resource consumption. So it's basically all of the land and energy that could be used to feed and glow than shelter humans gets appropriated by H -100s, then that is the case.

34:01However, if you pass a law that says this land is reserved for growing human food, if we actually were to just pass a simple law saying that you have to use these resources, these resources reserved for human. But at that point, the competitive advantage that the competitive advantage is that. At that point, human labor has nothing to do with this. The only reason this system works is that you are basically transferring resources. You've come up with a sort of like intricate way to transfer resources to humans. It's just like, this resource is for you, you have this land and therefore you can survive.

34:33And this is just like an inefficient way to allocate resources to humans. It's true that it is an inefficient way. I think people will like cure this argument of a competitive identity like, oh, there's some intrinsic reason that you take you be honest with. We take you be honest with you. Yeah, okay. Yeah, I mean, sure. But then again, we typically do not see the first best, most efficient political solution implemented for things like redistribution. In real world, redistribution happens be a thing like the minimum wage or like letting the AMA decide how many doctors this can be. So redistribution in the real world is not always the most efficient thing.

35:05So I'm just saying that like comparative advantage, if you're talking about will humans actually continue to get high paid work, yes or no, it depends on political decisions that may, it depends on physical constraints that will happen. But the high paid jobs are literally because like, you have said that there must be high paying jobs politically. I understand. in this case you said it in an indirect way, but you still said it. You're absolutely right. Yeah, you're not wrong. I guess it's like incredibly different from what somebody might assume. It has almost nothing to do with the comparative management.

35:35But that's true, but that's true of a lot of jobs that exist now. With a lot of jobs that exist now, I'm not sure what university professors, because there's a lot of those jobs, or like credit rating agencies, or there's a lot of things where probably we could ring out some significant TFP growth more or less by eliminating those things, but we don't because our politics is a clue, Jocke, or say, I think this is one of Tyler's points. I mean, I do think it's important to point out in advance, like basically it would be better if we just bit to the bullet about AGI so that instead of doing redistribution by expanding Medicaid and then Medicaid can procure all the amazing services that AI will create, it would be better if we just said, look, this is coming and I'm not saying we should do a UBI today, But like if all human wages go below subsistence, then the only way to deal with that is through some kind of UBI.

36:25Rather than if you happen to sue OpenAI, you get a trillion dollar settlement, otherwise you're screwed. Right. Some people said the bear case for UBI was something around COVID as an example. You gave people a bunch of money and would they go do, go ride to the streets? I'm teasing. But are people gonna use that money in effectively? I mean, that was literally what happened. Yeah, so is UBI the form that you would think that What is the most effective method? The reason I feel our UBI is this thing where in a future world with explosive growth, we're going to see so many new kinds of goods and services that will be possible that are not available today.

37:00And so distributing just like a basket of goods is just inferior to saying, oh, if we solve aging, here's some fraction of GDP, go spend your tens of millions partly on buying this aging cure, whatever this new thing that AI enables, rather than here's a food stamps equivalent of the AGI world that you can have access to. Of course, I mean, this discussion may be academic because I believe that you said that we got phones in the world look the same. I mean, no, it doesn't. Fones have destroyed the human race. Like the fertility crash that's happening all around the world, nobody has replacement level fertility is going far below replacement everywhere because of technology.

37:38And is that the phone or the pill or? Well, no, it's a phone. I mean, we'll know the pill and other things, like women's education, whatever. like lowered fertility, like quite a bit, but some countries were still out replacement level, some were still around replacement level, but the crash we've seen since everybody got phones is epic and is just unbounded. The human race does not have a desire, a collective desire to perpetuate itself. Yes, we're gonna get lonely, but we'll have company through AI and through the internet, social media, until there's just a few of us and we do indolend -windol.

38:10Yeah, I mean, like technology has already destroyed the human race, and basically UBI is just like keeping us around on life support for a little while while that plays out. I do think so far there's been a lot of negative effects from widespread TikTok user or whatever that we're still up to learning about. I am somewhat optimistic that in the long run there's some optimistic vision here that could work just because right now the ratio of like it's impossible for Steven Spielberg to make every single TikTok and direct it in a sort of really compelling way that's like genuine content and not just video games at the bottom and some music video at the top.

38:46In the future, it might genuinely be possible to give every single person their own dedicated Steven Spielberg and create incredibly compelling but long narrative arcs that include other people they know, et cetera. So in the long run, I'm like, maybe this... What's the next step happen? I don't think TikTok is like the best possible medium. No, I also don't think TikTok is unique in destroying human rights. I think that interacting online instead of interacting in person, that's a great thing. That's a great tool. I didn't make your money work out. I agree. We're all making, we're all making the money destroying our species.

39:17You don't think we got isolated to dating apps? No, I'm saying like as long as you can get your why did humans perpetuate the human species, it was not because they wanted to see the human species perpetuated, it was because it's like, ooh, I had sex and there came a baby. And that's done. We've severed that. That is the end. We did not evolve to want our species to continue. Right, but you're saying the reasons why we're not having babies is because we can make friends on the internet, but is it that dating apps have created just much more efficient market and thus there is maybe a pair of but I don't know.

39:43I mean like people having less sex if Elon gets his way everybody will just sit there gooning to some sort of rock and panning and apocalypse seems upon us. He's available right now. What's the website? Oh no.

39:57This podcast got silly. But anyway, I guess the point is that the idea of a humanity that just keeps increasing in numbers and spreading out to the galaxy. I don't see a lot of evidence that is in our future and that we have to go to great lengths to make sure that future is compatible with AGI because I don't think it's happening in any case. AGI or none. By the way, not to cope to our, but in a world where AGI happens, how important is increasing properly? I mean, population has so far been the decisive factor in terms of which countries are powerful. Like the reason China, if the US was not involved, the reason China could take over Taiwan is just that there's 1 .4 billion Chinese people and there's 20 million Taiwanese people.

40:39Now, if in future your population is your effective labor supply is like largely AIs, then this dynamic just means that like your inference capacity is literally your geopolitical power, right? I want to shift to short term a bit. You've had some people in the podcast, you had AI at 2027 folks who believe that AGI's perhaps two years away, I think they updated it to three years away, and then you've also had some folks on who said it's not for 30 something years. Maybe you could steal man, both arguments, and then share where you net it out. Yeah, so two years if I'm stealing manning them is that look, he just looked at the progress over the last few years.

41:13It's reasoning. Air startles, like the thing that makes humans is reasoning. It was not that hard, right? Like, training on math and code problems and having like think for a second, and you get reasoning, like it's crazy. So what is a secret thing that we won't get? Can I ask a stupid question? Why was stuff like O3 type models? Why are those called Reasing Models, but like GPT 40 is not called Reasing? What are they doing different that's reasoning? One, I think it's GPT 3 can technically do a lot of things GPT 4 can, but GPT 4 just does it way more reliably. And I think this is even more true of Reasing Models relative to GPT 40, where 40 can solve math problems.

41:53And in fact, like modern day 40 has been probably trained a lot of math and code, but the original GPT -4 just wasn't trained that much on math and equipment problems. So it didn't have whatever meta circuits there exists where like how do you backtrack? How do you be like wait, but I'm on the wrong track? I gotta go back. I gotta pursue the solution this way. Algarithically, I have a okay idea of what a reasoning model does that the non -reasoning model don't. But in terms of how does that map to a thing that we call reasoning? What is the definition of what it means to reason that these people are using?

42:22The operational definition here. Cause I don't understand that myself. I mean, 4 -0 can't get a gold and IMO. Okay, but I can reason and I can't get a gold and IMO. But I can reason. Yeah, I mean, I can't get a gold either, but I don't think I can reason as well as maclumpia. At least in the relevant domain. I agree that reasoning is not just about mathematics, but this is true of any word you come up with. Like the zebra, what about the thing that like is a mixture of zebra and a giraffe and they have a baby, is that a zebra still? I agree, there's edge cases to everything. But there's a general conceptual category of zebra, and I think there's a general conceptual category of reasoning.

42:53Okay, I'm just wondering what it is. Like when you have a checkout clerk, right? That checkout clerk wouldn't, wouldn't look at an IMO problem back, would. But then like you have a checkout clerk and the checkout clerk you're like, okay, so you put the thing on this shelf and therefore someone has, he looked for it and didn't find it so something else must have happened. But I think a reasoning model. I think a reasoning model will be more reliable and be better at solving that kind of problem than for a. So you were still manning the AI 2027? Yes. So a lot of things we previously thought were hard have just been incredibly easy.

43:24So whatever additional bottlenecks you are anticipating, whether it's this continual learning on the job training thing, whether it's computer use, this is just going to be the kind of thing we're in advance, it's just like how would we solve this? And then deep learning just works so well that we like, I don't know, try to train it to do that and then it'll work. The long -time ones people will say, I don't know, there's a sort of long argument, I don't know how much to bore you with this. But basically, the things we think of as very difficult and requiring intelligence have been some of the things that machines have gone first.

43:52So just adding numbers together, we got in the 40s and 50s. Reasoning might be another one of those things, where we think of it as the apagy of human abilities, but in fact, it's only been recently optimized by evolution over the last few million years, whereas things like just moving about in the world and having common sense and so forth, and having this long -term memory, evolution spend hundreds of millions, if not billions of years optimizing those kinds of things. So those might be much harder to build into these AI models. I mean, the reasoning models still go off in these crazy hallucinations that they'll never admit were wrong And we'll just gaslight you infinitely on some crap that made up like just knowing truth from falsehood Yeah, I've met a couple of humans who don't seem to be able to know truth from falsehood.

44:31They're weird So but oh three sometimes does this I mean, I think it's a question. Do they hallucinate more than the average person? I think no less they get an hallucinate meaning like getting something wrong and then when they push them on it They're like, no, whatever. And eventually they'll like a seed if they're clear they're wrong. I think like, they're actually more reliable than the average human. But so the thing about the average human is you can get the average human to not do that with the right consequences. And maybe AI we haven't found the right reinforcement learning function or whatever to get them to not do.

45:01Okay, now let's get to the view that it's 30 years away. What's that view? Just this thing of reasoning is relatively easy in comparison to forget about robotics, which is just going to be, that was just been billions of years trying to get robotics to work. But there's other things involved with tracking long run state of, you know, a lion can follow up, pray for a month or something, but these models can't do a job for a month, and these kinds of things are actually much more complicated than even reasoning. And where you've netted out is, it's either going to happen in a few years or not for quite some time?

45:33Yeah, basically the progress in AI that we've seen over the last decade has been largely driven by stupendous increases in compute. So the compute used on training a frontier system has grown for actually year for I think like the last decade. And that just over four years has 160 x, right? So that's over the course of a decade that's hundreds of thousands of times more compute. That physically cannot continue if you just like, okay, what would it mean right now we're spending 1 .2 % of GDP or something on data centers? Not all of that is returning of course, but what would it mean to continue this for another decade.

46:08For maybe five more years, you could keep increasing the share of energy that we're spending on training data centers or the fraction of the TSMC's leading edge nodes wafers that we dedicate to making AI chips or even the fraction of GDP that we can dedicate to AI training. But at some point, you can't keep this like 4x trend going a year. And after that point, then it has to just like come from new ideas of like here's a new way to get trained a model. And by the way, when I was writing that comparative advantage post and I was thinking about AI -specific aggregate constraints, resource constraints.

46:39That's what I was thinking about, actually. That that expansion of compute has to solve that. But how much that matters? That's for training and for the labor that will be like the inference will also use the same, that's the same bucket of compute. Is the case that for the amount of compute it costs to train a system, if you've set up a cluster to train a system, you can usually run 100 ,000 copies of that model at typical token speeds on that same cluster. That's still obviously not like billions. But if we've got all this compute to be training these huge systems in the future, it would still allow us to set state and population of hundreds of millions, if not billions of AIs.

47:12At that point, maybe we obviously will still need more AIs. What is a single AI? I mean, when you're talking to cloud, it's a single instance. It's not going to exist. So instances. Yeah. Okay. So what's going to determine whether it's in a few years or... Right now, we're basically riding the wave of this actual compute. That's why is getting better every year mostly. In terms of the contribution of new algorithms, it's a smaller fraction of the progress that's explained by that. So if we've just got this rocket, like how high will it take us? And does it get a space or not? And if it doesn't, then we just have to rely on the algorithmic progress, which has been this slur, yeah.

47:46But you think it might get a space? Yeah, I think there's like a chance that like, oh, continue learning is also like, you know, I had this whole theory about, oh, it's so hard and how do you start it in? And they're like, I fucking trained it to do this. Like, what are we talking about here? That leads into another thing that I've thought about, which is how poor our track record for making predictions about the future of AI has been. The first time you had a hung out, I don't know if you remember this, was with Leopold. Yeah, I remember that. Yeah, it was your old house. And Leopold is just pronouncing a whole bunch of pronouncements from the couch.

48:17And he released this big situational awareness thing. How long ago was that a year and a half? Yeah, yeah. I would say that already most of the things he predicted have been invalidated or made irrelevant in the last year and a half. And especially in terms of all the stuff about competition with China, it turns out filtration was able to get them a whole lot of things that he never predicted. It turns out that so many of the things other than just the idea that, hey, I would keep getting better, which he predicts a lot of people predict. But then I feel like a lot of the specific predictions about US capabilities and Chinese capabilities and what would be the bottlenecks and what would be the things that, you know, here's how we can view China.

48:51There's all been proven wrong since. I think this is actually an interesting trend in the history of science where some of the scientists who are the smartest in thinking about the progression of the atom bomb or progression of physics just had these ideas about the only way we can sustain this is if we have one world government. I'm pretty sure I'm an after world war two. There's no other way we can deal with this new technology. I do think relative to the technological productions, Leo, I think the main way in which we've been wrong is that it didn't take some breaking the servers in order to learn how both three or something works.

49:21It was just public just see you being able to use the model. You can talk to them learn what it knows. Just knowing a reasoning model works and then you can like use it and you see like, oh, what is the latency? Like how fast is the operating tokens? That will teach you like how big is the model? Like you learn a lot just from publicly using a model and knowing anything is possible. He has been right in one big way, which is like he identified three key things that would be required to get us from GPT -4 to BBAGi kind of thing, which was being able to think so test time compute onboarding, which you talk about test time can be in that Yeah, yeah.

49:51One of his three big unhobblings. Then like onboarding in terms of the workplace. And then I think the final one was computer use. Look, one out of three and it was a big deal. So I think you got something to write. It's something's wrong. But yeah. And was your take on the model of automating AI research as the path to HGI? The meter uplift paper contrary to expectations, they found that whenever senior developers working in repositories that they understood well, use AI, they were actually slowed down by 20%. Yeah, I did see that. Yeah, whereas they themselves thought that they were spread up 20%.

50:24And so there's a bunch of, I'm getting things done. This was actually worth the worry about the phones are destroying us. That is this update towards the idea that AI is not on this trend to be this super useful assistant that's helping us already make the sort of process of training AI are much faster. And this will just be the feedback loop and exponential. I have other independent reasons. I'm like, I don't know, I'm like 20 % that we'll have some sort of intelligence explosion. One of the other labeled predictions was nationalization. Is that something you could potentially foresee in the next few years?

50:57I don't think it's politically plausible, especially given this administration. I don't think it's desirable. First, I think it would just drastically slow down the eye progress because look, this is not 1945 America, and also building in an atom bomb is like a way easier project than building AGI. But China's quasi -nationalizing, most of it's in, I mean, China doesn't control BIDs day -to -day decisions about what to build, but then is trying to do this BID does it as does every Chinese. I mean, that's kind of the relationship of American companies have the US government as well. You think so?

51:27I mean, somewhat. Also, the big difference is what do we mean by nationalization? There's one thing which is like there's a party cadre who is in your company. Exactly. There's another which is that each province is like just pouring a bunch of money into building their own competitor to BID in this potentially wasteful way. That like distributed competitive process It seems like the opposite of nationalization to me. Like when people imagine EGI nationalization, I don't think they're saying like, Montana will have their HDI and Wyoming will have their HDI and they'll all compete against each other.

51:56I think they imagine that all the laps will merge, which is actually the opposite of how China does industrial policy. But then you do think that the American government, basically if it says do this, then like XAI and OpenAI will do it. No, actually I think in that way, obviously the Chinese system and the US is no different. Although it has been interesting to see that whenever, I don't know, we've noticed the way that different and loud leaders have changed their tweets in the aftermath of the election. I mean, also, we're bullish up at source. And didn't say I'm up the thing, we're, I think previously he said that AI will take jobs.

52:25How do we deal with this? And then, didn't he recently say something at a panel where I think President Trump is correct that AI will like create jobs or something? Where I don't think in the long run you believe this. But the reason why humans should be excited about even their jobs being taken is just, they'll be so rich that why do they even need it? Yeah. Much richer than in there, I know. Right. module with this redistribution slash not fucking it over with some guild like thing. Yeah. You mentioned the atomic bomb and we also mentioned off camera that you don't think the nuke is a good comparison for what happens.

52:53How does it play out when a lab figures out AGI? What then happens? Is there a huge advantage if one country has it first or if one lab has first today, they dominate? I think it's less like the nuclear bomb where there's a self -contained technology that is so obviously relevant to specifically the psychofensive capability. And you can say, well, there's nuclear power as well. But like, neither of those nuclear power is just like this very self -contained thing. Whereas I think intelligence is much more like the industrial revolution. Well, there's not like this one machine that is the industrial revolution.

53:23It is just this like broader process of growth and automation and so forth. So that means - So Brad belongs, right? And Robert Gordon is wrong. If Robert Gordon said this only, it's four things. It's just four big things. Oh, and Brad belongs. And along is like, no, it's a process of discovering. That's not any right. Interesting. where we're rubs four things again? Oh, I mean electricity. Test time for a few. Not kidding. That's not good. The internal combustion engine, steam power, and then what was the fourth one? Like maybe like plumbing. Right. I think was the fourth one. Yeah, or even in that case, maybe that actually is, maybe that's closer to how I think about it.

53:55But then you need so many complimentary innovations. So internal combustion engines that I didn't invent in the 1870s. Drake finds the oil well in Pennsylvania in the 1850s. Obviously it takes like a bunch of complimentary innovations before, like, these two things can merge before they're just like using the oil for the kerosene to light lamps. Right. But regardless, so if it's this kind of process, it was the case that many countries have achieved industrialization before other countries. And like, China was dismembered and went through a terrible century because the Qing Dynasty wasn't up to date on the industrialization stuff.

54:26And much smaller countries are able to dominate it. But that is not like we developed the atom bomb first and now we have decisive advantage. I think it's because of us. Because if that had been Nazi Germany, the Soviet Union, it would have gone differently. How do you see the US trying to competition playing out in terms of AI? I genuinely don't know. Yeah, I think it's like possible that there could be some positive some, like not like a nuclear weapon, both countries can just adopt AI. And there is this dynamic where if you have higher inference capacity, not only can you deploy AI's faster and you have more economic value that's generated, but you can have a single model learned from the experience of all of its copies and you can have this basically broadly deployed intelligence explosion.

55:09So I think it really matters to get to that discontinuity first. I don't have a sense of at what point if ever is treated like the main geopolitical issue that countries are prioritizing. I also from the misalignment stuff. The main thing I worry about is the AI playing us off each other rather than us playing the AI's off each other. I mean, AI just like telling us all to hate each other, the way like Russian trolls currently tell us all to hate each other. A more so like the way that the East India company was able to play different provinces in India off of each other. And ultimately at some point you realize, okay, like they control India.

55:44And so you could have a scenario like, okay, think about the conquistadors, right? A couple hundred people show up to your border and they take over an empire of 10 million people. And this happened not like once, it happened two to three times. Okay, so why was this possible? Well, it's that the Aztecs, the Incas, weren't communicating with each other. Like, they didn't even know the other empire existed, whereas Cortez learns from the subjectation of Cuba, and then he takes over the Aztecs. Pizarro learns from the subjectation of the Aztecs and takes over the Incas. And so they're able to just learn about, okay, you take them for hostage, and then this is the strategy you employ, et cetera.

56:18It's interesting that Aztecs and Incas never met each other in that worked both times, sort of. Yeah. That's interesting that these totally disconnected and civilizations both had the similar vulnerabilities. Yeah, I mean, it was literally the exact same label. The crucial thing they went wrong is that, at this point in the 1500s, we actually don't have modern gods, we have archibuses, but the main advantage that the Spanish had was they had horses, and then secondly, they had an armor, and it was just incredibly, you'd have thousands of warriors if you were fighting on an open plane, the horses with armor were just like trounds all of them.

56:49Eventually, the Incas had this rebellion and they learned they can like roll rocks down hills and the rebellion was moderately successful, even though it's eventually, we know what happened. You could say that the Spanish on their side had guns, germs, and steel. So how could this have turned out differently if the assets had learned this and then had like told the Incas? I mean, they weren't in contact. Well, if there's some way for them to communicate, like, here's how you take down a horse. I think what I would like to see happen between the US and China, basically, is like the equivalent of some red telephone during the Cold War where you can communicate, look, we noticed this, especially when AI becomes more integrated with the economy and governance, etc.

57:24We noticed this crazy attempt to do some sabotage. Be aware that this is a thing they can do, train against it, etc. AI is trying to trick you into doing this. Watch out. Yeah, exactly. Though to a prior level of trust, I'm not sure it's plausible, but that's the optimal thing that would happen. At the lab level, do you think it's a multipolar or is there consolidation and who's your bet to it? I've been surprised. You would expect, over time, as the cost of competing at the frontier has increased. You would expect there to be fewer players at the frontier. This is what we've seen in the semiconductor companies, right?

57:53That you get some more expense over time. There's now maybe one company that's at the frontier in terms of like global semiconductor manufacturing. We've seen the opposite trend in AI where there's like more competitors today than there were a year ago, even though it's gone more expensive. I don't know where the equilibrium here is because the cost of training these models is still much less than the value they generate. So I think it'll like would still make sense to 10x the amount of investment. Somebody knew to come into this field and 10x the amount of investment. Do you have a take on worthy cool agreements?

58:22Well, I mean, it has to do with entry barriers. Basically, it's all about entry barriers. It's the question of if I just decide to point down this amount of money. So if the only entry barrier is fixed costs, I'd say we have such a good system for like just loaning people money that that's not going to be that big a deal. But if there's entry barriers that have to do with if you make the best AI, it gets even better. So, you know, why enter? That's a big question. I don't actually know the answer question. There's a broad question. What we ask in general is like, what are the network effects here?

58:51And what is the responsibility? And it seems often to be brand. Yeah, I mean, I'm not sure it's a network effect. But brand like OpenAI at ChatGBT is the Kleenex of AI in the Kleenex is actually called a tissue, but we call it a Kleenex because there was a company called Kleenex. Where are you going with this? Everybody in the Bronx doing things? I'm going to go to the Fox. I don't know. Well, what's another example? Xerox. You Xerox this thing. Xerox is just one company that makes a copy here. Right, not even the biggest, but everybody knows that it's your oxygen. Right. And so chat GPT gets massive rents from the fact that everyone just says, Oh, use AR.

59:25What's an AI? Chat GPT. I'll use it. And so like brand is the most important thing. But I think that's mostly due to the fact that this key capability of learning on the job has not been unlocked. And so I don't know what I was saying. That could be a technological network effect that could supersede the brand effect. Right. Yeah. Yeah. And I think that that will have to be unlocked before most of the economic value of these models can be unlocked. And so by the point there is a lot generating hundreds of billions dollars a year, or maybe millions dollars a year, they will have had to come up with this thing, which will be a bigger advantage in my opinion than brand network effects.

1:00:01Is Zuck throwing away money wasting it on hiring all of the time? No, people have been saying like, look, the messaging could have been better or whatever. I mean, I think it's just much better to have worse messaging or something, but the not sleepwalk towards losing. Also, we just think about like, if you pee and employ a hundred million dollars and they're a great year researcher and they make your computer your training or your inference 1 % more efficient. Zucker spending on the order of like 80 billion dollars a year on compute. It's made 1 % more efficient. That's easily worth a hundred million dollars.

1:00:31Like, a hundred million dollars is below the break even point for this extra researcher. So the real question is like, why haven't we hit that break even point yet? And if we as podcasters encourage one researcher to join Meta, I mean, What's the price on that? Yes, she had. And this has been a phenomenal conversation. No, Ed Dorkash. Thank you so much for coming on. It's been great. Awesome. Thanks, Eric. Thanks for listening to the A16Z podcast. If you enjoyed the episode, let us know by leaving a review at ratethispodcast .com slash A16Z. We've got more great conversations coming your way. See you next time.