AGI is Still 30 Years Away — Ege Erdil & Tamay Besiroglu

27:09So these are two, two components of our view. We should just harden people's think and depends a lot on compute scale. So can you put a finer point on what is the kind of thing? What is an example of the kind of task which is very dissimilar from train and classifier or debugger classifier that is relevant to AIR and D? I think it's like examples might be introducing novel, having novel innovations that are very useful for unlocking innovations in the future. So that might be introducing some novel way of thinking about a problem or introducing. So maybe a good example might be in mathematics where we have these reasoning models that are extremely good at solving math problems.

27:56I mean, very short horizon. Sure. Maybe not extremely good, but certainly better than I can and better than maybe most undergrad scan. And so they can do that very well, but they're not very good at coming up with novel conceptual schemes that are useful for making progress in mathematics. So it's able to solve these problems that you can kind of neatly excise out of some very messy contacts and it's able to make a lot of progress there. But within some much messier context, it's kind of not very good at figuring out what directions are especially useful for you to build things or kind of make incremental progress on that enables you to have a big kind of innovation later down the line.

28:44So thinking about both this larger contacts as well as maybe much longer horizon, kind of much fuzzier things that you're optimizing for, I think it's much worse at those types of things. Right, so I think one interesting thing is if you just look at these reasoning models, they know so much, especially the large ones, because I mean, they know in the little terms more than any human does in some sense. And well, we have unlocked these reasoning capabilities on top of that knowledge. And I think that is actually what is enabled in terms of a lot of these problems. But if you actually look at the way they approach problems, they, like the reason what they do looks impressive to us is because we have so much less knowledge.

29:27And the model is approaching problems in a fundamentally different way compared to a human world. A human world have much more limited knowledge and they would usually have to be much more creative in solving problems because they have this lack of knowledge. While the model knows so much, or you'd ask it some obscure math question, where you need like some of those specific theorem from 1850 or something. And then it would just like know that, if it's like a large model. So that makes it difficult to provide very different. And if you look at the way they approach problems, the reasoning models, they are usually not creative.

29:59They are very effectively able to leverage the knowledge they have, which is extremely vast, and that makes them very effective in a bunch of ways. But you might ask the question, has a reasoning model ever come up with a math concept that even seems like slightly interesting to a human mathematician? And I've never seen that. I mean, they've been around for all of six months. But that's a long time. Lots of people have been, I mean, that's a little time like. Like just thinking about it. One mathematician might have been able to like, do a bunch of work over that time. And they have produced orders of magnitude fewer tokens on math.

30:34That's right. And then, like I just want to emphasize it because just think about the sheer scale of knowledge that these models have. Like it's enormous from a human point of view. So it is actually quite remarkable that there is no interesting recombination, no interesting, oh, this thing in this field looks kind of like this thing in this other field. There's no innovation that comes out of that. And it doesn't have to be like a big math concept. It could be just like a small thing that maybe you could add to like a Sunday magazine on math that people used to have. But there's even like an example of that.

31:09I think it's useful for us to explain like a very important framework for our thinking about what AI is good at and what AI is lagging in, which is this idea of kind of more of X paradox that things that seem very hard for humans, AI systems tend to make much faster progress on. Whereas things that look a bunch easier for us, kind of AI systems that totally struggle are often totally incapable of doing that thing. And so this kind of abstract reasoning, playing chess, playing go, playing jeopardy, doing kind of advanced math and solving math problems. There are even stronger examples like multiplying a hundred digital numbers in your head, which is the one that calls solved first out of almost any other problem.

31:56Or like following like very complex sort of symbolic logic arguments, like deducting arguments. People actually struggle with that a lot. Like how do premise is like logically fall from conclusions? Like people have a very hard time with that very easy for formal proof systems. An insight that is related and it's quite important here is that the kind of very, the tasks that a human seems to struggle on and AI systems seem to make much faster progress on are things that kind of emerge fairly recently in evolutionary time. So advanced language use, emerged in humans maybe a hundred thousand years ago and certainly playing chess and go and so on are very recent innovations.

32:38And so evolution has had much less time to optimize for them in part because they're very new, but also in part because when they emerged, there was a lot less pressure because it conferred kind of small fitness gains to humans. And so evolution didn't optimize for these things very strongly. And so it's not surprising that on these specific tasks that humans find very impressive when other humans are able to do it, that AI systems are able to make a lot of fast progress. In humans, these things are often very strongly correlated with other kind of competencies, like being a good, good at just achieving your goals or being a good coder is often very strongly correlated with solving kind of coding problems or being a good engineer is often correlated with solving competitive coding problems.

33:29But in AI systems, the correlation isn't quite as strong. And even within AI systems, it's the case that, you know, the strongest systems on competitive programming are not even the ones that are best at actually helping you code. So like, you know, O3 minis, high seems to be maybe the best at solving competitive code problems, but it isn't the best that actually helping you write code and making most of the enterprise revenue from plays like Cursero or whatever, like, let's just clawed, right? So but an important insight here is that, you know, the things that we find very impressive when humans are able to do it, we should expect that AI systems are able to make a lot more progress on that.

34:11And, but we shouldn't update too strongly about just their general competence or something, because we should recognize that this is a very narrow subset of relevant tasks that humans do in order to be a competent, economically valuable agent. Yeah. First of all, I actually just like really appreciate that there is an AI organization out there where, because there's other people who take the compute perspective seriously, or try to think empirically about scaling laws and data and whatever. And it's striking how often that, like, taking that perspective seriously leads people to just be like, okay, 2027 AGI, which might be correct, but it is like just interesting to get, like, no, we've also looked at the exact same arguments the same papers, the same numbers, and like, we've come to a totally different conclusion.

35:02On all these arguments, I think this is all fascinating. Okay, so I asked Dario this is exact question two years ago when I interviewed him. And what viral did he say AGI in two years? That Dario's always had short timelines. Okay, but the year two years later. Did he say two years? I think he actually did taste two years. Or did he say three years? So we have one more year, one more year. Better work hard. But he's, I mean, I think he's like, he in particular has not been that well -calibrated. He's like, oh, like in 2018, he had like, I remember talking to like a very senior person who's now at Anthropic in 2017.

35:39And then he told various people that they shouldn't do a PhD because by the time they completed it, you know, that's right, that's right. Everyone would be automated. Yeah, yeah. So anyways, I asked him this exact question, right? Because the actual timeline is, which is that if a human knew the amount of things these models know, they would be finding all these different connections. And in fact, we, this is, I was asking Scott about this the other day when I interviewed him, Scott Alexander, and he said, like humans also don't have this kind of logical omniscience. And I'm not saying we're omniscient, but we have examples of humans finding these kinds of connections.

36:09This is not an uncommon thing, right? I think his response to that was that these things are just not trained in order to find these kinds of connections. But if you, like, their view is that it would not take that much extra compute in order to build some oral environment in which they're incentivized to find these connections. Next token prediction just isn't incentivizing them to do this, but the oral required to do this would not be that, or set up some sort of scaffolds. I think actually Google deep mind did do some similar, like scaffold to make new discoveries. And I didn't look into like how impressive the new discovery was, they claim that some new discovery was made by NIL as a result.

36:45On the more of X paradox thing, this is actually a super interesting way to think about AI progress. But I would also say there that if you compare animals to humans, long -term intelligent planning, like an animal is not gonna help you book a flight either. An animal, an animal is not gonna do remote work for you. Or even do the kinds of things. I think what separates humans from other animals is that we can hold long -term plan. We can like come up with a plan and execute on it. Whereas other animals are often had to go by instinct or within the kinds of environments that they have evolutionary knowledge of, rather than like I'm putting the middle of the savannah, I'm putting the middle of the desert or I'm putting the middle of the chundra and I'll learn how to make use of the tools and whatever they're.

37:34I actually think there's like a huge discontinuity between humans and animals and they're really related to surviving different environments, just based on their knowledge. And so it's like a recently optimized thing as well. And then I'd be like, okay, well, AI is what it's like, we got it soon, AI is what optimized it for a fast. Right, so I would say if you're comparing animals to humans, it's kind of a different thing. I think animals, like if you could put the competences that the animals have into AI systems, that might just already get you to like AGI, like already. I think the reason why there is such a big discontinuity between animals and humans is because animals have to rely entirely on natural world data, basically, to train themselves.

38:19Like imagine that the only thing as a human that you saw was like nobody talked to you, you didn't read anything, you just had to learn by experience, maybe to some extent by imitating other people, but not that you have no experience with communication, well, it would be very inefficient. Like what's actually happening is that you have this, I think some other people have made this point as well, is that evolution is sort of this outer optimizer, that's improving the software efficiency of the brain in a bunch of ways, there's some genetic knowledge that you inherit, not that much, because there isn't that much space in the genome, and then you have this lifetime learning, which is, you don't actually see that much data during lifetime learning, a lot of this is redundant and so on.

39:00So what seems to have changed with humans compared to other animals, is that humans became able to have culture, and they have language, which enables them to have like a much more efficient training data modality compared to animals. They also have, I think, stronger ways in which they tend to imitate other humans and learn from their skills, so they also enables this knowledge to be passed on. I think animals are pretty bad at that compared to humans. So basically as a human, you're just being trained on much more efficient data, and that creates further insights to be then efficient at learning from it, and then that creates this feedback with where the selection pressure gets much more intense.

39:44So I think that's roughly what happened with humans, but a lot of the capabilities that you need to be like a good worker in the human economy, animals already have, so they are able to, like they have quite sophisticated sensory motor skills, I think they are actually able to do, like animals are actually able to pursue long -term goals. But ones that they have been installed by evolution, like I think like a lion will find a gazelle, and that is a complicated thing to do, and requires stalking and blah, blah, blah. But when you say it's still by evolution, like there isn't that much information in the genome too.

40:16But I think if you put the lion in the Sahara, and you're like, go find lizards instead, okay, so suppose you're pretty human, and they haven't seen the relevant training data. I think they'd like, they do slightly better, slightly better, but not that much better. Like I think a lot of the, like again, didn't you have recently have an intro -suffeneric? Yeah, so like you would probably tell you that, well, I think what you're making is actually a very interesting and subtle point, that is an interesting implication. So often people point to, they say that ASI will be this huge discontinuity, because while we have this huge discontinuity in the animal to human transition, we're like something, it's like not that much change between pre -human primates and humans genetically, but it resulted in this humongous change in capabilities.

41:08And so they say, well, why not expect to think similar between human level intelligence and superhuman intelligence? And the point you're making is that, or at least one implication of the point you're making is that actually it wasn't that we just gained into this incredible intelligence because of biological constraints. We, the animals have just been held back in this really weird way that no AI system has been arbitrarily held back of not being able to communicate with other copies or with other knowledge sources. And so since AI's are not held back artificially in this way, there's not gonna be a point where you would take away that hobbling, and then now they're like, now I'll explain.

41:47Now, actually I think I would disagree with that. The implication that I made, I would actually disagree with. I'm like a sort of like, unsteerable chain of thought, just, but because as we wrote a blog post together about AI corporations, where we discussed, actually there will be a similar unhobbling with future AI's, which is not about the intelligence, but a similar level of bandwidth and communication and collaboration with other AI's, which is a similar magnitude of change from non -human animals to humans in terms of their social collaboration that AI's will have with each other because of their ability to copy all their knowledge exactly to merge, to distill themselves, to scale.

42:28So maybe before we talk about that, I think just like a very important point to make here, which I think underlies some of this disagreement that we have with others about both this argument from the transition from kind of non -human animals to humans is this like focus on intelligence and reasoning and R &D, which is enabled by that intelligence as being just enormously important. And so if you think that you get this very important difference from this transition from primates, non -human primates to humans, then you think that in some sense, you get this enormously important unlock from fairly small scaling and say brain size or something.

43:14And so then you might think, well, yeah, I could be, if we scale beyond the size of training runs that the amount of training compute that the human brain uses, which is maybe on the order of, 1, 8, 24, flop or whatever, which we've recently surpassed, then maybe surpassing it just a little bit more enables us to unlock very sophisticated intelligence in the same way that humans have much more sophisticated intelligence compared to non -human primates. And I think part of our disagreement is that intelligence is kind of important, but just having a lot more intelligence and reasoning and good reasoning isn't something that will kind of accelerate technological change in economic growth very substanti.

43:59Like it isn't the case that the world today is just like totally bottlenecked by not having, you know, not having enough good reasoning and that's not really what's bottlenecking the world's ability to grow much more substanti. I think that we might have some disagreement about this particular argument, but I think what's also really important is just that we have a different view as to how this acceleration happens that it's not just having like a bunch of really good reasoners that give you this technology that then accelerates things very drastically because that alone is not sufficient. You need kind of complimentary innovations in other industries, you need the economy as a whole growing and supporting the development of these various technologies, you need the very supply chains to be upgraded, you might need demand for the various products that are being built.

44:51And so we have this view where actually this very broad upgrading of your technology and your economy is important rather than just having very good reasoners and very, very good reasoning tokens that gives us this acceleration. All right, so this brings us back to the intelligence explosion. Here is the argument for the intelligence explosion. Look, you're right that certain kinds of things might take longer to come about, but this core loop of software R &D that's required. If you just look at what kinds of progress is needed to make a more general intelligence, you might be right that needs more experimental compute, but like we're just getting, as you guys have documented, we're just getting like a shit ton more compute every single year for the next few years.

45:41So you can imagine that intelligence explosion for the next few years where in 2027, there will be like 10x more compute than there is now for AI. And you'll have this effect where the AI's that are doing software R &D are finding ways to make running copies of them more efficient, which has two effects. One, you're increasing the population of AI's where doing this research. So more of them in parallel can find these different optimizations. And a subtle point that they'd often make here is software R &D in AI is not just alia type coming up with new transformer like architectures. To your point, it actually is a lot of, like you've got to like, I mean, I'm not a AI researcher, but I assume there's like, from the lowest level libraries to the kernels to making RL environments to finding the best optimizer to, there's just like so much to do.

46:40And in parallel, you can be doing all these things or finding optimizations across them. And so you have two effects going back to this. One is you, if you look at the original GPT -4 compared to the current GPT -40, I think it's like, how much cheaper is it to run? It's like 100. Yeah, yeah. So you have the same capability or something. So they're finding ways in which to run more copies of them, like, you know, 100 X -Sheeper or something, which means that the population of them is increasing and the higher population is then helping you find more efficiencies. And not only does that mean you have more researchers, but to the extent that what's the complementary input is experimental compute.

47:21It's not the compute itself, it's the experiments. And the more efficient it is to run a copy or to develop a copy, the more parallel experiments you can run because now you can do a GPT -4 scale training run for much cheaper than you could do it in 2024 or 2023. And so for that reason, also, this like software -only singularity sees more researcher copies who can run experiments for cheaper. Dot, dot, dot, they initially are maybe handicapped in certain ways that you mentioned. But through this process, they are rapidly becoming much more capable. What is wrong with this logic? So I think the logic seems fine.

48:00Yeah. I think this is a decent way to think about this problem. But I think that it's useful to draw on a bunch of work that say economists have done for studying the returns to R &D and what happens if you 10X your inputs, the number of researchers, what happens to innovation or the rate of innovation. And they point out these two effects where as you do more innovation, then you get the stand on top of the shoulders of giants and you get the benefit from past discoveries and it makes you as a scientist more productive. But then there's also kind of diminishing returns that the low hanging fruit has been picked and then becomes harder to make progress.

48:39And overall, you can summarize those estimates as thinking about the kind of returns to research effort. And we've looked into the returns to research effort in software specifically. And we look at a bunch of domains in traditional software or linear integer solvers or SAT solvers, but also an AI, like computer vision and RL and language modeling. And there, if this model is true that all you need is just cognitive effort, it seems like the estimates are a bit ambiguous about whether this results in this acceleration or whether it results in just merely exponential growth. And then you might also think about, well, it isn't just your research effort that you have to scale up to make these innovations because you might have complementary input.

49:29So as you mentioned, experiments are the thing that might kind of bottleneck you. And I think there's a lot of evidence that in fact, these experiments and scaling up hardware is just very important for getting progress in the algorithms and the architecture and so on. So in AI, this is true for software in general where if you look at progress in software, it often matches very closely the rate of progress we see in hardware. So for traditional software, we see about a 30 % roughly increased per year, which kind of basically matches more so. And in AI, we've seen the same until you get to the deep learning era and then you get this acceleration, which in fact coincides with the acceleration we see in compute scaling, which gives you a hint that actually the compute scaling might have been very important.

50:19Other pieces of evidence, you know, besides this coincidental rate of progress, other kind of pieces of evidence are, you know, the fact that innovation in algorithms and architectures are often concentrated in GPU rich labs and not in the GPU poor parts, you know, for the world like academia or maybe smaller research institutes, that also suggested having a lot of hardware is very important. If you look at specific innovations that seem very important, the big innovations over the past five years, many of them have some kind of scaling or hardware related motivation. So, you know, you might look at the transformer itself was about how to harness more parallel compute.

51:05Things like flash attention was literally about how to implement the attention mechanism more efficiently or things like the Chinchilla scaling law. And so many of these big innovations were just about how to harness your compute more effectively. That also tells you that actually the scaling of compute might be very important. And I think there's just like many pieces of evidence that points towards this complementarity picture. So I would say that not only, like even if you assume that experiments are not particularly important, the evidence we have both from estimates of AI and other software, although the data is a bit is not great, suggests that, you know, maybe you don't get this kind of hyperbolic faster than exponential, you know, super growth in the overall algorithmic efficiency of systems.

51:56I'm not sure I buy the argument that because these two things compute and AI progress have risen so concomitantly that this is a sort of causal relationship. So broadly, the industry as a whole has been getting more compute and as a result, making more progress. But with it, if you look at the top players, there's been multiple examples of a company with much less compute, but a more coherent vision, more concentrated research effort, being able to beat an incumbent who has much more compute. So open AI initially beating Google DeepMind. And if you remember, there was these emails that were released between Elon and Sam and so forth.

52:35We're like, we gotta start this company because they've got the spotlight on the compute and like look how much more compute Google DeepMind has. And then open AI made a lot of progress, similarly now with open AI versus in Theropik and so forth. And then I think just generally, your argument is just like two outside view. When we just do know a lot about what is, like you're just like, this is very macroeconomic argument that I'm like, well why don't we just ask the AI researchers? I mean, AI researchers will often kind of overstate the extent to which just cognitive effort and doing research is important for driving these innovations because that's often kind of convenient or useful.

53:14They will say the insight was derived from some kind of nice idea about statistical mechanics or some nice equation in physics that says that we should do it this way. And then, but often that's kind of an ad hoc story that they tell to make it a bit more compelling to the kind of reviewers. So Daniel mentioned this like survey he did or where he Daniel Kukukatalo. It has to be a researchers if you had 130 at the amount of compute and he did 130 it because he is supposed to think 30 times faster. If you had 130 at the amount of compute, how much would your progress slow down and they say I make a third of the amount of progress, I normally do.

53:58So that's just a pretty good substitution effect of, if you get one tenth of compute, your progress only goes down one third. I was talking to an AI researcher the other day who's like just like one of these like cracked people, gets paid millions and attends a millions of dollars a year probably. And we asked him how much do these AI models help you in domains you already are, you know, how much does the AI models help you in AI research? And he said in domains that I'm already quite familiar with where I just closer to autocomplete, it's like saves me four to eight hours a week. And then he said but in domains where I'm actually less familiar, where it's like I need to try new connections, I need to understand how these different parts relate to each other and so forth.

54:41It saves me closer 24 to 36 hours a week, right? So that, and then that's like current models. And I'm just like, you didn't get more compute but it's still saved him like a shit ton more time. Like it just like draw that forward. It's like that's a crazy implication or crazy trend, right? I mean, I guess have we seen, like I'm skeptical of declames because that we have actually seen that much of an acceleration in the process of R &D. Like these claims seem to me like they're not borne out by the actual data I'm seeing. So I'm not sure how much to trust them. I mean, on the general intuition that cognitive effort alone can give you a lot of AI progress, right?

55:22We've had, it seems like a big important thing the labs do is this like science of deep learning. Like scaling laws is just, you, I mean, like it ultimately knitted out an experiment but the experiment is motivated by cognitive effort. So for what is worth, when you say that AMB are complimentary, you're not saying like just as you can't get a lot of progress, like just can, just as A can ball like you B can also ball like you. Yeah. So when you say you need a computer and experiments and data but you also need cognitive effort, like that doesn't mean the lab who has the most computer is gonna win, right?

55:54Like that's a very simple point. Like either one can be the ball. Like if you just have a really dysfunctional culture and you can't, you don't actually prioritize using your computer very well and you just wasted it, well then you're not gonna make a lot of progress, right? So like it doesn't contradict a picture that someone with a much better vision, a much better team, much better prioritization can make better use of their compute. If someone else was just bottlenecks heavily on that part of the equation. The question here is once you get these automated AI researchers and you start this software singularity, your efficiency, software efficiency is gonna improve by many orders of magnitude.

56:32While your compute stock at least in sort of the short run is gonna remain fairly fixed. So how many ooms of improvement can you get before you become bottlenecks by the second priority equation? And once you actually factor that in, like how much progress you're gonna expect because that's the kind of question and I think people don't have, I think it's hard for people to have good intuitions about this because people usually don't run the experiments. So you don't get to see at a company level or at an industry level, what would have happened if the entire industry had three times less compute?

57:06Maybe as an individual, like what happened if you had three times less compute, you might have a better idea about that, but that's a very local experiment. And you might be benefiting a lot from spillovers, from other people who actually have more compute. So because this experiment was never run, it's sort of hard to get direct evidence about this strength of complementarity. What is your probability of, if we live in the world where we get AGI in 2027, that there is a software only singularity? Quite high because you're conditioning on the AGI. And you're conditioning on the AGI. That's a compute not being very large.

57:40So it must be that you get a bunch of software progress. Yeah. Right, right. Like you just get a bunch of leverage from algorithmic progress in that world. Okay, that's right. So then maybe the, because I was thinking these are independent questions. I think a call I'd like to make is, I know that some labs do have multiple pre -training teams. And they give people different amounts of resources for doing the training and different amounts of cognitive effort, different size of teams. But none of that, I think, has been published. And I'd love to see the results of some of those experiments. I mean, I think even that won't update you very strongly, just because it is often just very inefficient to do this very imbalance scaling of your factor inputs.

58:24And in order to really get an estimate of how strong these complementarities are, you need to observe these very imbalance scale ups. Yeah. And so that really happens. And so I think the data that bears on this is just really quite poor. And then, you know, the intuitions that people have also don't seem clearly relevant to the thing that matters about what happens if you do this very imbalance scaling. And where does this net out? One question I have, which I like, that it would be really interesting if somebody can provide an example of, is maybe through history, there was some point at which, because of a war or some other kind of supply shock, you had to ramp up production or ramp up some key output that was people really cared about.

59:12While for some weird, like, historical reason, many of the key inputs were not accessible to a ramp up. But you could ramp up one key input. I'm talking very abstract terms. What do you say I'm saying, right? You need to make more like bombers, but like you ran out of aluminum and you just like need to figure out something else to do. And how successful these efforts have been or whether you just keep getting bottlenecked? Well, for, I think that is not quite the right way to do it. Because I think, like, if you're talking about materials, then I think there's a lot of sense in these different materials can be substitutable for one another in different ways.

59:46Like, you can use aluminum. I mean, aluminum is a great matter for making aircraft because it's a sort of light, and durable, and so on. But you can imagine that you make aircraft with like worse metals, and then it just takes more fuel, and it's like less efficient to fly. So there's a sense in which you can compensate and just cause more. Like, I think it's much harder if you're talking about something like complementarity between laboring capital, complementary between like remote work and in -person work or like skilled or unskillable work. Like, there are input pairs for which I would expect it to be much more difficult.

1:00:23For example, you're looking at the complementarity between like the quality of leadership of an army and its number of soldiers, right? I mean, did there is some effect there? Like, if you just scale up, you just have excellent leadership, but your army only has 100 people. Like, you're not going to get very far. So, can you unite us in the thermopoli? Well, they lost, right?

1:00:46Who would be funny if we were building models to suffer really singularity, and we're like, what exactly happened in thermopoli? It's like somehow relevant. I mean, I can actually talk about that, which we probably shouldn't. Okay, sure. By the way, so for the audience to know, my most popular guest by far is Sarah Payne. Not only she's my most popular guest, she's my most popular four guest, because I call her all four of those episodes that I've done on there, or like from a viewer -minute -adjusted basis, I host the Sarah Payne podcast, right? Occasionally, talk about AI.

1:01:20And anyways, we did this three -part lecture series where we're talking about, like, one of them was about India, Pakistan, wars through history. One of them was about, was it the Japan, like, Japanese culture before World War II, the third one was about the Chinese Civil War. And for all of them, my tutor, my history tutor, was Ege. And it's just like, why does he know so much about fucking random, like 20th century conflicts? But he did, and he suggested a bunch of the good questions I asked her, we'll get into that. And actually, what's going on there? I don't know. I mean, I don't really have a good question.

1:01:58I think it's interesting. I mean, I read a bunch of stuff, but it's a kind of boring answer, like, I know. Like, imagine you ask, like, a top AI researcher, like, what's going on? Like, how are you so good? And then they would probably give you, like, a boring answer. Like, I don't know, like, I did this. That itself is interesting. That often, these kinds of questions, illicit boring answers. Yeah. Like, it tells you the nature about the nature of the skill. How do you find them? We connected on, like, some, on a discord for Metaclis, which is this forecast -saving platform. And I was, I was a graduate student at Cambridge at the time doing research in economics.

1:02:38And I was having conversations with my peers there. And I was occasionally having conversations with EGET. And I was like, this guy knows a lot more about economics. And he's, at the time, he was a computer science undergrad in Ankara. And he knows more about economics and about, you know, these big trends in economic growth and economic history, than almost any of my peers at the university. And so, like, what the hell is up with that? So we started, like, having frequent collaborations and ended up hiring EGET for E -POC, because it clearly makes sense for him to work on these types of questions.

1:03:17And it seems like an E -POC just collected this, like, a group of internet misfits and weirdos. Yeah, that's right. How did you start E -POC? And then how did you accomplish this? Yeah, so I was at MIT doing more research. I was pretty unhappy with the bureaucracy there, where it was very hard for me to scale projects out, hire people. And I was pretty excited about a bunch of work that my PI wasn't excited about, because it's maybe hard to publish or isn't, it doesn't confer the same prestige. And so, you know, I was chatting with Chymes Sevilla, one of the co -founders, and we just collaborated on projects and then thought we should just start our own org, because we can just hire people and work on the projects we were excited about.

1:04:07And then I just hired a bunch of the insightful misfits that, like... But did you, like, was the thesis, like, oh, there's a bunch of underutilized internet misfits and therefore, like, this org was successful, or you started the org and then you were like... I think it's more of the latter. So it was more like, we could make a bunch of progress, because clearly, like, academia and industry is kind of dropping the ball on a bunch of important questions that academia is unable to publish. Interesting papers on. Industry is not really focused on producing useful insights. And so it seemed like very good for us to just do that.

1:04:43And also the timing was very good. So we started just before, you know, chat GPT and we wanted to have much more grounded discussions of the future of AI. Yeah. And I was frustrated with the quality of discussion that was happening on the internet about the future of AI. And, I mean, to some extent, or to a very large extent, I still am. Yeah. And that's, like, a large part of what, you know, motivates me to do this. It's just, like, born out of frustration with bad thinking and arguments about where AI is going to go. The part about my job that I enjoy the least is the post -production. I have to re -watch the episode multiple times, make all these difficult judgment calls.

1:05:22And I've been trying to automate all this work with LLM scripts. And I found that Google's Gemini 2 .5 Pro is the best model I've tried for these tools. So much of the post -production requires understanding the delivery, the context, all these other things that you don't get from a text -only transcript. Unlike other models I've tested, I can actually just shove in the four -hour raw audio file into Gemini because of its multimodal capabilities. And it can generate super high -quality transcripts, identify great snippets for clips, a bunch more other things. I've actually made a repo with all these tools and I've linked the GitHub in the description below in case you might find it helpful.

1:05:59I actually use 2 .5 Pro in order to write the code for these scripts. It's actually quite interesting to read as reasoning traces as it's thinking through your gnarly list of requests and tasks. Gemini 2 .5 Pro is now available in preview with higher rate limits. You can try it out at aistudio .google .com. Thanks to Google for sponsoring this episode and now back to Ege and Taame. Okay, so let me ask you about this. I can poke you from the, so just to set the scene for the audience. We're gonna talk about the possibility of this explosive economic growth and greater than 30 % economic growth rates.

1:06:39So I wanna poke you both on perspective of maybe suggesting that this isn't aggressive enough in the right kind of way because it's, maybe it's too broad. And then I'll poke you in the, I'll put you in the picture of the more normal perspective that hey, this is fucking crazy. I imagine it would be difficult for you to do the second thing. So that's the problem. Not mean like I think it might be fucking crazy. Let's see. The big question I have about this broad automation, like I get what you're saying about the industrial revolution, but in this case, we can just make this argument that you get this intelligence and then what you do next is you go to the desert and you build this like chenzen of robot factories which are building more robot factories which are building, if you need to do experiments and you build bio labs and you build chemistry labs and whatever.

1:07:30There are a lot of chenzen in it at the desert. I think that looks much more plausible than a sulfur on this singularity. But why, the way you're framing it, it sounds like McDonald's and Home Depot and a fucking whatever are growing at 30 % a year as well. And not just the aliens double view of the economy is that there's a robot economy in the desert that's growing at 10 ,000 % a year and everything else is the same old table. There is a question about what would be possible or physically possible and what would be the thing that would actually be efficient. So it might be the case. And again, once you're scaling up the hardware part of the equation as well as the sulfur part, then I think the case for this feedback loop gets a lot stronger.

1:08:18If you scale up data collection as well, I think it gets even stronger, like real world data collection by deployment and so on. But building chenzen in a desert, that's a pretty, like if you look, if you think about the pipeline, so far we have relied, first of all, we're relying on the entire semiconductor supply chain and industry depends on tons of inputs and materials and whatever, it gets from probably tons of random places in the world and creating that infrastructure, like doubling or tripling, whatever, that infrastructure, it looked the entire thing. That's very hard work, right? So probably you couldn't even do it even if you just have chenzen in a desert.

1:08:56Like that will be even more expensive than that. I want to talk about that so far, we have been drawing heavily on the fact that we have built up this huge stock of data over the past 30 years or something on the internet. Like imagine you were trying to train a state art model, but you only have like 100 billion tokens, right? The train all, that would be very difficult. So in a certain sense, we, our entire economy has produced this huge amount of data on the internet that we are now using the train models. It's plausible that in the future, when you need to get new competencies added to these systems, the most efficient way to do that will be to try to leverage similar kind of model of his data, which will also require this.

1:09:42Like you would want to deploy the systems broadly because that's going to give you more data. And maybe you can do the, maybe you can get where you want to be without that, but it would just be less efficient if you're sign from scratch, compared to if you're collecting a lot of data. Like I think this is actually a motivation for why labs want their LLMs to be deployed widely because sometimes when you talk to chat GBT, it's going to give you two responses and it's going to say, well, which one was good? Or like it's going to give you one response and it's going to ask you, it was as good or not.

1:10:12Well, why are they doing that? That's a way in which they are getting user data through this extremely broad deployment. So I think you should just imagine that thing to be, continue to be efficient and continue to increase in the future because it just makes sense. And then there's a separate question of, well, suppose you didn't do any of that, I guess suppose you just tried to imagine the most rudimentary, the most narrowest possible kind of infrastructure buildout and deployment that would be sufficient to get this positive feedback loop that leads to like much more efficient AI's. I agree that loop could in principle be much smaller than the entire world.

1:10:53I think it could probably, it couldn't be as small as chance in a desert but it could be much smaller than the entire world. But then there's a separate question of, would you actually do that? Or would that be efficient? I think some people have the intuition that there are just these like extremely strong constraints, maybe regulatory constraints, maybe social political constraints, to doing this broad deployment. They just think it's gonna be very hard. So I think that's part of the reason why they imagine these like more narrow scenarios where they think it's gonna be easier. But I think that's overstated.

1:11:27I think people's intuitions for like how hard this kind of deployment is comes from cases where the deployment of the technology wouldn't be like that valuable. So if I come from housing, like we have a lot of regulations in housing, maybe it comes from nuclear power, maybe it comes from supersonic flights. I mean those are all technologies that would be useful if they were like maybe less regulated but it they wouldn't like double economic output. I think the core point here is just the value of AI automation and deployment is just extremely large. Even just for workers, at least the ones that, at least after finding, there might be some kind of displacement and there might be some transition that you need to do in order to find a job that works for you.

1:12:14But otherwise the wages could still be very high for a while at least. And on top of that, the gains from owning capital might be very enormous. And in fact, a large share of the US population would benefit from housing. They benefit their own housing, they have 401Ks, those would do enormously better when you have this process of broad automation and AI deployment. And so I think there could just be very deep support for some of this, even when it's like totally changing the nature of labor markets and the skills and occupations that are in demand. So I would just say it's complicated. I think what the political reaction to it will be when this starts actually happening.

1:13:03I think the easy thing to say is that, yeah, this will become a big issue and then this will be maybe controversial or something. But what is the actual nature of the reaction in different countries? I think that's kind of hard to forecast. I think the default view is like, well, people are going to become unemployed. So it will just be very unpopular. I think that's very far from obvious. Yep. And I just expect heterogeneity in how different countries respond as some of them are going to be more liberal about this and going to have broader deployments. And those countries probably end up doing better.

1:13:32Just like doing industrial revolution. Some countries were just ahead of others. I mean, eventually almost the entire world adapted the norms and culture and values of the industrial revolution in various ways. And actually, you say they might be more liberal about it, but they might actually be less liberal. They might be less liberal in many ways. And in fact, that might be more, functional in this world in which you have broad AI deployment. I think we might adopt the kind of values and norms that get developed in say, you know, the UAE or something, which is like, maybe focused a lot more on making an environment that is very conducive for AI deployment.

1:14:12And we might start emulating and adopting various norms like that. And they might not be kind of classical liberal norms, but norms that are just more conducive to AI being functional and producing a lot of values. This is not meant to be like a strong prediction. This is just an illustrative. Yeah, yeah. It might just be that like the freedom to deploy AI in the economy and build out lots of physical things at scale. Maybe that ends up being more important in the future. Maybe that is still missing something. Maybe there's other things that are also important. But like the generic prediction that you should expect variance and some countries do better than others.

1:14:49I think that's much easier to predict than like the specific countries that end up doing better. Yeah, or the norms that that country will necessarily add. So I mean, one thing I'm confused about is if you look at the world of today versus the world of 1750, the big differences is just like, like we've got crazy tech that they didn't have back then. We've got these cameras and we've got these screens and we've got rockets and so forth. And that just seems like the result of technological growth and R &D and so forth. Cap select simulation. Well, I explain that to me because you're just talking about this infrastructure build out and blah, blah, blah.

1:15:29I'm like, well, why won't they just like fucking invented the kinds of shit that humans would have invented by 2020, 2020 50? I'm pretty sure you've seen this stuff that takes a lot of infrastructure build out. But that infrastructure is built out once you make the technology, right? I don't think that's right. Like there isn't this like temporal like difference where it's first you do the invention and then like often there's this interplay between the actual capital build up and the innovation and learning curves are about this right fundamentally. Like what has driven the increase in the efficiency of solar panels over the past 20, 30 years?

1:16:05Like it isn't just like people had the idea of like 20, 25 solar panels. Like no one ever had like nobody 20 years ago had like the sketch for a 20, 25 solar panel. That's not it's this kind of interplay between having ideas, building, learning, producing and other complimentary inputs also becoming more efficient at the same time. You might get better materials. Like for example, the fact that aluminum becomes something that's like for example, the fact that smelting process got a lot better towards the end of the 19th century. So it became a lot easier to work with metal. Maybe that was a crucial reason why aircraft technology later become became more popular.

1:16:47So it's not like someone came up with the idea of oh, well you can just like use something that just has wings and has a lot of thrust and then that might be able to fly. Like that it based ideas not that difficult. But then well how do you make it like actually a viable thing? That's right. Well that's much more difficult. Have you seen the meme where two beavers are talking to each other and they're looking at the Hoover Dam? And one of them's like, well I didn't build that. It's based on an idea of mine. That's right. The point you're making is that when this invention focused look on tech history, under place, the work that goes into making specific innovations practicable and to deploy them widely.

1:17:34It's just hard I think. Like it's just hard to even, suppose you want to write a history of this, right? Like you want to write a history of how was the light bulb developed or something. It's just really hard because to understand why specific things happen that specific times, you probably need to understand so much about the economic conditions of the time. Like for example, Edison spent a ton of time experimenting with different filaments using the light bulb. Like the basic idea is very simple. You make something hot and it glows. Then what actually works well for that in a product. What is durable?

1:18:11What has the highest ratio of light output versus heat and so that you have less waste than it's more efficient. And then even after you have the products, then your face is problem well. I mean it's like 1880 or something and then US homes don't have electricity. So then nobody can use it. So now you have to build power plants and build power lines to the houses so that people have electricity in their homes so that they can actually use this new light bulb that you created. So he did that. But then people presented as if it's like a, okay he just came out with the idea. Like it's light bulb.

1:18:45I guess if you think people would say is like, you're right about how technology would progress if we were humans deploying for the human world. But what you're not counting is there's just going to be this like AI economy where maybe they need to do this kind of innovation and learning by doing when they're figuring out how do I want to make more robots because they're helpful. And so like we're going to build more robot factories we'll learn and then we'll make better robots or whatever. But like that is just like a geographically that is a small part of the world that's happening in. Or you're just doing something like it's not like and then like then they walk in your building and then you do a business transaction with lunar society podcasts LLC.

1:19:28And then you know, I mean for what is worth? Like if you look at the total surface area of the world in my world with the case that the place that initially experiences this very fast growth is like a small percentage of the surface area of the world. Right. But I think that was the same for an industrial revolution was not different. Yeah. So but then like I'm just like what concretely does this explosive growth look like? If I look at this heat map of growth rates. Right. So on the globe is there's going to be like one area is like blinding hot and that's the you know that's like the desert factories with all these experiments.

1:20:02And like yeah. So I would say our idea is that it's going to be broader than that but probably initially. So eventually it would be probably most of the world. But as I said because of this heterogeneity because I think some countries are going to be faster in adoption than others. Maybe some cities will be faster. Option than others. And that will mean that there are there is differentials and some countries might like have much faster growth than other countries. But I would expect that at a jurisdiction level it would be like more homogenous. So for example, I expect the primary obstacles to come from things like regulation.

1:20:42And so I would just imagine it's being like more delineated by regulatory jurisdiction boundaries than anything else. So you maybe write that this infrastructure build out in capital deepening and whatever is necessary for a technology to become practical. But or you want to be discovered. Like there's the aspect of it that where you discover certain things by scaling up learning by doing. Yeah. Right in learning curve. And there's this separate aspect where you get to like suppose that you become wealthier. Well, you can invest that increased wealth in like you yeah, you use it to it can make more capital but you also can invest it in R &D.

1:21:20And other ways. Get Einstein out of the patent office. Like you need some amount of resources for that to make sense. And you need the economy to be of a certain scale. You also need demand for the product you're building. So like you know, you could have the idea but if the economy is just too small that you know there isn't enough demand for you to be specializing in producing the semiconductor or whatever. Because there isn't enough demand for it then it doesn't make sense. So you want the economy like a much larger scale of an economy is useful in very many ways. In delivering complimentary innovations and their discovery is happening through serendipity.

1:21:55Producing like having there be consumers that would actually pay enough for you to recover your fixed costs of doing all the experimentation and the invention. You need the supply chains to exist to deliver the germanium crystals that you need to do grow in order to come up with the semiconductor. You need a large labor force to be able to help you do all the experiments and so on. I think the point you're illustrating is like, look could you have just figured out that there was a big bang by first principles reasoning? Maybe but what actually happened is we had World War II and we discovered radio communications in order to fight and effectively communicate during the war.

1:22:37And then that technology helped us build radio telescopes and then we discovered cosmic microwave background and then we had to come up with an explanation of cosmic microwave. And then we discovered like the big bang as a result of like World War II. In fact, people under emphasized that like giant effort that goes into this kind of build up of all the relevant capital and all the relevant supply chains and the technology. I mean earlier you were making a similar comment when you were saying, oh, you know, reasing models actually in hindsight, they looked pretty simple. But then you were kind of ignoring this giant kind of upgrading of the technology stack that happened, you know, that took five to 10 years.

1:23:13Yeah. Prior to that. And so I think people just under emphasize the support that is had from the overall upgrading of your technology of the supply chains of various sectors that are important for that. And people focus on just specific individuals of like, you know, Einstein had this genius inside. And like, you know, he was the kind of very pivotal thing in the causal chain that resulted in these discoveries or Newton was just extremely important for discovering calculus without thinking about, well, there was this kind of all these other factors that produced lenses, that produced telescopes that got the right data and that made people ask questions about dynamics and so on that motivated some of these questions.

1:24:00And those are also extremely important for science, scientific and technological innovation. Yeah. You know, conquest, what is it? One of conquest laws is like the more you understand about a topic, the more conservative you become about that topic. And so there might be like a similar law here where like the more you understand about an industry, the more sort of like, obviously I'm just like a commentator or whatever or a podcaster. But I understand EI better than any other industry I understand. And there I have the sense from talking to people like you that oh, so much went into getting EI to the point where it is today.

1:24:38Whereas when I talk to journalists about EI, they're like, okay, who is the, who is a crucial person to be to cover? And they're like, should we get in touch with Jeffrey Hinton? Should we get in touch with Ilya? And I just have this like, you're kind of missing the picture. It's not, but then you should have that same attitude towards things you, maybe the more similar phenomenon is Juman Amnesia. We should have a similar attitude towards other industries that it's like much more complicated. Right, I mean, it's so Robe and Hansen has this abstraction of like seeing things in near mode or just far mode.

1:25:05Right. And I think if you don't know a lot about the topic because then you see it sort of in far mode. And you sort of simplify. And that's right. Things you know you see a lot more detail. Like in general, I think the thing I would say and the reason I also believe that just like abstract reasoning and like sort of structured reasoning or even Bayesian reasoning by itself is not like sufficient or like is not as powerful as many other people think is because I think there's just like enormous amount of like richness and detail in the real world that like you just can't like reason about it. Right.

1:25:37Like you need to see it. And obviously that like that is not an obstacle to AI being incredibly transformative. Because as I said, like you can scale your data collection. You can scale experiments. You do both in the AI industry itself and just more broadly in the economy. Right. So you just discover more things. More economic activity means we have more exposed surface area to have more discoveries. Like all of these are things that have happened in our past. Right. There's no reason that they couldn't speed up. Like the fundamental thing is that there's no reason fundamentally why economic growth can't be much faster than it is today.

1:26:10Like it's probably about as to us right now just because humans are such an important bottleneck. They both supply the labor. They play crucial roles in the process of like discovery of various kinds of productivity growth. There's just strong complementarity just to make sense with capital that you can't substitute machines and so on for humans very well. So the growth of the economy and growth productivity just as it being bottlenecks by the growth of human population. Publicly available data is running out. So major AI labs like Meta, Google DeepMind, and OpenAI partner with scale to push the boundaries of what's possible.

1:26:46Through scales data foundry, major labs get access to high quality data to fuel post -training, including advanced reasoning capabilities. As AI research forward, we must also strengthen human sovereignty. Scales research team seal provides practical AI safety frameworks evaluates frontier AI system safety via public leaderboards and creates foundations for integrating advanced AI into society. Most recently, in collaboration with the Center for AI Safety, scale published Humanities Last Exam, a groundbreaking new AI benchmark evaluating expert level reasoning and knowledge capabilities across a wider range of fields.

1:27:25If you're an AI researcher or engineer, and you want to learn more about how scales data foundry and research lab can help you go beyond the current frontier of capabilities go to scale .com slash the war cache. Let me ask this in a general question. What has happened in China over the last 50 years? Would you describe that as, in principle, the same kind of explosive growth that you expect from me? Because there's a lot of labor that makes the marginal product of capital really high, which allows you to have 10 % plus economic growth rates is that basically in principle for me AI? So I would say in some ways it's similar, in some ways it's not.

1:28:05The way probably the most important way in which it's not similar is that in China, you see it as relative, like you see a massive amount of capital accumulation, substantial amount of adoption of new technologies and probably also human capital accumulation, to some extent. And but you're not seeing a huge scale up in the labor force, the fake labor force. Well, for AI, you should expect to see a scale up in a labor force as well, not in human workforce, but in the AI workforce. I think you did kind of like maybe not consecutive increases in the labor force increase, but like you did. The key thing here is just the simultaneous scaling of both these things.

1:28:44And so you might ask the question, isn't it like basically half of what's going to happen with AI that you scale up, capital accumulation in China? But actually that's really not like if you get both of these things to scale, that gives you just much faster growth and a very different picture. But at the same time, if you're just asking like what would 30 % growth per year look like, like in terms of like if you're just going to have an intuition for how transformative that would be in concrete terms, then I think looking at Chinese not such a bad case, like especially in the 2000s or maybe late 90s, like that gives you a good, that seems slower than over forecast.

1:29:23I think also looking at the industrial revolution is pretty good. Well, natural revolution is very slow. So but just in terms of the types of the margins along which we may progress in terms of products. So what didn't happen, the thing that didn't happen during the industrial revolution is we just produced a lot more of things that people were producing prior to the industrial revolution, like producing a lot more crops and maybe a lot more kind of pre -industrial revolution style houses or whatever on farms. Instead, what we got is along pretty much every main sector of the economy. We just had many different products that are totally different from what was being consumed prior to that.

1:30:10So in transportation, in food. I mean, healthcare is a very big deal and to be out of it. So another question, because I'm not sure I understand how you're defining the learning by doing versus explicit R &D. Because there's the way for taxes that companies say what they call R &D. But then there's the intuitive understanding of R &D. So if you think about how AI is boosting a TFP, you could say that right now if you just had replaced the TSMC process engineers with AI's and they're finding different ways in which to improve that process and improve efficiencies, improve yield. Right. I would kind of call that R &D.

1:30:50On the other hand, you emphasize this, the other part of TFP, which is like better management than doing by doing. That kind of stuff. Anybody doing could be, you could, I mean, how much, how much on for you're gonna get to the, you're gonna get to like the fucking Dyson Spear by Better Management. Like, it's not just, but that's not the argument, right? Like the point is that there are all these different things that like somewhat are maybe more complimentary than others. The point is not that you can get to a Dyson Sphere by just scaling labor and capital. Like, that's not the point. Like, you need to scale everything at once.

1:31:22So just as you can't get to a Dyson Sphere by just scaling labor and capital, you also can't get to it by just scaling TFP. That doesn't work. I think there's a very important distinction between what is necessary to scale, to get to this Dyson Sphere world and what is important. Like in some sense, producing food is necessary. But of course, producing food doesn't get you to a Dyson Sphere, right? So I think R &D is necessary, but on its own isn't sufficient. And scaling up the economy is also necessary. On its own is not sufficient. And then you can ask the question, what is the relative importance of each?

1:32:00Yeah. So I think our view here is like very much the same. We're like, it is very connected to our view about the software R &D thing, where we're just saying like there are these bottlenecks. So you need to scale everything at once. Like this is just a general view. But I think people like misunderstand us sometimes as saying that R &D is not important. Like no, that's not what we were saying. We're saying it is important. It is less important in relative terms than some other things, none of which are by themselves sufficient to enable this growth. So the question is like, how do you do the credit attribution?

1:32:32I mean, one way in economics is to standard do that is to look at the elasticity of output to the different factors. Capital is less important than labor because the output elasticity of like labor elasticity output is like 0 .6 for capital is like 0 .3. But neither are by themselves sufficient. Like if you just scaled one of them and the other remained fixed, then neither would be sufficient to indefinitely scale outputs. One question that Daniel posed to me is like, because I made this perspective about everything being interconnected when you were talking about like, another example of people often bring up is, what would it take to build the iPhone in the year 1000?

1:33:13And it's like unclear how you could actually do that without just like replicating every intermediate technology, or most intermediate technologies. And then he made the point like, okay, fine, whatever. Nanobots, like nanobots, there's not a crux here. The crux, at least to the thing he cares about, which is human control, is just by when can the robot economy, whether the AI economy, whether it's a result of capital deepening or whether it's a result of R &D, by when will they have just like the robots? And they have more sort of like human love physical power. Right, but he's imagining like a separate thing called the AI economy.

1:33:53That's why, why would you imagine that? That seems like a, I think it's probably downstream of his views about the software only singularity, but again, like those are views that we don't share. So, it's just much more efficient for AI to operate in our economy and benefit from the existing supply chains and existing markets rather than like set up shop on some island somewhere and do its own thing. Yeah, and then he's not being clear. Like for example, people might have the intuition I brought this up before, like the distinction between what is the minimum possible amount of build out that would be necessary to get this feedback loop up and running, and what will be the most efficient way to do it, which are not the same question.

1:34:31Right, but then people have this view that, oh, like the most efficient thing, in principle, we can't do that because like, I think the example he might give is, when the conquistadors arrived to the New World or when the East India trading company arrived to India, they did integrate into the existing economy. In many cases, like it depends on how you define an integrate, but like the Spanish relied heavily on New World labor in order to do silver mining and whatever. East India trading company was like, it's just a ratio of British people to Indian people which was like not that high, right?

1:35:06So they just had to rely on the existing labor force. But they were still able to take over because of, I don't know what the analogous thing here is, but you see what I'm saying. And so he's concerned about, by when will they, even if they're ordering components off of, Ali Baba or whatever, by what, and sorry, I'm being tried, we're usually only saying, like even if they're going to get into the supply chains, by when are they in a position where, because this part of the economy has been growing much faster, they could take over the government or if they wanted to. That's right, yeah. Okay, so I think that's, eventually, you expect the AI systems to be driving most of the economy.

1:35:52And I don't think that's, unless there are some very strange coincidences where humans are able to somehow uplift themselves and become competitive with the AI's, but stopping being biological humans or whatever seems very unlikely early on, then AI's just going to be much more powerful. So, and I agree that in that world, if the AI is just somewhat coordinated and decided, okay, I wish you just take over or something, they just somehow coordinated to have that goal, then they could probably do it. Okay, but okay, that's also probably true in our world. Like in our world, if the US wanted to invade central island, then probably they could do it.

1:36:35I don't think anyone could stop them. But like, what does it actually mean? I mean, there is this dramatic power imbalance, but that doesn't mean, like, that doesn't tell you what's going to happen, right? Like why doesn't the US just invade Guatemala or something? Like why don't you do that? Seems like they could easily do it. Because the value to the US of the land of, yeah. But I, like, there's, so basically just seems, I agree that there might be true for AI's, because like most of the shoulders in space, and once you get like, you want to do the capital deepening on Mars and like the surface here of the side instead of, instead of like, I mean, in New York City.

1:37:13But so it's deeper than that. Like, there are also the, there's also the fact that if the AI's are going to be integrators into our economy. So basically they start out as like a smaller part of our economy or our workforce, and over time they grow, and over time they might, they become the vast majority of the actual sort of work power of an economy. But they are, they are growing in this existing framework where we have like norms and rules for better coordination and then undermining those things as a cost. So if getting the things that is making the humans wealthier than they used to be for and more comfortable, like, yeah, like you would probably better off if you could just take that from them.

1:37:56But the benefit to you, if you already are getting almost all of the income in the economy, will be fairly small. I mean, I feel like the Sentinel Nile thing is not the, there's one reference class that includes that, but historically there's like a huge reference class that includes. Eastern Eatured Inc. could have just kept trading with the Mughals. They just like took over, right? Like they could have like kept trading with the, like the 50 different nation states and pre -colonial India, but yeah, that's right. I mean, that's what they were initially doing. And then whatever, like I'm not gonna, but that is the reference class of like, yeah.

1:38:30I agree. So like if the question is, like if they are entirely, if they have some totally different values and then they represent most of the economy then like would they take over, right? I still don't know because I'm not sure to what extent the class of OAAI is like a natural class. It's sort of like why don't the young people in the economy like try and show the old nicely. So I agree that sometimes these kinds of class arguments are misused, for example, when Marxists are like, why don't this class uprise against the others? Did anybody have the interesting argument that if you look at the history of the conquistadors, when Cortez was making his way through the new world, he had to actually go back and fight off a Spanish fleet that had been sent to arrest him.

1:39:17And then go back, right? So you can have this fight within this conquering AIs and then that's still, the Native Americans getting disempowered. But with AIs in particular, they're just like copies of each other. And in many other ways, they just have a lot, they have lower transaction costs when they trade with each other or interact with each other. There's other reasons to expect them to just be more compatible, coordinating with each other, than coordinating with the human world. Sure, but then, like I guess I'm still not seeing the... I mean, if the question is just that, is it possible for that to happen, which is like a weak reclaim, then yeah, I mean, it seems possible.

1:39:58But they are writing a lot of arguments, just pushing it against it. Probably actually the biggest one is the fact that AI preferences are just not like, just look at the AIs we have today. Like, can you imagine them doing that? I think people just don't put a lot of weight on that because they think, once we have enough optimization pressure and once they become super intelligent, they're just gonna become misaligned. But I just don't see the evidence for that. No, I mean, I think that's actually, like I agree there's some evidence that they're like good boys. Like no, there's more than some evidence.

1:40:29No, but there's also some evidence that there's a new opening AIPAPER where in Chain of Thought, like reward hacking is such a strong basin that if you were like, hey, let's go solve this coding problem. In Chain of Thought, they'll just be like, okay, let's hack this and then figure out how to hack it. So imagine that you gave students that a school attest and then the answer to key was like one. But the reference to guys of humans does include Cortez and the ECD and Trading Company. Sure. So I think one issue here is that I think people are doing this very kind of partial equilibrium analysis or something where they're thinking about just this raw abilities of AI systems in a world where AI systems are kind of dominant and like human civilization has done very little in terms of integrating itself and the AI is integrating itself into the human world, maybe making, you know, insofar as it's poor at communicating and coordinating with AI's addressing those deficiencies and improving that insofar as that's posing a risk or creating inefficiencies because it's unable to benefit from coordinating and trading, then it should have this enormous incentive to address that.

1:41:43Insofar as there is a lot of value to be gained from dominating and taking over humans, like what you might get is a more negotiated settlement. If that's in the case, then a war would just be inefficient. And so you would want to negotiate some settlement that results in some outcomes that are mutually beneficial. I think to the kind of factual, not compared to, like there was a mutually beneficial trade that was made between the Qing Dynasty and the British after like in the opium wars, right? But it was like, it was like maybe better than like China, pre -industrial China going to war with the British Empire, but it wasn't better than like never having interacted with the British Empire in the first place.

1:42:27So I think like one mistake that I feel people make is they have this very naive analysis of what creates conflict. And I think Matthew has written a bit about this, a colleague of ours, where they say, you know, there's misalignment. And so that then creates conflict. But that's actually not what the literature on what causes conflict says creates conflict. It's not just misalignment. It's also other issues, like not being able, like being, having bad understanding of the relative strengths of your armies versus theirs, or maybe having these very strong commitments that you think some grounds are secret.

1:43:13And so you're not willing to do any trade in order to give up some of that in order to gain something else. And so then you have to pose it some additional things other than just the base value misalignment part. I think you're making a good arguing against, like let's let humans take up the spears and the machetes and go to war against the AI data centers, because maybe like that, yeah, there's not this asymmetric information that often leads to conflicts in history. But there's argument, there's not addressed at all the risk of like just like takeover, which can be the result of a peaceful end negotiation or a human society just like, look, we're totally outmatched and we let just like take these meager concessions rather than going to war.

1:43:57But in so far as it's more peaceful, then I think it's like much less, much less of a thing to worry about, right? Like I think there could be this trend where we indeed have this gradual process where AI is much more important in the world economy and actually deciding and determining what happens in the world. But this could be beneficial for humans where we're getting access to this much larger economy and much more advanced technological stock. Yeah, so I think it's important to be clear about what is the thing that you're actually worried about. I think some people are just like say that, oh, we're gonna, humans are gonna lose control of the future, we're not gonna be the ones who are making the important decision.

1:44:43However, we can see that's also kind of nebulous. But okay, so is that something to worry about? Well, if you just think biological humans should remain in charge of all important decisions forever, then I agree, the development of AI seems like kind of a problem for that. But in fact, other things also seem like kind of a problem for that, I just don't expect to generally be true. Like a million years from now, if even if you don't develop AI, biological humans, the way we recognize them today are still making all the important decisions and they have something like the culture that we would recognize from ourselves, so they are pretty surprised by that.

1:45:16So I think there's a, I think Robin Henson has again talked about this where he said, a bunch of the things that people fear about AI are just things they fear about change and fast change. So the thing that's different is that AI has a prospect of accelerating a bunch of this change so that it happens in a narrower period. Sorry, no, I think there's like not the, I think it's not just the kind of change that would have happened from let's say genetically modifying humans and blah, blah, blah, is instead happening in a compressed amount of time. I think the worry comes more from like, it's not just that change, compressed, it's a very different vector of change.

1:45:53Yeah, but what is the argument for that? I have never seen a good argument for this. You should expect a bunch of change if you accelerate just human change as well. Like you might expect different values to become much more dominant. You might expect people that don't discount the future as much to be much more influential because they save more and they make good investments but give some more control. Higher risk tolerance. Higher risk tolerance, like because they are more willing to make kind of bets that maximize expected value. And so get much more influence. So just generically, the accelerating human change would also result in a lot of things being lost that you might care about.

1:46:35And so I do think there's, I think the argument is that like maybe the speed of the change determines what fraction of the existing population or stakeholders or whatever have some causal influence on the future. And maybe the thing you care about is like, look, there's gonna be change but it's not just gonna be like one guy presses a button. That's like the software singularity extreme. Right. And it's more like, you know, like over time norms change and so forth. So if you're looking at the software singularity, a picture I agree that picture looks different. And again, I'm coming back to this because obviously Daniel and maybe Scott to some extent, they probably have this view that the software on these singularities is like more plausible.

1:47:19And then one person could just be in a position to like, we could end up in a situation where they're idiosyncratic preferences or something end up in a very financial. Yeah. Even in the, I agree that is, like that makes the situation look different from if you just like have this broader process of automation. But even in that world, like I think a lot of people have this view about things like value lock in, or like they think this moment is like a pivotal moment in history. And then we're just gonna have like someone is gonna get this AI, which is very powerful because say, if it's software singularity, and then they're just gonna like lock in some values.

1:47:56And then those values are just gonna be stable for like millions of years. And I think that just looks like very unlike anything that has happened in the past. So I'm kind of confused why people think it's very plausible. I think people have the argument that, like they see the future again in my view in sort of far mode. They think there's gonna be one AI. It's gonna have some kind of utility function. That utility function is gonna be very stable over time. So it's not gonna change. There won't be this messiness of like lack of coordination between different AI's or different like over time values drifting for various reasons, maybe because they become less functional or in an environment, maybe because of other reasons.

1:48:38And so they just don't imagine that. They say, well, I mean, utility functions, we can preserve them forever. We have the technology to do that. So it's just gonna happen. And I'm like, well, that seems like such a weak argument to me. I'm not sure. Actually, if you look at just like, if the, often the idea is because this is like digital, you can preserve the information better and copy it with higher fidelity and so on. But actually, if you look, even if you look at just like information on the internet, you have this thing called link rot, which happens very quickly. And actually information that's digital isn't preserved for very long at all.

1:49:15Or the point that Matthew was making is that also this, like the fact that the information is digital has like a, not maybe like to, but at least been associated with faster cultural change. Cultural change, exactly. And basically, the technology changes can create incentives for cultural change. Just as they make preserve. That's right. I think there's two key arguments that I've heard. One is that we will soon reach something called technological maturity. And once you, one of the key ways in which society has been changing recently is that it's not, like maybe actually this culture would have changed even more.

1:49:49Actually, no, no, I think this argument is wrong that you're making because we do know that language should actually change a lot more. Like we can read everything that was written after like 1800s when literacy became more common. But it's actually even just go back a couple hundred years after that and you're reading Old English and it's hard to understand. And that is a result of literacy and the codification of language. That information was better preserved. What about other kinds of cultural practices? But I think the argument would be that maybe they would have actually changed more. If that change was the result of technological change in general, not the result of information being digitized.

1:50:24Maybe that culture would have actually changed more if information wasn't as well preserved or technology continued to proceed. And the argument is in the future, we're going to reach some point at which like you've done all the tech, like ideas have gotten way too hard to find and you need like entire galaxies worth of, like you need to make a certain that's the size of a galaxy to progress physics and inch forward. And at that point, there's this like growth in technology just churning over civilization goes away. And then you just have the digital thing, which does mean that like a lock in is more plausible.

1:51:00So the technological maturity thing, I agree that results in a slow down in change and growth and so on and certain things might get more locked in relative to what proceeded it. But then what do we do today about that? What like what could you do to have a kind of positive impact by our lights? And so asking that question, Robyn Hansen had this question of what could someone do in the 1500s to have a positive impact on the world today, from their point of view, knowing all they knew back then. I think this question is even worse than that because I think the amount of change that happens between today and technological maturity is just orders of magnitude greater than whatever change happened between the 1500s and today.

1:51:44So it's an even worse position than someone in the 1500s thinking about what they could do to have an impact, positive impact in expectation, and like predictably positive today. And so I think it's just pretty hopeless. I don't know if we could do anything or find any candidate set of actions that would just make things better your post -locking. Or I mean, that's assuming lock -in is living on a habit which is not right. In the 1700s, a bunch of British, abolitionist were making the case against slavery. And I think the world has, we could just live in a slave system. Like I don't think there's any in principle reason why we couldn't have been a slave society to this day or the more the world couldn't have slavery.

1:52:24And I think what happened is just like the convincing of British people that slavery is wrong, the British empire put all its might into abolishing slavery and making that a norm. I think another example is Christianity and the fact that like Jesus has these ideals, you're like, I talk about these ideals, I think like the world is a more Christian place. Wait, it is a more Christian place, sure. And also has like more of the kind of place. Like I'm not saying Jesus Christ would endorse every single thing that happens in the world today. I'm just saying he endorses this timeline more than one in which he doesn't exist and doesn't preach at all.

1:52:59I don't know actually. I mean, I'm not sure if that's true. This seems like a hard question. But it's a, I think like a some from the Christian perspective, favorable cultural development. And you don't know the kind of factual. I agree that is always true. I just think like the world does have people who like read the Bible and are like, I'm inspired by these ideals to do certain things. And it just seems like that's more likely to relate to like, that is what I would call like a legacy effect or something. I mean, you can say the same thing about languages, like some cultures might just become more prominent and their languages might be spoken more.

1:53:28There are some symbols might become more prominent. But then there are things like, like how do cities look and how do cars look and what do people spend was there a time doing in their day and what do they spend their money on? And like, and those questions seem much more determined by the, like by how your values change as circumstances change. That might be true. But I am in the position with regards to the future where I'm like, I start a lot of things to be different and I'm okay with them being different. I care much more about the equivalent of slavery, which in this case is literally slavery.

1:54:03Which is, like I just put a fine point on it. Like the thing I really care about is, there's going to be trillions of digital beings. I wanted to be the case that they're not like tortured and put into conditions in which they don't want to work or whatever, or like I don't want galaxies worth of suffering. Okay, but that seems closer to like British abolition as being like, let's put our empires might against sliding slavery. I agree, but I would distinguish between the case of Christianity and the case of end of slavery. Because I think the end of slavery, like I agree, you can imagine us aside, like technologically, it's like feasible to have slavery.

1:54:36But like I think that's not the relevant thing which brought it to an end. The relevant thing is that the change in values associated with industrial revolution made it so that slavery just became like an inefficient thing to sustain in a bunch of ways. And a lot of countries at different times like stays out different things you could call slavery. So for example, Russia abolished Serfdom in the 1860s. They were not under British pressure to do so. Like Britain couldn't force Russia to do that. So they just did that on their own. There were various ways in which people in Europe were like tied to their land and they couldn't move.

1:55:17They couldn't go somewhere else. Those movement restrictions were lifted because they were inefficient. They were ways in which it used to be like the kind of labor that needed to be done in the colonies to grow sugar or to grow various crops. It was very hard labor. It was not the kind of thing that probably you could have paid people to do because they just wouldn't want to do it because the health hazards and so on were very great, which is why they needed people to force people to do that. And that kind of work over time became less prevalence in the economy. So again, that reduces the economic incentives to do it.

1:55:56I agree you could still do it. But I would emphasize that it's like the way you're painting the kind of fractures like, oh, but then in that world they would have just like phased out the remnants of slavery. But like there's a lot of historical examples where there's not necessarily only hard labor. Like Roman slavery, yes, it was difference. And I never even started about it recently. The episode hasn't come out. But he wrote a book about like the scope, I think it was like 20 % of Roman people under Roman control were slaves. And this was not just agricultural slavery. This was like every like, and his point was that it was this division of like the maturity of the Roman economy is what led to this level of slavery.

1:56:43Because the reason slavery collapsed in Europe after the fall of the Roman Empire was because the economy just lost a lot of complexity. I'm not sure if I would say that slavery collapsed. I mean, I think this depends on what you mean by slavery. I mean, you know, a lot of ways people who infudel Europe were like, but his point is that actually Serfen was not the descendant institution for Roman slavery. No, I actually was not descendant, but like the, in fact, this sort of point I'm trying to make is that values that exist at a given time, like what the values of we will have on 300 years or like from the perspective of someone 1000 years ago, what values people are gonna have in 1000 years?

1:57:18Those questions are much more determined by the technological and economic and social environment that's gonna be there in 1000 years. Which values are gonna be functional, which sides which values end up being more competitive and being more influential so that other people like adopt their values. And it depends much less on the individual actions taken by people in 1000 years ago. So I would say that the abolitionist thing, it's not the cause of why slavery came to an end. And slavery comes to an end also because people's own, like people just have natural preferences that I think are suppressed in various ways during the agricultural era, where it's more efficient to have settled societies in cities which are fairly authoritarian and don't allow for that much freedom.

1:58:18And you're in this melphusian world where people have very low wages, perhaps compared to what they enjoyed in the hunter -gatherer era. So it's just a different economic period. And I think people were not, they didn't evolve to have the values that would be functional in that era. So what happened is that there to be a lot of cultural assimilation where people have to adopt different values. And in the industrial revolution, people become also very wealthy compared to what they used to be. And that I think leads to different aspects of people values being expressed. Like people just have put a huge amount of value on equality.

1:58:52Like it's always been the case. But I think when it is sufficiently functional for that to be suppressed, they are keepable of suppressing it. I mean, if that's a story then this puts all the more reason, this makes alignment all the more important or like the value alignment all the more important because then you're like, oh, the AI's become wealthy enough, they actually will make a concerted effort to make sure the future looks more like the utility function we put into them. Which I think you have been under emphasizing. No, I'm not under emphasizing that. I think like there are, what I would say is there are certain things that are like path -dependent in history.

1:59:26So I should add, if someone had done something different, like something I've gone differently a thousand years ago, then today in some respects would look different. I think for example, which languages are spoken across which boundaries or like which religions people have? Or like which, like those kinds of fashion maybe, to some extent, they're not entirely. Like those things are more path -dependent, but then there are things that are not as path -dependent. So for example, if some empire, like if the Mongols had been more successful and they somehow, I don't know how realistic it is, but they became very authoritarian and had slavery everywhere.

1:59:58Would that have actually led to slavery being a much more enduring institution a thousand years later? That seems not true to me. Like the forces that led to the end of slavery seemed like they were not contingent forces. They seemed like deeper forces than that. And if you're saying, well, if we aligned AAS today to some bad set of values, then that could affect the future in some ways, which are more fragile, that seems plausible. But I'm not sure how much of the things you care about the future and how much, like the ways in which you expect the future to get worse, you actually have a lot of leverage on at the present moment.

2:00:40I mean, another example here might be factory farming where you could say like, oh, it's not like I was having better values over time, let's suffering going down. In fact, you're suffering might have gone up because a lot of people say that. The incentives that are, that led to factory farming emerging, or just like, probably when factory farming comes to an end, it will be because the incentives start doing that right. But suppose I care about making sure the digital equivalent of factory farming doesn't happen, where it may be more efficient, maybe in the all else being equal, it's just more economically efficient to have suffering minds doing labor for you than non -suffering minds because of the, because of the intermediary benefits of suffering.

2:01:24So that's something like that, right? What would you say to somebody like me, where I'm like, I really want that not to happen, I don't want the like home filled with suffering workers or whatever? Is it just like, well, give up, because this is the way economic history is, no, I don't think you should give up. It's more like, it's hard to anticipate the consequences of their actions in the very distant future. So like, I would just recommend that you should just discount the future, not for like a moral reason, not because like the future is worthless or something, but because it's just very hard to anticipate the effects of their actions.

2:02:00In the near term, I think there are things you can do that seem like they would be beneficial. Like for example, you could try to align your present AI systems to value the things that you're talking about, like they should value happiness and they should just like suffering or something. You might want to support like political solutions that would like basically, you might want to build up the capacity so that in the future, if you notice something like this happening, then we might have some ability to intervene. Like maybe you would think about the prospect of well eventually we're going to maybe colonize that are stars and like the civilization might become very large in communication delays might be very long between different places.

2:02:43And in that case, competitive pressures between different local cultures might become much stronger because there's like it's harder to centrally coordinate. That's right. And so in that world, you might expect competition to take over in a stronger way. And if you think the result of that is going to be a lot of suffering, maybe you would try to stop that. Again, I think at this point, it's not, it's very far from obvious that trying to say limit competition is actually a good idea. I would probably think it's a bad idea. But maybe in the future, we will receive some information and we'll be like, oh, we were wrong.

2:03:15Actually, we should stop this. And then maybe you want to have the capacity so that we could make that decision. But that's very hard. That's the never -listen how do you build that up? Well, I don't know. I mean, you would need to, like that's the kind of thing I would be trying to do. Yeah, I think the overall takeaway I take from the way that I think about it. And I guess we think about it as be more humble in what you think you can achieve. And just focus on the near term, not because it's more morally important than the longer term, but just because it's much easier to have a predictably positive impact on that.

2:03:49Well, one thing I noticed over the last few weeks of thinking about these bigger -fricture topics and interviewing Daniel and Scott and then you too, is how often I've changed my mind about everything from the smallest questions about when AI will arrive. It's funny that that's the small question in the grand scheme of things. To whether there will be an intelligence explosion or whether there will be an R &D explosion to whether there will be a growth or how to think about that. And if you're in a position where you are just like, incredibly epistemically uncertain about what's going to happen, I think it's important to just like directly acknowledge that this is instead of just, instead of becoming super certain about your next conclusion, just being like, let me just, at least for my perspective, I'm just, let me just take a step back.

2:04:38I'm not sure what's going on here. And I think a lot more people should be from that perspective unless you've had the same opinion about AI for many years, in which case I have other questions for you about why that's the case. And in other situations, I mean, generally how we as a society deal with topics on which we are this uncertain is just to have freedom, decentralization, both decentralized knowledge and decentralized decision making, take the reins and not to do super high volatility centralized moves. Like, hey, less nationalized, so we can make sure that we can make the software only singularities aligned or not to do make moves that are just incredibly contingent on one worldview that are brittle under other considerations.

2:05:27And that's become a much more salient part of my worldview. I think just classical liberalism is the way we deal with being this epistemically uncertain. And I think we should be more uncertain than we've ever been in history, as opposed to many other people who seem to be more certain than they are about other sort of more mundane topics. Yeah, I think it's very hard to predict what happens because of this acceleration basically means that you find it much harder to predict what the world might be in 10 years' time. I think these questions are also just very difficult and we don't have very strong empirical evidence.

2:06:06And then there's a lot of this kind of disagreement that exists. Like I would say that it's important to, like a lot of cases and a lot of situations, it's much more important to be maintain flexibility and ability to adapt to new circumstances, new information than it is to get a specific plan that's gonna be correct and that's gonna very detail then has a lot of specific policy recommendations and things that you should do. So that's actually also the thing that I would recommend. If I wanna make the transition to AI in this period of explosive growth go better, I would just prefer it to be in general had higher quality institutions.

2:06:47But I am much less bullish on someone sitting down today and working out what will this intelligence explosion or explosive growth be like, what should we do? I think plans that you work out today are not gonna be that useful when the events are actually occurred because you're gonna let go of so much stuff that you're gonna update on so many questions that these plans are just gonna become obsolete. It's one thing you could do is you could look at, say, the history of war planning and how successful war planning has been for actually anticipating what actually happens when the war actually happened.

2:07:22So for one example, I think I might have mentioned this in like off the record at some point. But before the second war happens, obviously people saw that they were all these new technologies, like tanks and airplanes and so on, which were now like they existed more or more on, but in a much more primitive setting. So they were wondering what is gonna be the impact of these technologies now that we have in them in much greater scale. And the British government had estimates of how many casualties there will be from aerial bombardment in the first few weeks of the second war. And they expected hundreds of thousands of that casualties basically in like two weeks, three weeks after the war begins.

2:07:59So the idea was that air bombing is basically this unstoppable force. All the major urban centers are gonna get bombed. Tons of people will die. So basically we can't have a war because if there's a war then it will be a disaster because we will have this air bombardment. But later it turned out that that was totally wrong. In fact, in all Britain, there were fewer casualties from air bombing in the entire sort of six years of the Second World War. Then the British government expected in the first few weeks of the war. Like they had less casualties in six years than I expected it in like few weeks.

2:08:36So why did they get it wrong? Well, I mean, there are lots of boring practical reasons. Like for example, it turned out to be really infeasible to bomb, especially early on to bomb cities in daytime because your aircraft would just get shot down. But then if you tried to bomb at nighttime, then your bombing was really imprecise. And only a very small fraction of it actually hit. And then people also underestimated the extent to which people on the ground could like firefighters and so on could just sort of go around the city and then put out fires from bombs that were falling on the structures.

2:09:06They overestimated the amount of economic damage that it would do. They underestimated how economically costly it would be. Like basically you're sending these aircraft and then they're getting shot down. Well, an aircraft is very expensive. So in the end, what turned out is when the Allies started bombing Germany, they were like for each dollar of capital they were destroying in Germany. They were spending like $45 on the aircraft and fuel and training of the pilots and so on that they were sending in missions. And the casual trade was very high, which later got covered up by the government because they didn't want people to worry about.

2:09:41Yeah. So that is a kind of situation where all the planning that you would have done in advance predicated on this assumption of like air bombing is gonna be this like nuclear weapons light. Basically, like extremely destructive. There's gonna be some aspect to which. I mean, it was, right? Like 84 ,000 people died in one night of fire bombing in Tokyo. Like Germany, like large fractions of their... But that was over the period of six years. Right. But there were like single fire bombing attacks. I mean, it was a case that during the end of World War II, when they were looking for the place to launch the atomic bombs.

2:10:20That's right. They just had to go through like a dozen cities because they were just wanna be worth nuking them because they're already destroyed by the fire bombing. That's right. But the level of destruction, if you look at the level of destruction, it was expected within the space of a few weeks. And then this level of destruction took many years. So there was like a two -order magnitude mismatch or something like that, which was pretty huge. Yeah. So that affected the way people think about it. Right. An important underlying theme of much of what we have discussed is like how powerful just reasoning about things is to making progress about what specific plans you wanna make, to prepare and make this transition to advance the I go, well, in our view is, well, it's actually quite hard and you need to make contact with the actual world in order to inform most of your beliefs about what actually happens.

2:11:13And so it's somewhat futile to think, to do a lot of war gaming and figure out how AI might go and what we can do today to make that go a lot better because a lot of the policies you might come up with might just look fairly silly. And I think there's in the thinking about how AI actually has this impact. Again, people think, oh, just AI reasoning about doing science and doing R &D just has this drastic impact on the overall economy or technology. And our view as well, actually, again, making contact with the real world and getting a lot of data from experiments and from deployment and so on. It's just very important.

2:11:53So I think there is this underlying kind of latent variable which explains some of this disagreement both on the policy prescriptions and about the extent to which we should be humble versus ambitious about what we ought to do today, as well as for thinking about the mechanism through which AI has this impact. And this underlying latent thing is like, what is the power of reason? Like how much can we reason about what might happen, how much can reasoning in general figure things out about the world and about technology? And so that is a kind of core underlying disagreement here. Yeah, yeah. I do wanna ask, you say in your announcement, we want to accelerate this broad automation of labor as fast as possible.

2:12:38As you know, many people think it's a bad idea to accelerate this, the broad automation of labor and AGR and everything that's involved there. Why do you think this is good? So the argument for why it's good is that we're going to have this enormous increase on economic growth which is going to mean like enormous amounts of wealth and incredible new products that you can't even imagine and like healthcare or whatever. And like the quality of life of the typical person is probably going to go up a lot. Early on probably also their wages are gonna go up because the AI systems are gonna be automating things that are complimentary to their work or like it's gonna be automating part of their work and then you'll be doing the rest and then you'll be getting paid much more on that.

2:13:21And in the long term, eventually we do extra wages to fall just because of arbitrage with the AI's. But by that point, we think humans will own enormous amounts of capital and there will also be ways in which even the people who don't own capital we think are just gonna be much better off than there today. Like I think it's just hard to express in words the amount of wealth and increased variety of products that we would get in this role. It will be probably more than a difference between like 1800 and today. So if you imagine that difference, it's like such a huge difference. And I would imagine like two times, three times, whatever.

2:13:57The standard argument against this is, why does this speed to get there matter so much? Because especially if the trade off against the speed is the probability that this transition is achieved successfully in a way that benefits humans. I mean, it's unclear that this trades off against the probability of it being achieved successfully or something like that. There might be an alignment tax. I mean, maybe. I don't like, you can also just do the calculation of how much a year's worth of delay costs for current people. Like, you know, this is this enormous amount of utility that people are able to enjoy.

2:14:35And that gets brought forward by a year or pushback by a year if you delay things by a year. And how much is this worth? Well, you know, you can, you can look at simple models of how concave people's utility functions are and do some calculations. And maybe that's worth on the order of tens of trillions of dollars per year in consumption. That is roughly the amount consumers might be willing to defer in order to get, you know, bring forward the date of automation one year. In absolute terms, it's high in relative terms, relative to if you did think it was going to nurture the probability when we're another of building systems that are aligned and so forth.

2:15:14Then it's like just so small compared to all of the future. I agree. So like, there are a couple of things here. First of all, I think the way you think about this matter. So first of all, we don't actually think that it's clear whether speeding things up or slowing things down actually makes, like, do me outcome more or less likely. Like, I think that's just a question that doesn't seem obvious to us. Like, we don't, like, partly because of our views on the software R &D side, we don't really believe that if you just pause and then you, like, do research for 20 years at a fixed level of compute scale, that you're actually going to make that much progress on relevant questions on alignment or something.

2:15:52Like, I think, like, imagine you were trying to make progress on alignment in 2016 with the compute budget of 2016. And the, like, well, you would have gotten nowhere, basically. Like, you would have discovered none of the things that people have today discovered and that turned out to be useful. And I think if you pause today, then we will be in a very similar position in 10 years, right? Like, we would have not made a bunch of discoveries. So the scaling is just really important to make progress on alignment in our view. And then there's a separate question of how long term should you be in a very different sense?

2:16:26So there's a moral sense, or like, how much should you actually care about people who are alive today, as opposed to people who are not yet born as some moral question? And there's also a practical question of, as we discuss, how certain can you be about the impact your present actions are actually going to have on the future? Okay, maybe you think it really doesn't matter whether you slow things down right now or you speed things up right now. But is there some story about why speaking them up from the alignment perspective actually helped? It's good to have that extra progress right now rather than later on, or is it just that, well, if it doesn't make a difference either way, then it's better to just get that extra year of people not dying and having cancer cures and so forth.

2:17:05I think I would say the second, but like, well, it's just important to understand the value of that, right? Even in purely economic terms, like imagine that you would be, like each year of delay might cause like maybe a hundred million people, maybe more, maybe 150, 200 million people who are alive today to end up dying, right? So the, even in purely economic terms, the value of a statistical life is like pretty enormous, especially in Western countries, so it's like sometimes people use the numbers as high as $10 million for a single life. So imagine you do like $10 million times 100 million people, that's like a huge number, right?

2:17:47So like, I think that is just so enormous that unless you're just, so I think for you to think that speeding himself is a bad idea, you have to first be like just have this long -term view where you look at the long -round future, you think your actions today have high enough leverage that you can predictably affect the direction of long -range. In this case, it's kind of different because you're not saying I'm going to affect what some Emperor a thousand years from now does, like somebody in the year zero would have to do to be a long -termist. In this case, you just think there's this incredibly important inflection point that's coming up, and you just need to have influence over that crucial period of explosive growth of intelligence or solution or something.

2:18:32So I think it is a much more practicable prospect than, so I agree in relative terms, so like in relative terms, I agree the present moment has like, is a moment of higher leverage, and you can expect to have more influence. I just think in absolute terms, the amount of influence you can have is still quite low. So it might be orders of magnitude greater than it would have been in 2000 years ago, and still be quite low. And again, I think there's this like difference in opinion about how broad and diffuse this transformation ends up being versus how concentrated within a specific labs where the very idiosyncratic decisions made by that lab will end up having very large impact.

2:19:10If you think those developments will be very concentrated, then you think the leverage is especially great. And so then you might be especially excited about having the ability to influence how that transition goes. But our view is very much that this transition happens very diffusing by way of many, many organizations and companies doing things, and for those actions to be determined a bunch by economic forces rather than have idiosyncratic preferences on the part of labs or these kind of decisions that have these kind of founder effects that last for very long. Right. OK, let's go through some of the objections to explosive growth, which is most people are actually more conservative, not more aggressive about the forecasts you have.

2:19:57So obviously one of the people who has articulated their disagreements with your view is Tyler Cowan. He made an interesting point when we did the podcast together, and he said, most of South Saharan Africa still does not have reliable clean water. The intelligence required for that is not scarce. We cannot so readily do it. We are more in that position that we might like to think along other variables. I mean, we agree with this. Like I think intelligence isn't the bottleneck that's holding back technological progress or economic growth. Right. It's like many other things. And so I think that this is very much consistent with our view that scaling up your over economy, accumulating capital, accumulating human capital, having all these factor scales is even consistent with what I was saying earlier that I was pointing out this like, oh, like good management and my good policies.

2:20:51And those just contribute to TFV. And they can be bottlenecks on my early. But like right now we could just plug in play our better management into South Saharan Africa. And this is hard. I don't think so. OK, so that's what I was maybe I should have said. One could theoretically imagine plugging in play with I could imagine many things. But we cannot so readily do it because of it's like hard to articulate why. And it wouldn't be so easy to do in just capital or labor. Why not think that the rest of the world will be in this position with regards to the advances that AI will make possible? I mean, if the AI advances are like the kind of geniuses in a data center, then I agree that that might be bottleneck by the rest of the economy not scaling up and being able to accumulate the relevant capital to make those changes feasible.

2:21:47So I kind of agree with this picture. And I think this is like, you know, an objection to the geniuses in a data center type view. Yeah. And like I buy basically this. And also the fact that like it's also plausible you're going to have the technology. But then some people are not going to want to deploy it or some people are going to have norms and laws and cultural things that are going to make it so that AI is not able to widely deploy in their economy or not as widely deployed as it otherwise might be. And that is going to make those countries or the size just slower. That's a, I mean, that's like some countries will be growing faster just like Britain and the Netherlands were sort of the leaders in the industrial revolution.

2:22:25They were the first countries to start experiencing rapid growth. And then other countries, even Europe sort of had to come from behind. Well, again, I just think we expect the same thing to be true for AI. And I mean, the reason that happened was exactly because of these kinds of reasons where those countries that I culture or governance systems or whatever, we should just worse than bottleneck the deployments and scaling of the new technologies and ideas, it seems very plausible. But you're seeing as long as there's one jurisdiction. But then again, you also previously emphasized the need and the need to integrate with the rest of the global economy and the human economy.

2:23:04So there's not that kind of thing. It doesn't often require a cultural homogeneity. Like we can, we trade with countries, like the US trades with China quite a lot actually and there's like a bunch of the agreements. But what if the US is like, I don't like the UAE is doing an explosive growth with AI. We're just going to like embargo them. That seems plausible. And then what is, would that not prevent explosive growth? I mean, like, I think that would be plausible at the point at which it's revealing a lot about the capabilities and the power of AI. And you should also think that that creates both an incentive to embargo, but also an incentive to adopt the very similar styles of governing that enable AI to be able to produce a lot of value.

2:23:48What do you make of this? I think people interpret explosive growth from an arms race perspective. And that's often why they think in terms of public private partnerships for the labs themselves. But just this idea that you have, that geniuses in the middle of the center, they like, you can have them come up with the mosquito drone swarms. And then those drone swarms will, you know, like if China gets to those swarms earlier, I mean, even within your perspective, where it's like not, is this the result of your whole economy being advanced enough that you can produce mosquito drone swarms? You being six months ahead means that you could decisively when, Ed, does it?

2:24:24I don't know. Maybe you being like a year ahead and an explosive growth means you could decisively win a war against China or China could win a war against you. So would that lead to an arms race dynamic? I mean, I think it would, to some extent, but I'm not sure if I would expect that like a year of lead to be enough to take a risk. Because if you go to war with China, I mean, for example, if the US went to war with, if you replace China today with China from 1999, or if you replace Russia today, with Russia from like 1987 or 1980, it's possible that there ICBM and whatever technology is already enough, like it's already enough to make, like have very strong returns.

2:25:04So maybe even that lead, a technological lead is not sufficient so that you would feel comfortable going to war. So that seems possible. Yeah. And actually this relates to a point that Guren was making, which is, he was like, okay, this is going to be a much more unstable period than the industrial revolution. Even though industrial revolution saw the, saw many countries gain rapid increases in their capabilities, because this is just like, within this span, if you're going to centuries where the progress compressed within a decade, one country gets to like ballistic missiles first, then the other country gets to railroads first and so forth.

2:25:45But if you have this more integrated perspective about what it takes to get to ballistic missiles into railroads, then you might think, no, basically this isn't some orthogonal vector, it just like you're just like turning on the tech tree further and further. Yeah, I mean, for what's worth, I do think, like it's possible if you have it just happen in a few countries which are relatively large and have enough land or something, like those countries could just, like they would be starting from a lower base compared to the rest of the world. So they wouldn't need to catch up to some extent. So like if they are just going to sort of grow internally and they're not going to depend on the external supply chains.

2:26:21But like that doesn't seem like something that's impossible to me. Yeah, some countries could do it. But it would just be like more difficult, but in this setting, if some countries are like a significant policy advantage over the rest of the world, then they start growing first and then they won't necessarily have a way to get other countries to adopt their norms and culture. So in that case, they might just, it might be more efficient for them to do the growth locally, right? So that's why I was saying the growth differential will probably be determined by like regulatory jurisdiction boundaries more than anything else.

2:26:53Like I'm not saying, say the US by itself, it had AI, but it couldn't get the rest of the world to adopt AI. I think that would still be sufficient for, I'm not supposed to grow. How where should we be about the fact that China today just has, because industrialized, relatively recently just has more industrial capacity and know how and all the other things are learning by doing and so forth. If we buy your model of how technology progresses with or without AI, how are we just under -estimating China? Because we have this perspective that like, what fraction of your GDP you're spending on research as it matters?

2:27:33But in fact, it's the kind of thing where like I've got all the factories in my backyard and I know how they work and I can go buy a component whenever I want. I don't think people are necessarily underestimating China. I mean, it depends on who you're looking at. But it seems like the discussion of China is, just this very big discussion when in these AI circles, right? And so people are like very much appreciating the power and the potential threat that China poses. But I think the key thing is not just like, the scale in terms of pure number of people or like number of firms or something, but the scale of the overall economy, which is just measured in how much is being produced in terms of dollars.

2:28:12But when they're, you know, the US is ahead. But we're not expecting all this explosive growth to come for financial services. We're expecting it to start from a base of industrial technology and industrial capacity. Even though financial services can be important if you want to scale very big projects. Yeah, financial services are very important for raising funding and getting investments in data centers. But if I understood you correctly, it just seems like, man, you know how to do all the, like you know how to build the robot factories and so forth. That like know how, which in your view is still crucial to technology growth and just general economic growth is lacking.

2:28:50And you might have more advanced financial services, but like it seems like the more you take your resuriously the more it seems like the, the having the shun zen locally matters a lot. I mean, relative to like what's starting point? Like I think people already appreciate the China is very important. And then I agree that there are some domains where China is leading, but then there are very many domains in which the US is leading or the US and its allies where you know countries that are producing relevant inputs REI that the US has access to, but China doesn't. So I think the US is just like ahead on the many dimensions.

2:29:25There's some that China is ahead or at least very close. So I don't think this should cause you to update very strongly in favor of China being a much bigger deal at least depending on where you start. I think people already think China is a big deal. Like this is the big underlying thing here. Like if two words, it's just very dismissive of China then maybe this would be a reason to update. But I get your argument that thinking about the economy wide acceleration is more important than focusing on the IQ of the smartest AI. But at the same time, do you believe in the idea of superhuman intelligence?

2:30:03Is that a coherent concept? In the way that you don't necessarily stop at human level go playing, you just go way beyond it in Elon's core, what we get to systems that are like that with respect to the broader range of human abilities. And maybe that doesn't mean they become God because there's other ASIs in the world. But you know what I mean? Will there be systems with such superhuman capabilities? Yeah, I mean, I do expect that. I think there's a question of how useful is this concept for thinking about this transition to a world with much more advanced AI? And I don't find this like a particularly meaning for a helpful concept.

2:30:41Like I think people introduce some of these notions that on the surface seem useful, but then actually when you delve into them, it's like very vague and kind of unclear what you're supposed to make of this. You have this notion of AGI, which distinguishes from narrow AI in the sense that it's much more general and maybe you know can do everything that a human can do on average. I mean, AI systems have these very jagged profiles of capability, so you have to somehow take some notion of average capabilities and what exactly does that mean? It just feels really unclear. And then you have this notion of ASI, which is AGI in the sense that it's very general, but then it's also better at humans than on every task.

2:31:22And you know, is this a meaningful concept? I guess it's coherent. I think this is not a super useful concept because I prefer just thinking about, you know, what actually happens in the world and you could have a drastic acceleration without having an AI system that can do everything better than humans can do. I guess you could have no acceleration when you have an ASI that is better at humans than you know, better than humans at everything, but it's just very expensive or very slow or something. So I don't find that particularly meaningful or useful. I just prefer thinking about, you know, the overall effects on the world and what AI systems are capable of producing those types of effects.

2:32:06Yeah, I mean, one intuition pump here is, compare John Monnoim and versus a human flight from the standard distribution. If you add in a million John Monnoim and into the world, what would the impact on growth be as compared to just adding a million people from the normal distribution? Well, I agree if it would be much clearer. But then like, because of more of X paradox type arguments that you made earlier that evolution is not necessarily optimized us for that long, along the kind of spectrum on which John Monnoim and is distinguished from the average human. And given the fact that already within this deviation, you have this much greater economic impact, why not focus on going, optimizing on this thing that evolution is not optimized that hard on further?

2:32:51I don't think we shouldn't focus on that. I think it's, but what I would say is, for example, if you're thinking about the capabilities of goal playing AI's, then the concept of a super human go AI, yeah, you can say, that could be a meaningful concept. But if you're developing the AI, it's not a very useful concept. If you just look at the scaling course, it just goes up and there's some human level somewhere, but the human level is not privileged in any sense. So the question is, is it a useful thing to be thinking about? The answer is probably not, it depends on what you care about. So I'm not saying we shouldn't focus on trying to make the system smarter than humans are.

2:33:29Like I think that's a good thing to focus on. Yeah, I guess I'm trying to understand whether we will stand in relation to the AI's of 2100, that human -standard relationship to other primates. Is that the right mental model we should have or is it going to be a much greater familiarity with their cognitive horizons? I mean, I think AI systems will be very diverse. And so it's not super meaningful to ask. Something about this very diverse range of systems and where we stand in relation to them. Will we be able to cognitively access the kinds of considerations they can take on board? Humans are diverse, but no chimp is going to be able to understand this argument in the way that another human might be able to, right?

2:34:13So I'm just like, if I'm trying to think about my place or a human's place in the world of the future, I think it is a relevant concept of, is it just that the economy has grown a lot and there's much more labor? Or are there beings who are in this crucial way super intelligent? I mean, there will be many things that we just will fill to understand. And to some extent, there are many things today that people don't understand about how the world works and how certain things are made. And then, how important is it for us to have access? Or in principle, be able to access those considerations. And I think it's not clear to me that that's particularly important that any individual human should be able to access all the relevant considerations that produce some outcome.

2:34:59Like that just seems like overkill. Like, why do you need that to happen? I think it would be nice in some sense, but I think if you want to have a very sophisticated world where you have very advanced technology, those things will just not be accessible to you. And then, so you have this trade off through accessibility and maybe how advanced the world is. And for my point of view, I'd much rather live in a world which has very advanced technology, has a lot of products that I'm able to enjoy and a lot of inventions that I can improve my life with. If that means that I just don't understand them.

2:35:38I mean, I think this is a very simple trade that I like are very willing to make. Okay, so let's get back to objections to explosive growth. We discussed a couple already. Here's another, which is more a question than an objection. Where is all this extra output going? Like, who is consuming it? If the economy is 100x bigger in a matter of a decade or something, like to what end? So first of all, I think even if you view that along what you might call the, the intensive margin, in the sense that you just have more of the products you have today, I think there is just a lot of, like there will be a lot of appetite for that.

2:36:18Maybe not quite 100x, but like that might start hitting some emission turns. For current GDP per capita on average in the world is 10k years or something. And there are people who enjoy millions of dollars. And so there's a gap between what people enjoy and don't seem to be super diminished in terms of marginal utility. And so there's a big room, there's a lot of room on just purely the intensive margin of just consuming the things we consume today. But more, and then there is this maybe much more important dimension along which we will expand, which is... Productivity. Yeah, extensive margin of what is the scope of things that you're consuming?

2:36:58And if you look at something like the industrial evolution, that seemed to have been the main dimension along which we kind of expanded to consume more. There's just on any kind of sector that you care about, transportation, medicine, entertainment and food. There's just this massive expansion in terms of a variety of things that we're able to consume that is enabled by new technology or new trade routes or new methods of producing things. And so that is, I think the really the key thing that we will see come along with this kind of expansion and consumption. Another point that Tyler makes is that there will be some mixture of BAMUELCOS disease where you're bottlenecked by the lowest growing thing, which grows in proportion.

2:37:50The fastest productivity things basically diminish their own sharing. Now put you in. That's right. I mean, like we totally agree with that. I would say that that's just like a kind of qualitative consideration. It doesn't... itself, it isn't self -sufficient to make a prediction about what growth rates are permitted given these BAMUELCOS effects versus not. It's just like a qualitative consideration and then you might need to make additional assumptions to be able to make a quantitative prediction. So I think it's a little bit... So the like, the commissing version of this argument would be if you did the same thing that we were doing earlier with the software and the singularity argument where we were pointing to...

2:38:34essentially the same rejection where there are multiple things that can bottleneck progress. So I would be much more convinced if someone pointed to an explicit thing, they would be like, here, like healthcare is this very important thing. And why should we expect AI to make that better? That doesn't seem like that would get better because of AI. So that maybe healthcare just becomes a big part of the economy and then that bottleneck. So if there was some specific sector... Maybe the argument is that if there's even one... No, if there's one though, if that's a small part of the economy then you could just still get a lot of growth.

2:39:04You just automate everything else. And that is going to produce a lot of growth. So it has to like, quantitatively work out. And so you actually have to be quantitatively specific about what the subjection is supposed to be. Right. So first of all, you have to be specific about, okay, what are these tasks, what are the current share in economic output? The second thing is you have to be specific about how bad do you think the complementarities are? So in numerical terms, economists use the constant elasticity of substitution to quantify this. So that gives you a numerical estimate of if you just have much more output on some dimensions but not that much on other dimensions, how much does that increase economic output overall?

2:39:41And then there's a third question. You can also imagine you automate a bunch of the economy. Well, a lot of humans were working on those jobs. So now, well, they don't need to do that anymore because those got automated. So they could work on the jobs that haven't yet been automated. So for example, as I gave the example earlier, you might imagine a world in which removed work tasks get automated first and then sensory motor skills lag behind. So you might have a world in which software engineers become physical workers instead. Of course, in that world, the wages of physical workers will be much higher than their wages are today.

2:40:18So that reallocation also produces a lot of extra growth even in the, like if bottlenecks are maximally powerful. Like even if it's literally, you just look at all the tasks in the economy and literally take the worst one for productivity growth. You would still get a lot of increase in output because of this reallocation. So I think one point that I think is useful to make our experience talking to economists about this is that they will bring up these kind of more qualitative considerations. Whereas the arguments that we make are like make specific quantitative predictions about growth rates.

2:40:52So for example, you might ask like how fast will the economy double? And then we can think about, you know, an H100 does about, there are some estimates of how much computation the human brain does per second. And it's about 1 E15 floor per so it's a bit unclear. And then it turns out that an H100 roughly does on that order of computation. And so you can ask the question of how long does it take for an H100 to pay itself back? If you run the software of the human brain. If you run the software of the human brain, you can then deploy that in the economy and earn, say, human wages on the order of 50 to 100 K a year or whatever in the US.

2:41:32And so then it pays itself back because it costs on the order of 30 K per H100. And so you get a doubling time of maybe on the order of a year. And so this is like a very quantitatively specific prediction about, you know, and then there's the response while you have bum -offacts. And then you're like, okay, well, what does this mean? Like, does it double? Does this predict it doubles every two years or every five years? Like, you need just more assumptions in order to make this a coherent objection. And so I think a thing that's a little bit, you know, confusing is just that there are these qualitative objections that I agree with.

2:42:12Like, Botanx R &D is important, which is part of the reason I'm more skeptical of this software singularity story. But I think this is not sufficient for blocking explosive growth. Hmm. The other objection that I've heard often, and it might have a similar response from you, is this idea that a lot of the economy is comprised of o -ring type activities. And this refers to, I think, the challenger spatial explosion. There is just like one component. I forgot what the exact problem with the o -ring was. But because of that being faulty, the whole thing collapsed. I mean, I think it's quite funny, actually, because the o -ring model is taking the product of many inputs.

2:42:56And then the overall output is the product of very many things. That's right. And so, but actually, this is like pretty optimistic from the point of view of having fewer Botanx because we pointed this out before, which again, talking about software on the singular that I said, like, if it's the product of computer experiments with resource - but if one of those products is because of human - But you have constant marginal product there, right? No, but yeah, but if one of those products doesn't scale, that doesn't limit, like, yeah, it means you're less efficient at scaling than your otherwise would be.

2:43:28But you can still get a lot of it. Unbounded, you can just have unbounded scaling in the o -ring world. So, actually, I disagree with Tyler that he's not conservative enough, that he should take his, you know, Botanx view more seriously than he actually is. And yet, I disagree with him about the conclusion. And I think that we're going to get explosive growth once we have AI that can flexibly subside. I'm not trying to understand, like, there will be entirely new organizations that AI's come up with. We've written a blog post about one such with the AI platforms. And you might get productive worker or a productive contributor in this existing organizers to this existing AI.

2:44:06In the AI world, many humans might just be like zero or even minus, I agree. Why won't that put that in the multiplication? Why would you put that in the loop? You're both saying that humans would have, like, humans would be like negatively contributing to output. But then you're also saying that, like, we should put them into the, like, it seems like these. Okay, fair, fair. The main objection often is a regulation. And I think we've addressed it implicitly in different points, but might as well just as we'll see address why won't regulations stop this. Yeah, so for what it's worth, like, we do have like a paper where we go over all the arguments for against explosive growth.

2:44:48And regulation, I think, is the one that seems like stronger, since it gets. Because, like, the reason it seems strong is because even though we have made arguments before about international competition and, like, variation of policies among jurisdictions, these strong incentives to adopt this technology, both for economic and national security reasons. So I think those are pretty compelling when taken together. But even still, like, the world does have surprising ability to, like, coordinate on just not pursuing certain technologies. Right. And human clothing. That's right. So I think, like, it's hard to be extremely confident that this is not going to happen.

2:45:28Like, I think it's less likely that we're going to do this for AI than it is for human cloning. But because I think human cloning patches on some other taboos and so on. But that's valuable and also less valuable. But, and probably less important also for national security in an immediate sense. But at the same time, as I said, it's just hard to rule with us. So I wouldn't say, like, if someone said, well, I think like there's a 10 % or 15 % whatever, 20 % chance that, like, there will be some kind of global coordination and in off -regulation. And that's going to just be very effective. Maybe it will be enforced through, like, sanctions on countries that defect or, you know, and then that is going to, like, maybe it doesn't prevent AI from being deployed, but maybe it's just, like, slow things down and off that you never quite get exposed to growth.

2:46:14Like, I don't think that's an unreasonable view of it. Like, 10 % chance could be. I think I don't know if there's any, I don't know, do you encounter any other, any other, what should I be hassling you about? Yeah. I mean, some things that we've heard from economists, like, again, there was this argument that, like, people sometimes respond to our argument of an explosive growth, which is like, a fact, there's an argument about growth levels. So we're saying, we're going to see 30 % growth per year, instead of 3%. They respond to that with an objection about levels. So they say, well, how much more efficient, how much more valuable can you make, like, hairdressing or, like, taking slides or whatever, or going to a restaurant?

2:46:55And, like, that is just fundamentally the wrong kind of objection. Like, we're talking, we're talking about the rate of change and you're objecting to it by making an argument about the absolute level of productivity. And as I said before, there's not an argument that economists themselves would endorse if it was made about a slower rate of growth, continuing for a longer time. So it seems more like a, like, special pleading, like, I mean, why not just the deployment thing where the same argument you made about AI, where you do learn a lot just by deploying to the world and seeing what people find useful, chat GPT was an example of this.

2:47:31Why would a similar thing happen with AI products and services where you're just, if you, one of the components is you put it out to the marketplace and people play with it and you find out what they need and it clings to the existing supply chain so forth. Doesn't that take time? Well, I mean, it takes time, but it is often quite fast. In fact, chat GPT grew extremely fast, right? And maybe that was just purely digital service. But, well, I think the important thing would be, like, yeah, one reason to be optimistic is if you think the AI will literally be dropping remote workers or dropping workers in some cases, if you have robotics, then companies that already kind of experienced that onboarding humans, like onboarding humans doesn't take like a very long time.

2:48:15Like, maybe it takes six months for like a, even in a particularly difficult job for a new worker to like start being productive. Well, that's not that long. So I don't think that would rule out, like, companies being able to onboard AI workers, assuming that they don't need to make like a ton of new complementary innovation discoveries, like take advantage. I think one way in which current AI systems are being inhibited and the reason we're seeing the growth maybe be slower than you might otherwise expect is because companies in the economy are not used to working with this new technology. They have to rearrange the way they work in order to take advantage of it.

2:48:51But if AI systems were literally able to substitute for human workers, then well, the complementary innovations might not be as necessary. So actually, this is a good excuse to maybe go to the final topic, which is AI firms. So this is this blockbuster we wrote together about what it would be like to have a firm that is fully automated. And the crucial point we were making was that people tend to overemphasize and think of AI from the perspective of how smart individual copies will be. And if you actually want to understand the ways in which they are superhuman, you want to focus on their collective advantages, which because of biology we are just precluded from, which are the fact that they can they can be copied with all their tacit knowledge.

2:49:36You can copy a Jeff Dean or Elias Hatske or whatever the relevant person is in a different domain. You can even copy Elon Musk and he can be the guy who's every single engineer and the SpaceX rig. And that's not inefficient. Yeah, yeah. And if that's it's not best to have Elon Musk or anything, you just copy the relevant like team or whatever that. And we have this problem with human firms where there can be very effective teams or groups, but over time their cultural dilutes or the people leave or die or get old. And this is one of the many problems that can be solved with these digital firms where you actually firms right now have two of the three relevant criteria for evolution.

2:50:23They have selection and they have variation, but they don't have high fidelity replication. And you could imagine a much more fast -paced and intense sequence of evolution for firms once you once you have this final piece click in. And that relates to the onboarding thing where right now, right now, you know, you like they are just aren't smart enough to be onboarded as full workers. But once they are, I just imagine for my own like the kinds of things I try to hire for it would just be such an unlock. Yeah, it doesn't even matter like the salaries are totally secondary. The fact that I can like this is the skill I need or the set of skills I need.

2:51:02And I can have a worker and just like I can have a thousand workers in parallel. There's something that has a high elasticity of demand. I think it's like probably along with the transformative AI, the most underrated, tangible thing that like you need to understand about what the future AI society will look like. Right. I mean, I think there's the point there's a first point about this like very macroeconomic picture where you just expect a ton of scaling of older relevant interests. And I think that is like the first order thing. Yeah. But then you might have more like micro questions about okay, like how does this world actually look like?

2:51:38How is it different from a world in which we just have a lot more people and a lot more capital and a lot more like because it should be different. And then I think these considerations become important. I think another important thing is just that AI can be aligned. Like you get to control the preferences of your AI systems in a way that you don't really get to control the preference of your workers. Like your workers you can just select we don't really have any other option. But for your AI's you can like fine tune them. You can like build AI systems which have the kind of preferences that you want.

2:52:09And you can imagine that's like dramatically changing basic problems that determine the structure of human firms. Like for example, the principal Asian problem might go away. Like this problem where the you as a worker have incentives that are either different from those of your manager or those that the entire firm or those of the shareholders are to farm. I actually think the incentives is a smaller piece of the puzzle. I think it's more about like bandwidth and information sharing where it's often just with a large organization. It's very hard to have a single coherent vision. And the most of the forms we see today is where for an unusual amount of time a founder is able to keep their vision instilled in the organization like SpaceX or Tesla are examples of this.

2:52:55People talk about Nvidia this way. But just imagine a future version where there's this hyper inference scale mega jensen who you're spending $100 billion a year on inference on and copies of him are constantly you know like writing every single press release and reviewing every pull request and answering every customer service request and so forth. And monitoring the whole organization making sure it's like proceeding along a coherent vision and getting merged back into the hyper hyper jensen hyper jensen mega jensen whatever yeah yeah I agree that's a bigger deal at the same time I would point out that like part of the reason why it's important to have like a coherent vision and culture and so on and human companies might be that there's incentive problems that exist otherwise like I mean I wouldn't rule that out but I agree that the like aside from the overall macroeconomic thing I think the fact that they can be replicated is probably the biggest that's right that's right there yeah like that also enables additional sources of economies of scale or if you have like twice the number of GPUs you can run not only twice the number of copies of your old model but then you can train a model that's even better so you double your training computers and your inference compute and that means you not only double like you don't get just twice the number of workers you would have had otherwise you get more than that because they are also smarter right because you spend more training computers right so then that is additional source of economies of scale and then there's this benefit that you can like for humans you like every human has to learn things from scratch basically like they are born and then they have a certain and a lifetime learning that they have to do so in human learning there is a ton of duplication while for an AI system it could just learn once you could just have one huge training room which a tons of data and then that's where I could be deployed everywhere yeah so that's like another massive advantage that the AI is have over humans yeah this uh what maybe we'll close up with this one debate we've had often had offline which is will central planning work with these economies of scale so I would say that I mean again the question of like will it work what will it be optimal right yeah so I mean my guess is probably not optimal uh but like I think it's hard like I don't think anyone has like thought this question through in like a lot of so they were thinking about just like why one might expect yeah central planning to be slightly better in this world right so so one consideration is just a communication bandwidth being potentially much much greater than it is today and then like in the current world the information gathering and the information processing are like co -located like humans observe and also process what they observe in an AI world you can disaggregate that that's actually a really interesting point yeah and so you can have the sensors and you know that's right that's right not do much processing but just collect and then process centrally and that processing centrally might make sense for a bunch of reasons and you might get economies of scale from you know having more GPUs that produce better models and also be be able to you know think more deeply about what it's seeing it's worth noting that certain things are ready work like this for example Tesla FSD yep it will benefit from the data collected at the periphery from millions of miles of driving and then the improvements which are made as a result of this centrally directed it's like coming from yeah HQ being like we're gonna push an update that's right and so you do get some of this more centralized um and it can be a much more intelligent form than just whatever grading daverging that they I mean shrimp sure is more sophisticated than Tesla but it can be a much more like deliberate intelligent that's right update right so so that that's that's one reason to expect and the other reason I guess is just having like current leaders or CEOs are don't have bigger brains than than the workers do maybe a little bit I don't know if you want to open that but not by orders of magnitude right and so you could have just orders of magnitude more scaling of the size of the right models that are doing the planning than the people or the agents or workers doing the actions yeah and I think a third reason is the thing about like the incentive thing we're like uh you you wouldn't face this problem that uh like part of the reason you have a market is that it gives people the like dry kind of incentives uh but you might need not not need that as much if you're using AI so I think there there's an argument that if you just list the traditional arguments people have made against like why does central bank not work then you might expect them to become weaker now I think that is still um like there's a danger when you do doing that kind of analysis to fall into the same kind of like partial equilibrium analysis where you're like only considering some factors and then you're not considering other things like for example you consider get more complex you just have a much bigger economy and so like on the one hand your ability to uh kind of uh kind of collect information and process it improves but also the need for doing that also increases that seems to become more complex and I mean when I'm gonna illustrate that is like imagine if Apple the organization today with all its compute and whatever was tasked with managing the economy of orac right I think it like it actually could centrally plan the economy might like the economy work might work even better as a result but like Apple as it existed I cannot manage the economy the world economy is there's this today that's right I mean that's a good yeah yeah um all right actually this will be the final question look one of the things that makes AI so fascinating is that there's no domain of human knowledge that is irrelevant to studying it because what we're really trying to I don't know about that there's no serious domain of human knowledge that's that's better um that is not relevant to studying it because you're just fundamentally trying to figure out what a future society will look like and so it's like obviously computer science is relevant but also economics is even discussing history and how to understand history and many other things you've been discussing right especially if you have longer timelines and there is enough time for somebody to pursue a meaningful career here what would you recommend to somebody because both of you are quite young I mean you especially I gave it like both of you so it's like this is not you would think this is the kind of thing which requires crystallized intelligence or whatever especially given what we've said earlier about look as we get more knowledge we're going to have to factoring what we're learning into building a better model of what's going to happen to the world and if somebody is interested in this kind of career that you both have um what advice do you have for them yeah that's a hard question I mean I'm not sure like I think the there is an extent to which it's like it's difficult to deliberately pursue like the implicit strategy that we would have pursued like it's probably works better if it's spontaneous and like more driven by curiosity and interest then like you make a deliberate choice okay like I'm just going to learn about a bunch of things so that I can contribute to the discourse in AI I would think that strategy is probably less effective at least I haven't seen anyone who deliberately use that strategy and then what's successful it seems like yeah I guess not that I've contributed to discourse directly but maybe facilitate it other people contributing um I guess it wasn't deliberate strategy on my end but it was a deliberate strategy to do the podcast which inadvertently gave me the opportunity to learn about multiple fields yeah so given like if you're already interested and curious and reading a bunch of things and studying a bunch of things and thinking about these topics on the margin there are a bunch of things you can do to make more productive at having this of making some contributions to this and I think just speaking to people and writing your thoughts down and finding like especially useful people to chat with and collaborate with I think that's where you use right um so just seek out people that have similar views and you're able to have very high bandwidth conversations with and seemingly you know and kind of make progress on these topics and I think that's just pretty useful but I'd be like how how exactly like I know DM you like how do you yeah sure I like I don't know set up signal chats with with with your friends or whatever yeah I've done a lot actually I it's like crazy how much off I've got out of that but um yeah I mean I think like one of the in fact one advice I would give to people in general even even if they are not like thinking about AI specifically but I think it's also helpful for that is just like people should just be much more aggressive about reaching out like that's right yeah like um a lot of the communication that like like maybe people have an impression that if you like reach out to someone who looks really important and like they're not going to respond to you but like if the if what you send to them is just interesting and like high quality then it's very very likely that they will respond like like uh like there's like a lot more edge there that you can get which is just being more aggressive and like less ashamed or something of like doing looking dumb like that's the main advice of give because if you want to be productive then again like there are these complementarity's and so on there like different you know you need to be part of like some community or some organization and it goes back to the thing of reasoning alone not being that helpful yeah yeah it's just like other people have thought a long time and have randomly stumbled upon useful ideas that you can take advantage of that's right so you should just like try to place yourself in a situation where you you can become part of something larger right which is that working on the phone that's just a more more effective way of contributing and to do that you have to well let people know that's right that's right that's right and I think just coming to the Bay Area is especially for just an AI in particular yeah putting some areas nice just post like just writing things and like posting them or if you can see them just aggressively reaching out to people with that interesting uh comments provided your like thoughts or interest and so on I mean they probably are like in many cases I think it's like my thoughts my thoughts but still might not be interesting but people tolerate my cold emails and uh are like you know we'll still like do say collaborate with me and so forth yep um uh the other thing I've noticed tell me this is actually the wrong pattern or the wrong um yeah with people like you with car or something is that as compared to a general person who's intellectually curious or reading widely you tend to focus much more on key pieces of literature than say I'm gonna go read the classics or just generally read like it it's like I'm gonna I'm gonna just like put like a ton more credence in something like the rumor paper and a normal person might not even read the normal a normal person who's like intellectual curious or just like not be reading key pieces of literature yeah I think like you have to be very mindful of the fact that you have a very limited amount of time like you're not an AI model so you have to kind of aggressively prioritize what you're going to spend your time read like reading even AI models don't prioritize that heavily they read read it mostly or like a large part of their corpus is yeah yeah he misses a empirical literature at least right at least among you guys I mean they're not gonna be the most productive thing in general but I think that's useful I also just think it's useful to read twitter I think we're having this conversation about people often say that they should like they're spending too much time reading twitter and the which they spend more time reading archive but actually like the amount of information per unit time you get reading twitter is often just much higher yeah and and it's just much more productive for them to read to read to read to I think there are key pieces of literature that are kind of important and I think it's it's useful to figure out what people who have spent a lot of time thinking about this find important in their worldview so you know in AI this might be you know key papers like I don't know like the Andy Jones paper about scaling loss for inferences like a big thing and in economics like this Romer paper or the paper on explaining long run population from Kramer or from from the David Rudman and so on I think I think just like if people who you think are having really good who think very well about this suggests a certain paper and they like highly recommended them I think you should take that seriously and actually read those papers and for me it's been especially helpful to instead of just skimming a bunch of things just like really stop on like yeah if there's a key piece of literature or for yeah to in order to for example understand the transformer I like there's always a carpentry lectures but one of the sources I was really useful is the anthropics original transformer circuit paper and I just like just spending a day on that paper and instead of skimming yet and making a bunch of space repetition cards and so forth was much more useful than just like generally reading right like I think it's just much more important here to if you want to prioritize things correctly to be again to be part of a community or to be getting inputs from a community or get from people who have thought a lot and have a lot of experience about what is important and what is not yeah like this is true even an academic field so if you want to do math research but you're not part of like a graduate program you're not at a university where there are tons of people who like do math research all day for many years then you're not even going to know like what are the open problems that I should be working on what is reasonable to attack what is not reasonable to have to act like what what papers in this field are important contain important techniques you're just going to have no idea right so it's very important to be like plugged into that feeder information somehow but how did you know other ship before being plugged in because you weren't talking to you didn't on car I mean you don't need to talk I mean the internet is a pretty pretty useful thing in this respect and you don't need to necessarily talk to people like you can get a lot of benefit from reading like you just need to identify okay like where the people who seem like constantly most interesting and you can also get a lot of benefit or maybe you found one person and then uh often that person will know some other people who are interesting right and then you can like start tracing the social network so for example maybe um uh I don't know like one example I can give which I think is actually accurate is like maybe you know about Daniel Ellsberg so you like look for a podcast where he appears on you notice that he's appeared on 80 ,000 hours broadcasts he has and then you notice like there are some other guests on the 80 ,000 hours broadcasts so maybe there's Brian Kaplan who has also appeared on the podcast and then maybe Robin Hansen has also appeared on the podcast and then you know maybe there are some people those other people know and then like like just tracing that kind of social network and like figuring out which will listen to like that I think that can be and I think you're doing a very big service to making that possible where like I think your selection is often very good uh I'm actually curious you're up fine when I got wrong well actually I think I know you the answer to that so so you know and I think that that makes it a bunch easier to track like right you know who are the people doing the most interesting thinking on on various topics that's right cool I think that's a good place to end with you praising me okay um I again I highly recommend people follow epoch there's a great weekly newsletter gradient updates which I mean like people plug new sliders but this is like I can't believe this is a thing that comes out on a weekly basis and it's like it anyways um and you now have a new podcast which I will not plug as a competitor but you can check it out from then in your studio through that yeah that's very generous anyways uh cool thanks guys all right thanks