Dwarkesh and Ilya Sutskever on What Comes After Scaling

35:13And people say that a lot. And there is truth to that. But then I saw someone say on Twitter something like, if ideas are so cheap, how come no one's having any ideas? and I think it's true too I think like if you think about research progress in terms of bottlenecks there are several bottlenecks if you go back to the and one of them is ideas and one of them is your ability to bring them to life which might be compute but also engineering so if you go back to the 90s let's say you had people who had pretty good ideas and if they had much larger computers maybe they could demonstrate that their ideas were viable, but they could not.

35:58So they could only have very, very small demonstration and not convince anyone. So the bottleneck was compute. Then in the age of scaling, computers increased a lot. And of course, there is a question of how much compute is needed, but compute is large. So compute is large enough such that it's like not obvious that you need that much more compute to prove some idea. Like, I'll give you an analogy. AlexNet was built on two GPUs. That was the total amount of compute use for it. The transformer was built on eight to 64 GPUs. No single transformer paper experiment used more than 64 GPUs of 2017, which would be like, what, two GPUs of today?

36:50so the resnet right many like even the you could argue that the like 01 reasoning was not the most compute heavy thing in the world so they're definitely for research you need like definitely some amount of compute but it's far from obvious that you need the absolutely largest amount of compute ever for research you might argue and I think it is true that if you want to build the absolutely best system, if you want to build the absolutely best system, then it helps to have much more compute. And especially if everyone is within the same paradigm, then compute becomes one of the big differentiators.

37:37Yeah, I guess while it was possible to develop these ideas, I'm asking you for the history because you were actually there. I'm not sure what actually happened, but it sounds like it was possible to develop these ideas using minimal amounts of compute. But the transformer didn't immediately become famous. It became the thing everybody started doing and then started experimenting on top of and building on top of because it was validated at higher and higher levels of compute. Correct. And if you at SSI have 50 different ideas, how will you know which one is the next transformer and which one is, you know, brittle without having the kinds of compute that other frontier labs have?

38:17So I can comment on that, which is, the short comment is that, you know, you mentioned SSI, specifically for us, the amount of compute that SSI has for research is really not that small. And I want to explain why, like a simple math can explain why the amount of compute that we have is actually a lot more comparable for research than one might think. Now explain.

38:50So SSI has raised$3 billion, which is like not small, but it's like a lot by any absolute sense, but you could say, but look at the other companies raising much more. But a lot of their compute goes for inference. Like these big numbers, these big loans, it's earmarked for inference. That's number one. Number two, you need, if you want to have a product on which you do inference, you need to have a big staff of engineers, of salespeople. A lot of the research needs to be dedicated for producing all kinds of product-related features. So then when you look at what's actually left for research, the difference becomes a lot smaller.

39:37Now, the other thing is, is that if you are doing something different, do you really need the absolute maximal scale? To prove it? I don't think it's true at all. I think that in our case, we have sufficient compute to prove, to convince ourselves and anyone else that what we're doing is correct. There's been public estimates that companies like OpenAI spend on the order of$5-6 billion a year, just so far, on experiments. This is separate from the amount of money they're sending on inference and so forth. So it seems like they're spending more a year running research experiments than you guys have in total funding.

40:18I think it's a question of what you do with it. It's a question of what you do with it. Like they have, like the more, I think in their case and the case of others, I think there's a lot more demand on the training compute. There's a lot more different work streams. There are different modalities. There is just more stuff. And so it becomes fragmented. How will SSI make money? You know, my answer to this question is something like, right now we just focus on the research and then the answer to this question will reveal itself. I think there will be lots of possible answers. Is SSI's plan still to straight shot super intelligence?

41:00Maybe. I think that there is merit to it. I think there's a lot of merit because I think that it's very nice to not be affected by the day-to-day market competition. But I think there are two reasons that may cause us to change the plan. One is pragmatic if timelines turned out to be long, which they might. And second, I think there is a lot of value in the best and most powerful AI being out there impacting the world. I think this is a meaningfully valuable thing. But then, so why is your default plan to Straight Shots over intelligence? Because it sounds like, you know, OpenAI, Anthropic, all these other companies, their explicit thinking is, look, we have weaker and weaker intelligences that the public can get used to and prepare for.

41:56and why is it potentially better to build a super intelligence directly? So I'll make the case for and against. The case for is that you are, so one of the challenges that people face when they're in the market is that they have to participate in the rat race. And the rat race is quite difficult in that it exposes you to difficult trade-offs which you need to make. and there is, it is nice to say, we'll insulate ourselves from all this and just focus on the research and come out only when we are ready and not before. But the counterpoint is valid too. And those are opposing forces. The counterpoint is, hey, it is useful for the world to see powerful AI.

42:46It is useful for the world to see powerful AI because that's the only way you can communicate it. Well, I guess not even just that you can communicate the idea. Communicate the AI, not the idea. Communicate the AI. What do you mean communicate the AI? Okay, so let's suppose you read an essay about AI. And the essay says AI is going to be this, and AI is going to be that, and it's going to be this. And you read it and you say, okay, this is an interesting essay. Now suppose you see an AI doing this, an AI doing that. It is incomparable. Like basically, I think that there is a big benefit from AI being in the public.

43:25And that would be a reason for us to not be quite straight shot. Yeah. Well, I guess it's not even that, but I do think that is an important part of it. The other big thing is, I can't think of another discipline in human engineering and research where the end artifact was made safer mostly through just thinking about how to make it safe as opposed to why are airplane crashes per mile so much lower today than there were decades ago? Why is it so much harder to find a bug in Linux than it would have been decades ago? And I think it's mostly because these systems were deployed to the world. You noticed failures.

44:05Those failures were corrected and the systems became more robust. And I'm not sure why AGI and superhuman intelligence would be any different, especially given, and I hope we're going to get to this, it seems like the harms of superintelligence are not just about having some malevolent paper clipper out there, but it's just like, this is a really powerful thing, and we don't even know how to conceptualize how people will interact with it, what people will do with it. and having gradual access to it seems like a better way to maybe spread out the impact of it and to help people prepare for it. Well, I think on this point, even in the straight shot scenario, you would still do a gradual release of it.

44:50It's how I would imagine it. Gradualism would be an inherent component of any plan. It's just a question of what is the first thing that you get out of the door. That's number one. Number two, I also think, you know, I believe you have advocated for continual learning more than other people. And I actually think that this is an important and correct thing. And here is why. So one of the things, so I'll give you another example of how language affects thinking. And in this case, it is going to be two words, two words that have shaped everyone's thinking, I maintain. First word, AGI. Second word, pre-training.

45:37Let me explain. So the word, the term AGI, why does this term exist? It's a very particular term. Why does it exist? There's a reason. The reason that the term AGI exists is in my opinion, not so much because it's like a very important essential descriptor of some end state of intelligence but because it is a reaction to a different term that existed and the term is narrow AI if you go back to ancient history of game plane AI of checkers AI chess AI computer games AI everyone would say look at this narrow intelligence. Sure, the chess AI can beat Kasper off, but it can't do anything else. It is so narrow, artificial narrow intelligence.

46:30So in response, as a reaction to this, some people said, well, this is not good. It is so narrow. What we need is general AI. General AI, an AI that can just do all the things. The second, and that term, just got a lot of traction. The second thing that got a lot of traction is pre-training. Specifically the recipe of pre-training. I think the current, the way people do RL now is maybe undoing the conceptual imprint of pre-training. But pre-training had the property. You do more pre-training and the model gets better at everything more or less uniformly. General AI. Pre-training gives AGI.

47:24But the thing that happened with AGI and pre-training is that in some sense they overshot the target. Because by the kind, if you think about the term AGI, you will realize, and especially in the context of pre-training, you will realize that a human being is not an AGI. Because a human being, yes, there is definitely a foundation of skills, a human being lacks a huge amount of knowledge. Instead, we rely on continual learning. We rely on continual learning. And so then when you think about, okay, so let's suppose that we achieve success and we produce some kind of safe superintelligence. The question is, but how do you define it?

48:10Where on the curve of continual learning is it going to be? I produce like a super intelligent 15 year old that's very eager to go and you say okay I'm going to they don't know very much at all the great student very eager you go and be a programmer you go and be a doctor go and learn so you could imagine that the deployment itself will involve some kind of a learning trial and error period it's a process as opposed to you drop the finished thing okay I see so you're suggesting that the thing you're pointing out with super intelligence is not some finished mind which knows how to do every single job in the economy.

48:54Because the way, say, the original, I think, OpenAI Charter or whatever defines AGI is like, it can do every single job, every single thing a human can do. You're proposing instead a mind which can learn to do every single job. Yes. And that is super intelligence. But once you have the learning algorithm, it gets deployed into the world the same way a human laborer might join an organization. And it seems like one of these two things might happen. Maybe neither of these happens. One, this super efficient learning algorithm becomes superhuman, becomes as good as you, and potentially even better at the task of ML research.

49:38And as a result, the algorithm itself becomes more and more superhuman. The other is, even if that doesn't happen, if you have a single model, I mean, this is explicitly your vision. If you have a single model or instances of a model which are deployed through the economy, doing different jobs, learning how to do those jobs, continually learning on the job, picking up all the skills that any human could pick up, but actually picking them all up at the same time and then amalgamating the learnings. you basically have a model which functionally becomes super intelligent even without any sort of recursive self-improvement in software.

50:14Right? Because you now have one model that can do every single job in the economy. And humans can't merge our minds in the same way. And so do you expect some sort of intelligence explosion from broad deployment? I think that it is likely that we will have rapid economic growth. I think the broad deployment like there are two arguments you could make which are conflicting one is that look if indeed you get once indeed you get to a point where you have an AI that can learn to do things quickly and you have many of them then they will then there will be a strong force to deploy them in the economy unless there will be some kind of a regulation that stops it, which by the way, there might be.

51:08But I think the idea of very rapid economic growth for some time, I think it's very possible from broad deployment. The other question is how rapid it's going to be. So I think this is hard to know because on the one hand, you have this very efficient worker. on the other hand there is the world is just really big and there's a lot of stuff and that stuff moves at a different speed but then on the other hand now the ai could you know so i think very rapid economic growth is possible and we will see like all kinds of things like different countries with different rules and the ones which have the friendlier rules the economic growth will be faster hard to predict it seems to me that this is a very precarious situation to be in where, look in the limit, we know that this should be possible because if you have something that is as good as a human at learning, but which can merge its brains, merge their different instances in a way that humans can't merge, already this seems like a thing that should physically be possible.

52:10Humans are possible. Digital computers are possible. You just need both of those combined to produce this thing. And it also seems like this kind of thing is extremely powerful and economic growth is one way to put it. I mean, Dyson Spear is a lot of economic growth. But another way to put it is just like, you will have potentially a very short period of time because a human on the job can, you know, you're hiring people to SSI and six months they're like net productive probably, right? A human like learns really fast. And so this thing is becoming smarter and smarter very fast. What is, how do you think about making that go well?

52:45And why is SSI positioned to do that well? What is SSI's plan there basically is what I'm trying to ask. Yeah. So one of the ways in which my thinking has been changing is that I now place more importance on AI being deployed incrementally and in advance. one very difficult thing about AI is that we are talking about systems that don't yet exist. And it's hard to imagine them. I think that one of the things that's happening is that in practice, it's very hard to feel the AGI. It's very hard to feel the AGI. We can talk about it, but it's like talking about like the long future, like imagine like having a conversation about like, how is it like to be old when you're like old and frail and you can have a conversation, you can try to imagine it, but it's just hard and you come back to reality where that's not the case.

53:57And I think that a lot of the issues around AGI and its future power stem from the fact that it's very difficult to imagine. Future AI is going to be different. It's going to be powerful. Indeed, the whole problem, what is the problem of AI and AGI? The whole problem is the power. The whole problem is the power. When the power is really big, what's going to happen? And one of the ways in which I've changed my mind over the past year. And so that change of mind may back, may, I'll say, I'll hedge a little bit, may back propagate into the plans of our company is that, so if it's hard to imagine, what do you do?

54:55You gotta be showing the thing. You gotta be showing the thing. And I maintain that I think most people who work on AI also can't imagine it. Because it's too different from what people see on a day-to-day basis. I do maintain, here's something which I predict will happen. That's a prediction. I maintain that as AI becomes more powerful, then people will change their behaviors. And we will see all kinds of unprecedented things which are not happening right now. And I'll give some examples. I do like, I think for better or worse, the frontier companies will play a very important role in what happens, as will the government.

55:44And the kind of things that I think you'll see, which you see the beginnings of, companies that are fierce competitors starting to collaborate on AI safety. You may have seen OpenAI and Anthropic doing a first small step, but that did not exist. That's actually something which I predicted in one of my talks about three years ago, that such a thing will happen. I also maintain that as AI continues to become more powerful, more visibly powerful, there will also be a desire from governments and the public to do something. And I think that this is a very important force of showing the AI. That's number one.

56:31Number two, okay, so then the AI is being built. what needs to be done. So one thing that I maintain that will happen is that right now people who are working on AI I maintain that the AI doesn't feel powerful because of its mistakes. I do think that at some point the AI will start to feel powerful actually. And I think when that happens we will see a big change in the way all AI companies approach safety. They'll become much more paranoid. I think I say this as a prediction that we will see happen. We'll see if I'm right. But I think this is something that will happen because they will see the AI becoming more powerful.

57:14Everything that's happening right now, I maintain, is because people look at today's AI and it's hard to imagine the future AI. And there is a third thing which needs to happen. And I think this is, and I'm talking about it in broader terms, not just from the perspective of SSI, because you asked me about our company. But the question is, okay, so then what should the companies aspire to build? What should they aspire to build? And there has been one big idea that actually everyone has been locked into, which is the self-improving AI. And why did this happen? Because there is fewer ideas than companies.

57:56But I maintain that there is something that's better to build. And I think that everyone will actually want that. it's like the AI that's robustly aligned to care about sentient life specifically. I think in particular it will be, there's a case to be made that it will be easier to build an AI that cares about sentient life than an AI that cares about human life alone because the AI itself will be sentient. And if you think about things like mirror neurons and human empathy for animals, which is, you know, you might argue it's not big enough, but it exists. I think it's an emergent property from the fact that we model others with the same circuit that we used to model ourselves, because that's the most efficient thing to do.

58:46So even if you got an AI to hear about sentient beings, and it's not actually clear to me that that's what you should try to do if you solve the alignment, it would still be the case that most sentient beings will be AIs. There will be trillions, eventually quadrillions of AIs. Humans will be a very small fraction of sentient beings. So it's not clear to me if the goal is some kind of human control over this future civilization, that this is the best criterion. It's true. I think that it's possible. It's not the best criterion. I'll say two things. I think that thing number one, I think that if there, so I think that care for sentient life, I think there is merit to it.

59:36I think it should be considered. I think that it will be helpful if there was some kind of a short list of ideas that then the companies, when they are in this situation, could use. That's number two. Number three, I think it would be really materially helpful if the power of the most powerful superintelligence was somehow capped because it would address a lot of these concerns. The question of how to do it, I'm not sure, but I think that would be materially helpful when you're talking about really, really powerful systems. Before we continue the alignment discussion, I want to double click on that.

1:00:18How much room is there at the top? how do you think about superintelligence? Do you think, I mean, using this learning efficiency idea, maybe it's just extremely fast at learning new skills or new knowledge. And does it just have a bigger pool of strategies? Is there a single cohesive it in the center that's more powerful or bigger? And if so, do you imagine that this will be sort of godlike in comparison to the rest of human civilization? Or does it just feel like another agent or another cluster of agents? So this is an area where different people have different intuitions. I think it will be very powerful for sure.

1:00:56I think that what I think is most likely to happen is that there will be multiple such AIs being created roughly at the same time. I think that if the cluster is big enough, like if the cluster is literally continent-sized, that thing could be really powerful indeed. If you literally have a continent-sized cluster, those AIs can be very powerful. And all I can tell you is that if you're talking about extremely powerful AIs, like truly dramatically powerful, then yeah, it would be nice if they could be restrained in some ways or if there was some kind of an agreement or something. Because I think that if you are saying, hey, like if you really, like what is the concern of super intelligence?

1:01:53What is one way to explain the concern? If you imagine a system that is sufficiently powerful, like really sufficiently powerful, and you could say, okay, you need to do something sensible, like care for sentient life, let's say, in a very single-minded way, we might not like the results. That's really what it is. And so maybe, by the way, the answer is that you do not build a single, You do not build an IRL agent in the usual sense. And actually, I'll point several things out. I think human beings are a semi-IRL agent. You know, we pursue a reward and then the emotions or whatever make us tire out of the reward, we pursue a different reward.

1:02:32The market is like a very short-sighted kind of agent. Evolution is the same. Evolution is very intelligent in some ways, but very dumb in other ways. the government has been designed to be a never-ending fight between three parts which has an effect so i think things like this another thing that makes this discussion difficult is that we are talking about systems that don't exist that we don't know how to build right that's the other thing and that's actually my belief i think what people are doing right now will go some distance and then peter out it will continue to improve but it will also not be it so the it we don't know how to build and i think that a lot a lot hinges on understanding reliable generalization now say another thing which is like you know one of the things that you could say is what would that cause alignment to be difficult is that human value that it's it's um your ability to learn human values is fragile, then your ability to optimize them is fragile, you actually learn to optimize them.

1:03:42And then can't you say, are these not all instances of unreliable generalization? Why is it that human beings appear to generalize so much better? What if generalization was much better? What would happen in this case? What would be the effect? But those questions are right now still unanswerable. How does one think about what AI going well looks like? Because I think you've scoped out how AI might evolve. We'll have these sort of continual learning agents. AI will be very powerful. Maybe there will be many different AIs. How do you think about lots of continent compute size intelligences going around?

1:04:21how dangerous is that? How do we make that less dangerous? And how do we do that in a way that protects an equilibrium where there might be misaligned AIs out there and bad actors out there? So one reason why I liked the AI that cares for sentient life, and we can debate on whether it's good or bad, but if the first n of these dramatic systems actually do care for you know love humanity or something you know care for sentient life obviously this also needs to be achieved this needs to be achieved so if this is achieved by the first n of those systems then I can see it go well at least for quite some time and then there is the question of what happens in the long run what happens in the long run, how do you achieve a long run equilibrium and I think that there there is an answer as well and I don't like this answer but it needs to be considered in the long run you might say okay so if you have a world where powerful AI's exist, in the short run you could say, okay, you have universal high income.

1:05:43You have universal high income. And we're all doing well. But we know that, what do the Buddhists say? Change is the only constant. And so things change. And there is some kind of government, political structure thing, and it changes. Because these things have a shelf life. You know, some new government thing comes up and it functions, and then after some time it stops functioning that's something that you see happening all the time and so I think that for the long run equilibrium one approach you could say okay so maybe every person will have an AI that will do their bidding and that's good and if that could be maintained indefinitely that's true but the downside with that is okay so then the AI goes and like earns you know earns money for the person and advocates for their needs in the political sphere and maybe then writes a little report saying, okay, here's what I've done, here's the situation, and the person says, great, keep it up.

1:06:44But the person is no longer a participant. And then you can say that's a precarious place to be in. But, so I'm going to preface by saying, I don't like this solution, but it is a solution. and the solution is if people become part AI with some kind of neural link plus plus because what will happen as a result is that now the AI understands something and we understand it too because now the understanding is transmitted wholesale so now if the AI is in some situation now it's like you are involved in that situation yourself fully and I think this is the answer to the equilibrium I wonder if the fact that emotions, which were developed millions or in many cases billions of years ago in a totally different environment, are still guiding our actions so strongly is an example of alignment success.

1:07:45to maybe spell out what I mean. The brainstem has these, I don't know if it's more accurate to call it a value function or reward function, but the brainstem has a directive where it's saying mate with somebody who's more successful. The cortex is the part that understands what does success mean in the modern context. But the brainstem is able to align the cortex and say, however you recognize success to be, and I'm not smart enough to understand what that is, you're still going to pursue this directive. I think there is a more general point. I think it's actually really mysterious how the brain encodes high-level desires.

1:08:26Sorry, how evolution encodes high-level desires. It's pretty easy to understand how evolution would endow us with the desire for food that smells good. Because smell is a chemical, and so just pursue that chemical. It's very easy to imagine such evolution doing such a thing. But evolution also has endowed us with all these social desires. Like we really care about being seen positively by society. We care about being in a good standing. Like all these social intuitions that we have, I feel strongly that they're baked in. And I don't know how evolution did it. Because it's a high level concept.

1:09:09that's represented in the brain. Like what people think, like let's say you are like, you care about some social thing. It's not like a low level signal like smell. It's not something that, for which there is a sensor. Like the brain needs to do a lot of processing to piece together lots of bits of information to understand what's going on socially. And somehow evolution said, that's what you should care about. Yes. How did it do it? And it did it quickly too. Yeah. Because I think all these sophisticated social things that we care about, I think they evolved pretty recently. So evolution had an easy time hardcoding this high level desire.

1:09:50And I maintain, or at least I'll say, I'm unaware of good hypotheses for how it's done. I had some ideas I was kicking around, but none of them are satisfying. Yeah. And what's especially impressive is it was a desire that you learned in your lifetime. It kind of makes sense because your brain is intelligent. It makes sense why we'll be able to learn intelligent desires. But your point is that the desire is, maybe this is not your point, but one way to understand it is the desire is built into the genome and the genome is not intelligent, right? But it's able to, you're somehow able to describe this feature that requires, like, it's not even clear how you define that feature and you can get it into the, you can build it into the gene.

1:10:35Yeah, essentially. Or maybe I'll put it differently. If you think about the tools that are available to the genome, it says, okay, here's a recipe for building a brain. And you could say, here is a recipe for connecting the dopamine neurons to like the smell sensor. and if the smell is a certain kind of good smell you want to eat that I could imagine the genome doing that I'm claiming that it is harder to imagine it's harder to imagine the genome saying you should care about some complicated computation that your entire brain that like a big chunk of your brain does that's all I'm claiming I can tell you like a speculation I was wondering how it could be done and let me offer a speculation and I'll explain why the speculation is probably false so the speculation is okay so the brain it's like the brain has those regions you know the brain regions we have our cortex right it has all those brain regions and the cortex is uniform but the brain regions and the neurons in the cortex they kind of speak to their neighbors mostly and that explains why you get brain regions because if you want to do some kind of speech processing all the neurons that do speech need to talk to each other and because neurons can only speak to their nearby neighbors for the most part, it has to be a region.

1:11:53All the regions are mostly located in the same place from person to person. So maybe evolution hard-coded literally a location on the brain. So it says, oh, like when like, you know, the GPS of the brain, GPS coordinates such and such, when that fires, that's what you should care about. Like maybe that's what evolution did because that would be within the toolkit of evolution. Yeah, although there are examples where for example, people who are born blind have that area of their cortex adopted by another sense. And I have no idea, but I'd be surprised if the desires or the reward functions which require visual signal no longer worked.

1:12:37You know, people who have their different areas of their cortex co-opted. For example, if you no longer have vision, can you still feel the sense that I want people around me to like me and so forth, which usually there's also visual cues for. So I actually fully agree with that. I think there's an even stronger counter argument to this theory, which is, like if you think about people, so there are people who get half of their brains removed in childhood. And they still have all their brain regions, but they all somehow move to just one hemisphere, which suggests that the brain regions, their location is not fixed.

1:13:13And so that theory is not true. It would have been cool if it was true, But it's not. And so I think that's a mystery, but it's an interesting mystery. Like the fact is somehow evolution was able to endow us to care about social stuff very, very reliably. And even people who have like all kinds of strange mental conditions and deficiencies and emotional problems tend to care about this also. What is SSI planning on doing differently? So presumably your plan is to be one of the frontier companies when this time arrives. and then what is presumably you started SSI because you're like I think I have a way of approaching how to do this safely in a way that the other companies don't what is that difference?

1:13:57So the way I would describe it as there are some ideas that I think are promising and I want to investigate them and see if they are indeed promising or not it's really that simple, it's an attempt I think that if the ideas are now to be correct these ideas that we discussed around understanding generalization if these ideas turn out to be correct then I think we will have something worthy will they turn out to be correct? we are doing research we are squarely age of research company we are making progress we've actually made quite good progress over the past year but we need to keep making more progress more research and that's how I see it I see it as an attempt to be an attempt to be a voice and a participant.

1:14:50People have asked your co-founder and previous CEO left to go to Meta recently. And people have asked, well, if there was a lot of breakthroughs being made, that seems like a thing that should have been unlikely. I wonder how you respond. Yeah, so for this, I will simply remind a few facts that may have been forgotten. And I think these facts which provide the context, I think they explain the situation. So the context was that we were fundraising at a 32 billion valuation. And then Meta came in and offered to acquire us. And I said, no, but my former co-founder, like in some sense, said yes. and as a result he also was able to enjoy from a lot of near-term liquidity and he was the only person from SSI to join Meta.

1:15:48It sounds like SSI's plan is to be a company that is at the frontier when you get to this very important period in human history where you have superhuman intelligence and you have these ideas about how to make superhuman intelligence go well but other companies will be trying their own ideas. Because what distinguishes SSI's approach to making superintelligence go well? The main thing that distinguishes SSI is its technical approach. So we have a different technical approach that I think is worthy. And we are pursuing it. I maintain that in the end, there will be a convergence of strategies. so I think there will be a convergence of strategies where at some point as AI becomes more powerful it's going to become more or less clearer to everyone what the strategy should be and it should be something like yeah you need to find some way to talk to each other and you want your first actual like real super intelligent AI to be aligned and somehow be you know care for sentient life care for people democratic one of those some combination of thereof and i think this is the condition that everyone should strive for and that's what the sci is striving for and i think that this time if not already all the other companies will realize and they're striving towards the same thing.

1:17:23And we'll see. I think that the world will truly change as AI becomes more powerful. And I think a lot of these forecasts will, like, I think things will be really different and people will be acting really differently. Speaking of forecasts, what are your forecasts to this system you're describing which can learn as well as a human and subsequently, as a result, become superhuman? I think like five to 20. Five to 20 years? Mm-hmm. So I just want to unroll your, how you might see the world coming. It's like we have a couple more years where these other companies are continuing the current approach and it stalls out.

1:18:02And stalls out here meaning they earn no more than low hundreds of billions in revenue. Or how do you think about what stalling out means? Yeah. I think it could stall out and I think stalling out will look like, it will all look very similar. among all the different companies, something like this. I'm not sure because I think even with stolen out, I think these companies could make a stupendous revenue. Maybe not profits because they will need to work hard to differentiate each other from themselves, but revenue definitely. But something in your model implies that when the correct solution does emerge, there will be convergence between all the companies.

1:18:50and I'm curious why you think that's the case. Well, I was talking more about convergence on their largest strategies. I think eventual convergence on the technical approach is probably going to happen as well. But I was alluding to convergence to the largest strategies. What exactly is the thing that should be done? I just want to better understand how you see the future enrolling. So currently we have these different companies and you expect their approach to continue generating revenue, but not get to this human-like learner. So now we have these different forks of companies. we have you, we have thinking machines, there's a bunch of other labs.

1:19:22Yes. And maybe one of them figures out the correct approach. But then the release of their product makes it clear to other people how to do this thing. I think it won't be clear how to do it thing, but it will be clear that something different is possible. Right. And that is information. And I think people will then be trying to figure out how that works. I do think, though, that one of the things that I think, you know, not addressed here, not discussed, is that with each increase in the AI's capabilities, I think there will be some kind of changes, but I don't know exactly which ones in how things are being done.

1:20:03And so, like, I think it's going to be important, yet I can't spell out what that is exactly. And how are the, by default, you would expect the company that has, the model company that has that model to be getting all these gains because they have the model that is learning how to do all, has the skills and knowledge that it's building up in the world. What is the reason to think that the benefits of that would be widely distributed and not just end up at whatever model company gets this continuous learning loop going first? Like I think that empirically what happens, so here is what I think is going to happen.

1:20:39Number one, I think empirically when let's look at how things have gone so far with the AIs of the past. So one company produced in advance and the other company scrambled and produced some similar things after some amount of time. And they started to compete in the market and push their prices down. And so I think from the market perspective, I think something similar will happen there as well even if someone we are talking about the good world by the way where what's the good world what's the good world where we have these powerful human like learners that are also like and by the way maybe there's another thing we haven't discussed on the on the spec of the super intelligent AI that I think is worth considering is that you make it narrow.

1:21:42It can be useful and narrow at the same time. So you can have lots of narrow, super intelligent AIs. But suppose you have many of them and you have some company that's producing a lot of profits from it. And then you have another company that comes in and starts to compete. And the way the competition is going to work is through specialization. I think what's going to happen is that the way competition, like competition loves specialization. And you see it in the market, you see it in evolution as well. So you're going to have lots of different niches and you're going to have lots of different companies who are occupying different niches in this kind of world.

1:22:26We might say, yeah, like one AI company is really quite a bit better at some area of really complicated economic activity and a different company is better at another area. and the third company is really good at litigation. But is this contradicted by what human learning implies? Is that like it can learn? It can, but you have accumulated learning. You have a big investment. You spent a lot of compute to become really, really, really good. Really phenomenal at this thing. And someone else spent a huge amount of compute and a huge amount of experience to get really, really good at some other thing.

1:22:59You apply a lot of human learning to get there, but now you are at this high point where someone else would say, look, I don't want to start learning what you've learned to do for this. I guess that would require many different companies to begin at the human-like, continual learning agent at the same time so that they can start their different research in different branches. But if one company, you know, gets that agent first or gets that learner first, it does then seem like, well, you know, like if you just think about every single job in the economy, you just have instance learning each one seems tractable for a company.

1:23:40Yeah, that's a valid argument. My strong intuition is that it's not how it's going to go. My strong intuition is that, yeah, like the argument says it will go this way, but my strong intuition is that it will not go this way. That this is the, you know, in theory, there is no difference between theory and practice, and practice there is, and I think that's going to be one of those. a lot of people's models of recursive self-improvement literally explicitly state we will have a million ILLIAs in a server that are coming up with different ideas and this will lead to a super intelligence emerging very fast do you have some intuition about how parallelizable the thing you are doing is what are the gains from making copies of ILLIA?

1:24:23I don't know I think I think there'll definitely be there'll be diminishing returns because you want people who think differently rather than the same. I think that if they were literal copies of me, I'm not sure how much more incremental value you'd get. I think that, but people who think differently, that's what you want. Why is it that it's been, if you look at different models, even released by totally different companies, trained on potentially non-overlapping data sets, it's actually crazy how similar LLMs are to each other? Maybe the data sets are not as non-overlapping company as it seems.

1:25:02But there's some sense that it's like, even if an individual human might be less productive than the future AI, maybe there's something to the fact that human teams have more diversity than teams of AIs might have. But how do we elicit meaningful diversity among AI? So I think just raising the temperature just results in gibberish. I think you want something more like different scientists have different prejudices or different ideas. How do you get that kind of diversity among AI agents? So the reason there has been no diversity, I believe, is because of pre-training. All the pre-trained models are the same, pretty much, because they pre-train on the same data.

1:25:39Now, RL and post-training is where some differentiation starts to emerge because different people come up with different RL training. Yeah. And then I've heard you hint in the past about self-play as a way to either get data or match agents to other agents of equivalent intelligence to kick off learning. How should we think about why there's no public proposals of this kind of thinking working with LLMs? I would say there are two things to say. I would say that the reason why I thought self-play was interesting is because it offered a way to create models using compute only without data. And if you think that data is the ultimate bottleneck, then using compute only is very interesting.

1:26:28So that's what makes it interesting. Now, the thing is that self-play, at least the way it was done in the past, when you have agents which somehow compete with each other, it's only good for developing a certain set of skills. It is too narrow. It's only good for like negotiation, conflict, certain social skills, strategizing, that kind of stuff. And so if you care about those skills, then self-play will be useful. Now, actually, I think that self-play did find a home, but just in a different form, in a different form. So things like debate, prove a verifier, you have some kind of an LLM as a judge, which is also incentivized to find mistakes in your work.

1:27:21You could say this is not exactly self-play, but this is, you know, a related adversarial setup that people are doing, I believe. and really self-lay is an example of, is a special case of more general, like competition between agents, right? The response, the natural response to competition is to try to be different. And so if you were to put multiple agents and you tell them, you know, you all need to work on some problem and you're an agent and you're inspecting what everyone else is working, you're going to say, well, if they're already taking this approach, it's not clear I should pursue it.

1:27:54I should pursue something differentiated. And so I think that something like this could also create an incentive for a diversity of approaches. Yeah. Final question. What is research taste? You're obviously the person in the world who is considered to have the best taste in doing research in AI. You were the co-author on many of the biggest, the biggest things that have happened in the history of deep learning from AlexNet to GPT-3 to so on. what is it that how do you characterize how you come up with these ideas i can answer so i can comment on this for myself i think different people do it differently but one thing that guides me personally is an aesthetic of how ai should be by thinking about how people are but thinking correctly like it's very easy to think about how people are incorrectly but what does it mean to think about people correctly so I'll give you some examples the idea of the artificial neuron is directly inspired by the brain and it's a great idea why?

1:29:12because you say sure the brain has all these different organs it has the folds but the folds probably don't matter why do we think that the neurons matter? because there is many of them it kind of feels right so you want the neuron you want some kind of local learning rule that will change the connections you want some local learning rule that will change the connections between the neurons right it feels plausible that the brain does it the idea of the distributed representation the idea that the brain you know the brain responds to experience and neural net should learn from experience not response, the brain learns from experience the neural net should learn from experience and you kind of ask yourself is something fundamental or not fundamental how things should be.

1:29:55And I think that's been guiding me a fair bit, kind of thinking from multiple angles and looking for almost beauty, beauty, simplicity, ugliness. There's no room for ugliness. It's just beauty, simplicity, elegance, correct inspiration from the brain. And all of those things need to be present at the same time. And the more they are present, the more confident you can be in a top-down belief. And then the top-down belief is the thing that sustains you when the experiments contradict you because if you just trust the data all the time well sometimes you can be doing a correct thing but there's a bug but you don't know that there is a bug how can you tell that there is a bug how do you know if you should keep debugging or you conclude it's the wrong direction well it's the top down well how should you can say the things have to be this way right something like this has to work therefore we got to keep going.

1:30:45That's the top down. And it's based on this like multifaceted beauty and inspiration by the brain. All right. We'll leave it there. Thank you so much. Thank you so much. All right. Appreciate it. That was great. Yeah. I enjoyed it. Yes, me too.

1:31:04Thanks for listening to this episode of the A60Z podcast. If you liked this episode, be sure to like, comment, subscribe, leave us a rating or review and share it with your friends and family. For more episodes, go to YouTube, Apple Podcasts, and Spotify. Follow us on X at A16Z and subscribe to our Substack at a16z.substack.com. Thanks again for listening, and I'll see you in the next episode. This information is for educational purposes only and is not a recommendation to buy, hold, or sell any investment or financial product. This podcast has been produced by a third party and may include paid promotional advertisements, other company references, and individuals unaffiliated with A16Z.

1:31:44Such advertisements, companies, and individuals are not endorsed by AH Capital Management LLC, A16Z, or any of its affiliates. Information is from sources deemed reliable on the date of publication, but A16Z does not guarantee its accuracy.