The Road to Autonomous Intelligence with Andrej Karpathy

20:11So I think you have to be just very careful to inject the entropy, maintain the distribution, and that's the hard part that I think maybe people aren't sufficiently appreciating as much in general. So I think basically synthetic data, absolutely the future, we're not going to run out of data, is my impression. I just think you have to be careful. What do you think we are learning now about human cognition from this research? I don't know if we're learning much of this. One could argue that figuring out the shape of reasoning traces we want, for example, is instructive to actually understand how the brain works.

20:38I would be careful with those analogies, but in general, I do think that it's a very different kind of thing. But I do think that there are some analogies you can draw. So as an example, I think transformers are actually better than the human brain in a bunch of ways. I think they're actually a lot more efficient system. And the reason they don't work as good as the human brain is mostly data issue, roughly speaking, as the first order approximation, I would say. And actually, as an example, transformer memorizing sequences is so much better than humans. If you give it a sequence and you do a single forward-backward pass on that sequence, then if you give it the first few elements, it will complete the rest of the sequence.

21:08It memorized that sequence. And it's so good at it. If you gave a human a single presentation of a sequence, there's no way that you can remember that. And so the transformers actually... I do think there's a good chance that the gradient-based optimization, the forward-backward update that we do all the time for training neural nets is actually more efficient than the brain in some ways. And these models are better. They're just not yet ready to shine. But in a bunch of cognitive sort of aspects, I think they might come out. With the right inputs, they will be better. That's inherently true of computers for all sorts of applications, right?

21:39And putting memory to your point. Yeah, exactly. And I think human brains just have a lot of constraints. You know, the working memory is very small. I think transformers have a lot, lot bigger working memory. And this will continue to be the case. They're much more efficient learners. The human brains function under all kinds of constraints. It's not obvious that human range is backpropagation. It's not obvious how that would work. It's a very stochastic, sort of dynamic system. It has all these constraints it works under, so ambient conditions, etc. So I do think that what we have is actually potentially better than the brain, and it's just not there yet.

22:11How do you think about human augmentation with different AI systems over time? Do you think that's a likely direction? Do you think that's unlikely? Augmentation? Augmentation of people with AI models. Oh, of course. I mean, but in what sense maybe? I think in general, absolutely. Because I mean, there's an abstract version of it you're using as a tool. That's the external version. There's the, you know, the merger scenario. Oh. You know, a lot of people end up talking about. Yeah, yeah. I mean, we're already kind of merging. The thing is like there's a, you know, there's the IO bottleneck. Sure.

22:39But for the most part, you know, at your fingertips, if you have any of these models. Yeah, but that's a little bit different because I mean, people have been making that argument for I think 40, 50 years where technological tools are just extension of human capabilities, right? Yeah, the computer is the bicycle for human mind. Correct. Yeah, exactly. So there's a subset of the AI community that thinks that, for example, the way that we subsume some potential conflict with future AI or something else would be through some form of... Yeah, like the Neuralink pitch, et cetera. Exactly. Yeah, I don't know what this merger looks like yet, but I can definitely see that you want to decrease the IO to tool use.

23:12And I see this as kind of like an exocortex while building on top of our neocortex, right? And it's just the next layer, and it just turns out to be in the cloud, et cetera, but it is the next layer of the brain. Yeah, Accelerando, a book from the early 2000s, has a version of this where basically everything is substantiated in a set of goggles that are computationally attached to your brain that you wear. And then if you lose them, you must feel like you're losing a part of your persona or memory. I think that's very likely, yeah. And today the phone is already almost that. And I think it's going to get worse.

23:38When you put your techno stuff away from you, you're just like naked human in nature. Well, you lose part of your intelligence. It's very anxiety-inducing. A very simple example of that is just maps, right? So a lot of people now, I've noticed, can't actually navigate their city very well anymore because they're always using turn-by-turn direction. And if we have this, for example, like Universal Translator, which I don't think is too far away, you'll lose the ability to speak to people who don't speak English if you just put your stuff away. I'm very comfortable repurposing that part of my brain to do further research.

24:06I don't know if you saw the video of the kid that has a magazine and is trying to swipe on the magazine. What's fascinating to me about it is this kid doesn't understand what comes with nature and what's technology on top of the nature. Because it makes it so transparent. And I think this might look similar where people will just start assuming the tools. And then when you take them away, you realize like, I guess like people don't know what's technology and what's not. If you're wearing this thing that's always translating everyone or like doing stuff like that for you, then maybe people like lose the...

24:33Basic cognitive abilities may not exist. Yeah. I think it exists. Yeah. It's like, whoa, like by nature. Yeah, yeah. We're going to specialize. You can't understand people who speak Spanish? Like what the hell? Or like when you go to objects, like in Disney, all the objects are alive. And I think we are going to potentially come to that kind of a world where why can't I talk to things? Like already today you can talk to Alexa and you can ask her for things and so on. Yeah, I've seen some toy companies like that where they're basically trying to embed an LLM in a toy so they can interact with a child.

24:59Yeah, like isn't it strange that when you go to a door you can't just say open? Like what the hell? Another favorite example of that, I don't know if you saw either Demolition Man or iRobot. People make fun of the idea that you, yeah, like you can't just talk to things and what the hell. If we're talking about an exocortex, that feels like a pretty fundamentally important thing to democratize access to. How do you think the current market structure of what's happening in LLM research, there's a small number of large labs that actually have a shot at the next generation progressing training. How does that translate to what people have access to in the future?

25:40So what you're kind of alluding to maybe is the state of the ecosystem, right? So we have kind of like an oligopoly of a few closed platforms, and then we have an open platform that is kind of like behind. So like MetaLama, etc. And this is kind of like mirroring the open source kind of ecosystem. I do think that when this stuff starts to, when we start to think of it as like an exocortex, so there's a saying in crypto, which is like, not your keys, not your tokens. Like, is it the case that if it's like not your weights, not your brain? That's interesting because a company is effectively controlling your exocortex and therefore a big part of your...

26:08It starts to feel kind of invasive. If this is my exocortex... I think people care much more about ownership. Yeah, you realize you're renting your brain. Like it seems strange to rent your brain. The thought experiment is like, are you willing to give up ownership and control to rent a better brain? Because I am. Yeah. So I think that's the trade-off. I think we'll see how that works. But maybe it's possible to like, by default, use the closed versions because they're amazing. But you have a fallback in various scenarios. And I think that's kind of like the way things are shaping up today even.

26:37When APIs go down on some of the closed source providers, People start to implement fallbacks to the open ecosystems, for example, that they fully control and they feel empowered by that. So maybe that's just the extension of what it will look like for the brain is you fall back on the open source stuff should anything happen. But most of the time, you actually... So it's quite important that the open source stuff continues to progress. I think so, 100%. And this is not an obvious point or something that people maybe agree on right now, but I think 100%. I guess one thing I've been wondering about a little bit is what is the smallest performant model that you can get to in some sense, either in parameter size or however you want to think about it?

27:13And so I'm a little bit curious about your view because you've thought a lot about both distillation, small models, you know. I think it can be surprisingly small. And I do think that the current models are wasting a ton of capacity remembering stuff that doesn't matter. Like they remember SHA hashes, they remember like the ancient... Because the data set is not curated the best. Yeah, exactly. And I think this will go away. And I think we just need to get to the cognitive core. And I think the cognitive core can be extremely small. And it's just this thing that thinks. And if it needs to look up information, it knows how to use different tools.

27:43Is that like 3 billion parameters? Is that 20 billion parameters? I think even a billion suffices. We'll probably get to that point. And the models can be very, very small. And I think the reason they can be very small is fundamentally, I think, just like distillation works. It may be like the only thing I would say. Distillation works like surprisingly well. Distillation is where you get a really big model or a huge amount of compute or something like that, supervising a very small model. And you can actually stuff a lot of capability into a very small model. Is there some sort of mathematical representation of that or some information theoretical formulation of that?

28:17Because it almost feels like you should be able to calculate that now in terms of what's the... Maybe one way to think about it is we go back to the internet data set, which is what we're working with. The internet is like 0.001 % cognition and like 99.99 % of like information is just like, you know, garbage. Yeah. And I think most of it is not useful to the thinking part. Yeah. I guess maybe another way to frame the question is like, is there a mathematical representation of cognitive capability relative to model size? Or how do you capture cognition in terms of, you know, here's the min or max relative to what you're trying to accomplish?

28:51And maybe there's no good way to represent that. So I think maybe a billion parameters gets you sort of like a good cognitive core. I think probably right. I think even one billion is too much. I don't know. We'll see. It's very exciting given if you think about, well, you know, it's a question of like on an edge device versus on the cloud, but. And also this raw cost of using the model and everything. Yeah, it's very exciting. Right. But at less than a billion parameters, I have my exocortex on a local device as well. Yeah. And then probably it's not a single model, right? Like it's interesting to me to think about what this will actually play out like, because I do think you You want to benefit from parallelization.

29:25You don't want to have a sequential process. You want to have a parallel process. And I think companies, to some extent, are also kind of like parallelization of work. But there's a hierarchy in a company because that's one way to... You have the information processing and the reductions that need to happen within the organization for information. So I think we'll probably end up with companies for other LLMs. I think it's not unlikely to me that you have models of different capabilities specialized to various unique domains. maybe there's a programmer, et cetera, and it will actually start to resemble companies to a very large extent.

29:56So you'll have the programmer and the program manager and, you know, similar kinds of roles of LLMs working in parallel and coming together and orchestrating computation on your behalf. So maybe it's not correct to think about. It's more like a swarm. You're actually like a swarm. It's like an ecosystem. It's like a biological ecosystem where you have specialized roles and niches. And I think it will start to resemble that. You have automatic escalation to other parts of the swarm depending on the difficulty of the problem and the specialization. The CEO is like a really brilliant cloud model.

30:24Yeah. But the workers can be a lot cheaper, maybe even open source models or whatnot. And my cost function is different from your cost function. Yeah, so that could be interesting. You left OpenAI. You're working on education. You've always been an educator. Like, why do this? I would start with I've always been an educator, and I love learning, and I love teaching. And so it's kind of just like a space that I've been very passionate about for a long time. And then the other thing is, I think one macro picture that's kind of driving me is, I think there's a lot of activity in like AI. And I think most of it is to kind of like replace or displace people, I would say.

30:57It's in the theme of like sliding away the people. But I'm always more interested in anything that kind of empowers people. And I feel like I'm kind of on a high level like team human. And I'm interested in things that AI can do to empower people. And I don't want the future where people are kind of on a side of automation. I want people to be very in an empowered state and I want them to be amazing, much more amazing than today. And then other aspects that I find very interesting is like, how far can a person go if they have the perfect tutor for all the subjects? And I think people could go really far if they had the perfect curriculum for anything.

31:28And I think we see that with, you know, if some rich people maybe have tutors and they do actually go really far. And so I think we can approach that with AI or even let's surpass it. There's very clear literature on that actually from the 80s, right? Where one-on-one tutoring, I think, helps people get one standard deviation better. Two, Bloom. Is it two? Yeah, it's the Bloom stuff. Yeah, exactly. There's a lot of really interesting precedents on that. How do you actually view that as substantiating through the lens of AI? Or what's the first types of products that will really help with that?

31:58Or, you know, because there's books like The Diamond Age, where they talk about the young lady's illustrated primer and all that kind of stuff. So I would say I'm definitely inspired by aspects of it. So like in practice, what I'm doing is trying to currently build a single course. and I want it to be just like the course you would go to if you want to learn AI. I think the problem with basically is like I've already taught courses like I taught 231N at Stanford and that was the first deep learning class and was pretty successful. But the question is like, how do you actually like really scale these classes?

32:23Like how do you make it so that your target audience is maybe like 8 billion people on earth? And they're all speaking different languages and they're all different capability levels, et cetera. So a single teacher doesn't scale to that audience. And so the question is how do you use AI to sort of like do the scaling of a really good teacher? And so the way I'm thinking about it is the teacher is kind of doing a lot of the course creation and the curriculum, because currently at the current AI capability, I don't think the models are good enough to create a good course, but I think they're good to become the front end to the student and interpret the course to them.

32:53And so basically the teacher doesn't go to the people and the teacher is not the front end anymore. The teacher is on the back end designing materials on the course and the AI is the front end and it can speak all the different languages and it's kind of like takes you through the course. Should I think of that as like the TA type experience or is that not a good analogy here? That is like one way I'm thinking about it. It's AITA. I'm mostly thinking of it as like this front end to the student, and it's the thing that's actually interfacing with the student and taking them through the course. And I think that's tractable today, and it just doesn't exist, and I think it can be made really good.

33:24And then over time, as the capability increases, you would potentially refactor the setup in various ways. I like to find things where, like the AI capability today and having a good model of it, And I think a lot of companies that maybe don't quite understand intuitively where the capability is today, and then they end up kind of like building things that are kind of like too ahead of what's available or maybe not ambitious enough. And so I think, I do think that this is kind of a sweet spot of what's possible and also really interesting and exciting. I want to go back to something you said that I think is very inspiring, especially coming from like your background and understanding of where exactly we are in research, which is essentially like we do not know what the limits of human performance from a learning perspective are.

34:04given much better tooling. And I think there's like a very easy analogy to, we just had the Olympics like a month ago, right? And, you know, a runner and it's the very best mile time or pick any sport today is much better than it was putting aside performance enhancing drugs like 10 years ago, just because like you start training earlier, you have a very different program. We have much better scientific understanding. We have technique, we have gear. The fact that you believe like we can get much further as humans, if we're starting with like the tooling in the curriculum is amazing. Yeah, I think we haven't even scratched what's possible at all.

34:36So I think there's two dimensions basically to it. Number one is the globalization dimension of like, I want everyone to have really good education. But the other one is like, how far can a single person go? I think both of those are very interesting and exciting. Usually when people talk about 101 learning, they talk about the adaptive aspect of it, where you're challenging a person at the level that they're at. Do you think you can do that with AI today? Or is that something for the future? And it's more, today it's about reach and multiple languages. I think the Lohenberg is things like, for example, different languages.

34:59Super Lohenberg. I think the current models are actually really good at translation, basically. and can target the material and translate it at the spot. So I think a lot of things are low-lying fruit. This adaptability to a person's background, I think, is not at the low-lying fruit, but I don't think it's too high up or too much away. But that is something you definitely want because not everyone is coming in with the same background. And also what's really helpful is if you're familiar with some other disciplines in the past, then it's really useful to make analogies to the things you know.

35:25And that's extremely powerful in education. So that's definitely a dimension you want to take advantage of. But I think that starts to get to the point where it's not obvious and needs some work. I think like the easy version of it is not too far where you can imagine just prompting model. It's like, oh, hey, I know physics or I know this and you probably get something. But I guess what I'm talking about is something that actually works, not something that like you can demo and work sometimes. So I just mean like it actually really works in the way a person would. Yeah, and that's the reason I was asking about adaptability because also people learn at different rates or certain things they find challenging that others don't or vice versa.

35:56And so it's a little bit of how do you modulate relative to that context? And I guess you could have some reintroduction of what the person is good or bad at into the model over time. That's the thing with AI. I feel like a lot of these capabilities are just kind of like prompt away. So you always get like demos, but like, do you actually get a product? You know what I mean? So in this sense, I would say the demo is near, but the product is far. So one thing we were talking about earlier, which I think is really interesting, is sort of lineages. That happens in the research community where you come from certain labs and everybody gossips about being from each other's labs.

36:25I think a very high proportion of Noble Laureates actually used to work in a former Noble Laureates lab. So there's some propagation of, I don't know, if it's culture or knowledge or branding or what. In an AI education-centric world, how do you maintain lineage or does it not matter? How do you think about those aspects of propagation of network and knowledge? I don't actually want to live in a world where lineage matters too much, right? So I'm hoping that AI can help you destroy that structure a little bit. It feels like kind of gatekeeping by some finite scarce resource, which is like, oh, there's a finite number of people who have this lineage, et cetera.

36:56So I feel like it's a little bit of that aspect. So I'm hoping it can destroy that. It's definitely one piece, like actual learning, one piece pedigree. Yeah. Well, it's also the aggregation of, it's a cluster effect, right? It's like, why is all of the, or much of the AI community in the Bay Area? Or why is most of the fintech community in New York? Yeah. And so I think a lot of it is also just you're clustering really smart people with common interests and beliefs. And then they kind of propagate from that common core. And then they share knowledge in an interesting way. You've got to agree a lot of that behavior has shifted online to some extent, particularly for younger people.

37:26I think one aspect of it is kind of like the educational aspect where like if you're part of a community today You're getting a ton of education and apprenticeship, etc Which is extremely helpful and gets you to a point of empowered state in that area I think the other piece of it is like the cultural aspect of what you're motivated by and what you want to work on What does the culture prize and what do they put on a pedestal and what do they kind of like worship basically? So in academic world for example is the H index Everyone cares about the H index the amount of papers you publish, etc And I was part of that community and I saw that and I feel like now I've come to different places and there's different idols in all the different communities.

37:57And I think that has a massive impact of what people are motivated by and where they get their social status and what actually matters to them. I also was, I think, part of different communities like growing up in Slovakia, also a very different environment. Being in Canada, also a very different environment. What mattered there? Hockey. Sorry. Thank you. Hockey. Yeah, hockey. As an example, I would say in Canada, I was in the University of Toronto and Toronto, I don't think it's a very entrepreneurial pill environment. it doesn't even occur to you that you should be starting companies. I mean, it's not something that people are doing.

38:29You don't know friends who are doing it. You don't know that you're supposed to be looking up to it. People aren't like reading books about all the founders and talking about them. It's just not a thing you aspire to or care about. And what everyone is talking about, oh, is where are you getting your internship? Where are you going to work afterwards? And it's just accepted that there's a fixed set of companies that you are supposed to pick from and just align yourself with one of them. And that's like what you look up to or something like that. So these cultural aspects are extremely strong and maybe actually the dominant variable.

38:54Because I almost feel like today, the education aspects, I think, are the easier one. Like, a ton of stuff is already available, etc. So I think mostly it's a cultural aspect that you're part of. So on this point, like, one thing you and I were talking about a few weeks ago is, and I think you also posted online about this, there's a difference between learning and entertainment. And learning is actually supposed to be hard. And I think it relates to this question of, like, you know, status. And what, like, status is a great motivator. Like, who the idol is. how much do you think you can change in terms of motivation through systems like this if that's like a blocking factor?

39:32Are you focused on give people the resources such that they can get as far as possible in the sequence for their own capability as they can like further than any other point in history, already inspirational? Or do you actually want to change how many people want to learn or at least bring themselves down the path? Want is a loaded word. I would say like I want to make it much easier to learn. and then maybe it is possible that maybe people don't want to learn. I mean, today, for example, people want to learn for practical reasons, right? Like they want to get a job, etc., which makes total sense.

40:01So in a pre-AGI society, education is useful, and I think people will be motivated by that because they're climbing up the ladder economically, etc. Oh, but in the post-AGI society, we're just all going to be... In the post-AGI society, I think education is entertainment to a much larger extent. Including, like, successful outcomes education, right? not just letting the content wash over you. Yes, I think so. Outcomes being like understanding, learning, being able to contribute new knowledge or however you define it. I think it's not an accident that if you go back 200 years, 200 years, the people who were doing science were nobility or people of wealth.

40:34We will all be nobility learning with Andre. Yeah. I do think that I see it very much equivalent to your quote earlier. I feel like learning something is kind of like going to the gym, but for the brain, right? Like it feels like going to the gym. I mean, going to the gym is fun. People like to lift, et cetera. Some people don't go to the gym. No, no, no. Some people do, but it takes effort. Yeah, it takes effort, but it's also kind of fun. And you also have a payoff. You feel good about yourself in various ways, right? And I think education is basically equivalent to that. So that's what I mean when I say education should not be fun, etc.

41:02I mean, it is kind of fun, but it's like a specific kind of fun, I suppose, right? I do think that maybe in a post-AGI world, what I would hope happens is people actually, they do go to the gym a lot, not just physically, but also mentally. And it's something that we look up to as being highly educated and also, you know, just, yeah. Can I ask you one last question about Eureka just because I think it'll be interesting to people? Like, who is the audience for the first course? The audience for the first course, I'm mostly thinking of this as like an undergrad level course. So if you're doing undergrad in a technical area, I think that would be kind of the ideal audience.

41:36I do think that what we're seeing now is we have this like antiquated concept of education where you go through school and then you graduate and go to work, right? Obviously, this will totally break down, especially in a society that's turning over so quickly. people are going to come back to school a lot more frequently as the technology changes very, very quickly. So it is kind of like undergrad level, but I would say anyone at that level at any age is kind of in scope. I think it will be very diverse in age, as an example, but I think it is mostly people who are technical and mostly actually want to understand it to a good amount.

42:09When can they take the course? I was hoping it would be late this year. I do have a lot of distractions that are piling on, but I think probably early next year is kind of like the timeline. Yeah, I'm trying to make it very, very good. And yeah, it just takes time to get there. I have one last question, actually, that's pseudo-related to that. If you have little kids today, what do you think they should study in order to have a useful future? There's a correct answer in my mind. And the correct answer is mostly like, I would say like math, physics, CS kind of disciplines. And the reason I say that is because I think it helps for just thinking skills.

42:42It's just like the best thinking skill core is my opinion. Of course, I have a specific background, etc., so I would think this. But that's just my view on it. I think me taking physics classes and all these other classes just shaped the way I think, and I think it's very useful for problem-solving in general, etc. And so if we're in this world where, pre-AGI, this is going to be useful, post-AGI, you still want empowered humans who can function in any arbitrary capacity. And so I just think that this is just the correct answer for people and what they should be doing and taking. And it's either useful or it's good.

43:14And so I just think it's the right answer. And I think a lot of the other stuff you can tack on a bit later, but the critical period where people have a lot of time and they have a lot of kind of like attention and time, I think should be mostly spent on doing these kinds of simple manipulation-heavy tasks and workloads, not memory-heavy tasks and workloads. I did a math degree and I felt like there was a new groove being carved into my brain as I was doing that. And it's a harder groove to carve later. And I would, of course, put in a bunch of other stuff as well. Like I'm not opposed to all the other disciplines, etc.

43:43I think it's actually beautiful to have a large diversity of things, but I do think 80 % of it should be something like this. One, we're not efficient memorizers compared to our tools. Thank you for doing this. It's so much fun. Yeah, it's amazing. Yes. It's great to be here. Find us on Twitter at NoPriorsPod. Subscribe to our YouTube channel if you want to see our faces. Follow the show on Apple Podcasts, Spotify, or wherever you listen. That way you get a new episode every week. And sign up for emails or find transcripts for every episode at no-priors.com. We'll be right back.