How François Chollet Is Building A New Path To AGI

24:37François Chollet:And I think their task cost was a lot more efficient too. And that approach they basically took is similar to this. I think they built the harnesses on top of it in order to get the LLMs to go and build different tasks and program through it. Which then, for me, I was like, wow, is this batch? During the batch, they only worked on it for a couple of months and they were able to saturate the benchmark that has been around for a long time. It's like something special is happening. Yeah, yeah. There's a lot of progress right now that's driven by custom harnesses around the task. And the harness is basically a way for the human programmer to input into the model, like higher level solution strategies, basically.

25:23I mean, to me, the fact that you need humans to engineer these harnesses is also a sign that we're short of AGI today. Because if we had AGI, you know, AGI would just make its own harness. It would not need to be told how to solve a problem. It would just figure it out. But it is very effective. Like harnesses, I don't think they get us closer to HGI in any sense. But it's a very valuable area of research because that can lead to task automation at scale. YC's next batch is now taking applications. Got a startup in you? Apply at ycombinator.com slash apply. It's never too early and filling out the app will level up your idea.

26:00Okay, back to the video.

26:02François Chollet:Can you tell us about then what V3 is going to measure that's just got released? Yeah, absolutely. So if you look at V1, V2, it was really focusing on your ability to produce causal models of a pattern that was just given to you, like the data was given to you. So it was static, it was passive and really focused on modeling. And V3, it's completely different. We are trying to measure agentic intelligence. So it's interactive, it's active, like the data is not provided to you, you must go get it. The idea is that your agent is dropped into a new environment, which is kind of like a mini video game.

26:44And it's not provided any instructions. It's not told what to do. It's not told what the goal even is or what the controls even are. And it must figure out everything on its own via trial and error. So we are not just measuring the AI's ability to model its environment. we're also looking at its exploration efficiency, its ability to acquire goals on its own, like goal setting, and of course its ability to plan through the model of the environment that's created and to execute the plan. And so together, all of these abilities, we call that agentic intelligence. And we are looking for AI systems that could learn to play these games and crack them with the same degree of action efficiency as a human.

27:35If you look at the human, they are dropped into this new environment. They try a few things. They start understanding how things work. They can solve the environment in a few hundreds to thousands of actions. We're trying to look for AI systems that could match this efficiency. And by the way, we know that all of these test environments in Arc3 are solvable by humans with no prior training because we actually tested them on regular people. Yeah. At first you just see this screen and you have these keys available, but you know what they do. And you must figure out everything from scratch. And humans are really good at that, by the way.

28:12They're really good at exploring efficiently, making sense of something new and eventually cracking the game. And frontier models today, they are not very good at it. If the reasoning models cracked v1 and the reinforcement learning environments cracked v2, Do we need a new advance to crack v3? Do even the best techniques currently not work? Yeah, I'm pretty curious to see how Frontier Labs are going to react to v3 and how they're going to start to target it. It is designed to be more resistant to the same kind of targeting strategy as what we saw for v2 in particular. Of course, you can try to just make more Arc3-like games and then train your agents in them.

28:57But the thing is, we've deliberately tried to create a private set of environments that is significantly different from the public set. You can look at the public set, which is not actually giving you that much information about what's in the private set. In the private set, you will have very different games with very different concepts. And also, the public set is meant to be substantially easier. So your performance on the public set is not representative of how well the system will do on the private set. So for this reason, it's going to be harder to target. And that makes it a better test of fluid intelligence as opposed to a test of how much effort you put into cracking it.

29:35I'm so curious. How do you come up with these games? They're so creative. Yeah, we set up an entire video game studio to create them. So we got over 250 games. And they're pretty quick to play. Each game takes you maybe 10 minutes or a bit less to play from scratch, like upon first contact. And we have like 250 plus. And we set up this very productive game studio where we had any given week, we had multiple games in progress. We had like this pipeline, including design, implementation, review, human testing, and many iteration cycles to make sure that the game comes out right. Who's working in the studio?

30:20Right. Who are the creators? Yeah, we hired a team of game developers and we built our own game engine. Wow. So it's actually people who previously worked in the game, in the video game industry. That's right. That's right. So one thing to be in mind, though, is that the games in Ark 3 are unique, right? They're trying to not borrow elements, concepts from previous video games. They're built entirely on top of core knowledge priors, like things like just, you know, elementary knowledge, like basic physics, understanding of objects, understanding of the notion of agents, for instance, like an agent in objects with goals and intentions.

31:01But we are not incorporating any language, any cultural symbols like arrows, for instance, or the color green meaning go and color red meaning stop, that sort of thing. There's no external knowledge that's involved in these games.

31:17François Chollet:It's like one of those IQ tests that are just pattern matching, but now it has time series. Yeah. It's not just time series. It's interactive. You must create your own path through game space, right? You must, you know, in an IQ test like problem, like, you know, what Arc 1 and 2 is, the data that you must model is provided to you. You already have the data. You just need to find the causal rule to explain it. With Arc 3, you actually must gather the data. and you must do so efficiently. Like, of course, you could say, well, I'm just going to, you know, brute force mine the space of every possible game state, and then I find the solution.

31:56You cannot do that because if you try to do that, you would score extremely low, even if you manage to solve the level. Because you're scored on your efficiency, you must match human-level efficiency. It's funny. It's like almost coming full circle. This level of AGI with games sort of is the match pair to OpenAI writing. I mean, you know, Tom Brown, one of the co-founders of Anthropic, had to write like the harness code to allow like, you know, pre-GPT AI at OpenAI to play StarCraft. Yeah, yeah. OpenAI worked on, in particular, on Dota 2. The OpenAI 5 model, which was, if I recall correctly, so this was like not just pre-GPT, but mostly pre-Transformers because they were working with a stack of LSTM.

32:44Yeah. Layers, if I recall correctly. And even before Pennyi, DeepMine worked a lot on video game, solving video games via DeepIsle. And they were the first to do Atari games, right, back in 2013. They were very, very early, very visionary in that sense to work on this problem so early with these methods, which are still very modern methods. So the big difference is that if you look at Atari games, for instance, or even Dota, you're training on the same environment as what you use for testing. So effectively, you're just trying to memorize the best strategies. You're trying to, at training time, explore the full space of possible game states and productionize, operationalize that knowledge into the model.

33:38And then at inference time, you're basically just recalling that knowledge. and that's explicitly what you're trying to avoid with ARK 3. You're not playing games that you've seen before. You're not playing games that you've been trained on like for millions of hours, like the OpenAI 5 model, for instance, was playing a restricted version of Dota 2 and it was trained on like tens of thousands of hours of gameplay, effectively. I think maybe in millions, but it's just an insane amount of trained data. With ARK 3, you're being evaluated on games that you're seeing for the very first time. And every action you spend exploring is counted towards your efficiency score, right?

34:19So you're really focused on measuring fluid intelligence, your ability to efficiently explore, efficiently produce a world model of the environment and then use this model to infer goals, plan towards these goals, and eventually crack the game. One of the arguments for India is that you're able to do all of the intelligent tasks for, you know, an ARC task might be like 0.3 cents for an ARC task, but, you know, for the same task on a foundation model with LLMs, it's, you know, a dollar to$10. And then there's this other aspect that we've been tracking where it seems like more and more intelligence, at least on the LLM side, can be distilled down into smaller and smaller models.

35:07And so on the one hand, like they're scaling up, but then they're like distilling smarter and smarter small models. I guess your approach might indicate that it's not billions of parameters. Like the, you know, NDIA achieving AGI might not be, you know, sort of inherently a scale thing at all. There's a platonic ideal of the NDIA model that achieves AGI. Yeah. Do you ever think about it in terms of like, well, it would fit on a floppy disk? Well, okay, there are two things to separate. There's the sort of like fluid intelligence engine. I think it's going to be a very, very small code race and a very small set of models that's suited with it.

35:47And it's probably going to be on the order of megabytes, right? And then you have the knowledge base, so to speak, that's going to be layered below this fluid intelligence engine. like, you know, fluid intelligence has to draw on some knowledge, and that knowledge is going to take up a lot more space. I think it's important to differentiate the two. I do believe that, you know, when you create a GI, retrospectively it will turn out that it's a code base that's less than 10 ,000 lines of code, and that if you had known about it back in the 1980s, you could have done a GI back then, using the computer resources available back then.

36:28Wow, that's a crazy prediction. I think retrospectively this will turn out to be true. Wow. So it was just like hiding under our noses in plain sight for like 40 years. It took us like 40 years to figure it out. Yeah, that's right. That's right. Well, that second thing sounds like Douglas Lenat's like psych project, or is that the wrong way to think about it? It's like there's sort of knowledge about the world. Yeah. And then there's methods. Like the program, what I hear is like the program might be 10 ,000 lines and then it operates on like... On knowledge base, it's very large. So the problem with Psyche, I mean, there were many issues with it, but one of the big issues is that there was no learning involved.

37:04Yeah, it's just the knowledge. The knowledge was encrafted. It was like purely symbolic knowledge and it was probably inaccurate. The way you want to be building a GI is that you want to be removing humans from the improvement loop as much as possible. You don't want a system where every improvement in system capability has to involve a human engineer doing something. And it's actually the strength of deep learning and foundation models is that you can just scale up the knowledge base. Like an LLM is effectively a knowledge base. It's a bank of, you know, modular vector programs that map patterns of input tokens to patterns of output tokens.

37:43And you can scale up that knowledge base by just adding training data and training compute with no further human involvement. I mean, of course, there's still a little bit of human involvement in making sure the training job completes, but it's minor. You've managed to remove humans from this improvement as much as possible. And that's also what we want for our system. We want a system that's self-improving, where the improvements are compounding, meaning that every time the system increases its capabilities, it's also increasing the rate at which it increases its capabilities. I think this is a PG-ism.

38:17It's like, I'm sorry the essay is so long. If I had more time, I would make it shorter. Yeah. When you're looking at a hard problem, it's actually harder to produce a short, elegant, concise solution than a messy, over-engineered solution. Yeah, you can brute force it, but the more elegant version is very, very short. That's kind of like what you said with how this might come about. Yeah, this is literally the shape of the type of AI approach we are creating. And I think this is also the shape of science itself. science is fundamentally a symbolic compression process where you're looking at a big mess of observations, like the position of planets in the sky or something like that, and you're compressing that down to a very simple symbolic rule.

39:07You're saying, yeah, all these thousands of observations are actually just all this one simple equation. That's symbolic compression. And to do this, by the way, you need the model to be symbolic. You could not fit a curve and say, well, that curve is my model. That would never be optimal. It would never be concise or elegant enough. And that's not what science is doing. Science is not about curve-fitting. Science is about finding the equation, finding the most compressive symbolic model of your pile of observation. And that's the process that you are trying to recreate in software form. You could say that the NDA approach to program synthesis is that we are building science incarnate, science, the scientific method in algorithmic form.

39:51I'm curious if you compare it to biology. Clearly, LLMs don't learn the way that humans do because no baby reads the whole internet. Do you think program synthesis is closer to the way that humans learn? Or do you think that's yet a third branch where even if program synthesis is correct, there will be some yet as undiscovered third way to do it, which is the thing that we do. I think so. I do think humans do some amount of program synthesis. I think the way humans learn and the way the human mind works is very messy. It's not like there's one simple, elegant principle behind it all. It's an implementation of fundamental principles, the fundamental principles of intelligence, which I think we can identify these principles and re-implement intelligence from scratch, from first principles, in a way it will be much more efficient than the human brain.

40:44I think the human brain is messy and it can be a good source of inspiration for AI, but I think it would be counterproductive to just try to observe it and re-implement it and make it biologically plausible. I think that's counterproductive. It's not what we're trying to do at NDA. We're only trying to find what are the first principles of intelligence and what is the system that would best implement them. But yeah, I do believe the human mind does at the highest level something that looks a lot like program cities. We're currently building causal models of our surroundings. We're describing our surroundings in our mind as a set of objects and agents and narrations between objects that are fundamentally symbolic and causal in nature.

41:32This is exactly the process that lets us generalize so well and adapt so well to novelty on the fly. I'm curious about India, the company, and as you're building it, we've all here heard of the OpenAI founding story, and something that's always struck with me is just like both Sam and Greg say that it was a little odd in the early days because they didn't actually know what to do. It's sort of just like a bunch of people hanging out in an apartment. I would love to hear kind of what's that been like for India? What did the day one look like and maybe for just people who are interested in starting these alternative approaches who don't have sort of a researchy background, how should they think about that?

42:11Yeah, so we started on day one with the symbolic learning vision. We basically knew that we wanted to do symbolic program synthesis that we wanted to create a new approach to machine learning where you replace parametric curves with the shortest possible symbolic models. And then the big question was, okay, so how do we find these models? We started from the base idea, which is still the idea that we are following today, which is that we are going to do deep learning guided program search. You have a symbolic search space to explore and it's big. It's in fact a combinatorial role. You're not going to make progress if you just use brute force.

42:52It's not going to scale. You have to break the combinatorial role and the way do it is to add deep learning guidance. It's actually very similar to the principles that underlies something like AlphaGo or AlphaZero. That was our starting point. We also didn't have very clear ideas about how to build it. We tried many different things. We tried many different ideas. It took us half a year, roughly, to get to good foundations where we could start building a system that compounds. And I think that's what's really important when doing a lab like this, that you don't want to be in a situation where you're constantly trying something new.

43:32It's not reusing any learnings, any findings from the previous approaches. You want a compounding stack. You want to build reusable foundations and then the next layer and then the next layer. And of course, you want to be building onto the right foundation. So don't commit to the foundation layer too early, but also make sure that at some point you're building this compounding structure. And that's the situation that we're in now. Is ARC 3 the end, or will there be an ARC 4, 5, 6? Can you keep making it harder? Yeah, yeah. I think there will absolutely be ARC 4 and ARC 5. I mean, we're currently planning ARC 5.

44:11The point of the ARC AGI benchmark series is not to say that, well, you know, here's this test. If you pass it, this is AGI. Instead, what we're trying to do is we're targeting the residual gap of fair capabilities. Like Frontier is advancing and we're saying, well, if you compare it to human abilities, there's all these tasks, all these things. It's now doing well. So we're going to create a benchmark to target that. And so it's a moving target, right? It's not fixed points, it's a moving target. There will be ARC 4, which will be in the spirit of ARC 3, but more focused on continual learning and curriculum learning at longer timescales.

44:53So you're going to have fewer games, but they're going to have way more levels. And the levels are going to be compounding, meaning that for each level, you need to reuse stuff that you've learned before. And then that's going to be ARC 5. And I'm actually really, really excited with ARC 5. It's very, very new and different. It's all about invention. And I mean, you will see what that means. Eventually, I expect we will run out of things to test. Like as we get closer to AGI, eventually there will be no measurable difference between human capabilities and particular human learning efficiency and frontier AI.

45:30And when that happens, when it becomes effectively impossible to measure the gap, this is the AGI moment. Well, then the machines will take over and then they will create Arc ASI 1. and then it'll continue from there. If you had to put a guess, I mean, years, decades, months. My timeline to HDR, if you just try to extrapolate from the current rate of progress and the amount of investment that's going into not just the LLM stack, but also side ideas, side bets that might work out, like NDA, for instance, I think we're probably looking at AGI 2030, early 2030s, most likely. So around the time that you are going to be releasing like maybe ARC 6 or ARC 7, that's probably going to be AGI.

46:25You guys are doing a different approach to LLMs. Do you think there's room for more startups to explore other new approaches? And are there any other ones that you think are promising that don't have time to explore yourself? Yeah, absolutely. I mean, there are many different approaches that you could try. I've said that compute is a great equalizer. I think if you look at the amount of compute and resources that we've thrown at deep learning and gradient descent and scaling that up, if you had thrown the same amount of investment into almost anything else, you would also have seen extremely exciting results, like genetic algorithms, for instance.

47:03If you try to scale up genetic algorithms, I mean, I'm sure you do incredible things with that. You could in fact probably do new science because that's based on search and search is the best fit for automating the scientific method. I think so right now, there's also approaches that build on top of the current stack with their slightly alternative, state-space models for instance. There's the XLSCM architecture. You can basically, you know, current frontiers are a stack of things and you can take any layer in the stack and try to propose an alternative. If you propose an alternative architecture, you can be doing, for instance, more like recurrent models instead of transformers for the architecture.

47:49Or you can do even lower level. You're going to be like, okay, we're still going to be training parametric curves but you're going to get rid of current descent. We're going to use search. Maybe you're going to do new evolution. That's the slower level. And the lowest level is the level where we're operating, where we're saying, well, actually, forget about curves. Forget about parametric learning. Forget about gradient descent. We're just going to do something completely different. And I think if you want to build optimal AI, you're kind of forced to go back to the foundation of the stack. It cannot be like one layer added on top of the pile.

48:28François Chollet:So do you think for aspiring researchers to want to do a new neolab with a different approach, they should be reading research papers from the 70s or 80s and go deeply in those with approaches that were not as invested nowadays? That is actually a great idea because earlier in the history of the AI research timeline, people were exploring more things and very different things. You've had this sort of like collapse of everything into one approach. it's actually kind of a bad idea. Consider that not too long ago like about 20 years ago We had the collapse into SVMs too. Yeah, I mean I wouldn't describe it as a collapse because there weren't that many people doing SVMs and it was a much, much smaller field back then but there was this widespread understanding that neural networks were a failed approach that neural networks didn't work and it was a waste of time to keep trying that.

49:26François Chollet:right? Yeah. No, even in the late 2000s, this was a set of things. Basically, when I got into AI, people were telling me like, hey, neural networks, don't try that. I was like, yeah, but it looks a lot like what the brain is doing. I'm interested in that. If everybody is working on something, you are discarding ideas that will actually turn out to be very proactive ideas, right? And yeah, like back in the 70s, back in the 80s, people are trying more things. And I think genetic algorithms is actually a very good example of that. I think this is an approach that has a tremendous amount of potential, but not too many people are looking into scaling it up deeply.

50:07Are there any characteristics that you would be looking for? I mean, is it as simple as like, if there's a scaling law that could happen, then even if it's different, or is that too, like, you know, thinking by analogy? I think you are looking for approaches that scale. Yeah. I think it's a non-starter. If you're working on something, but the only way to increase the capabilities of the system is to have human engineers and researchers spend time on it, it will not work. Because even if the idea is very clever and very elegant and works really well, capabilities are going to be bounded. They're going to be bounded by human investment, right?

50:47You want to be in a setup where the system can improve its capabilities with no human in the loop, with no human bottlenecks. So you would say, like, don't just do it the way we did it like 10 years ago. Do it with the idea that recursive self-improvement is baked in at the beginning. Yeah, not necessarily recursive self-improvement because deep learning, for instance, is not recursively self-improving. But with the idea of scaling up with no human bottlenecks, you want to remove the human from the improvement loop. The great strength of deep learning is that the models got better and better simply by adding training, training compute and training data.

51:24I mean, it's a little bit of caricature because, of course, just adding these factors requires a lot of human involvement. But basically, that's the idea that you have this decoupling from the improvement curve and the amount of human effort that's needed to be injected into the system. Yes, or human effort that's already happened. The LMs do actually require an enormous amount of human effort. It's just it was the human effort to build the internet and we'd already built it. Yeah. Actually, less and less now that we are doing training in interactive verifiable environments. Because then you only need a small amount of human effort to create the environment.

52:00And from that small amount of effort, you're creating exponentially more trained data. But at first, I think to sort of like prime the machine, you need this tremendous amount of human-generated abstractions and coding in text data. And if you don't start from that, you cannot get the system into this loop. Do you have any advice for me starting a open source project, things to do, things not to do in the AI space? Because I am not sure how I signed up for this in the last 14 days, but I think I have, I don't know, on the order of like 10 to 30 ,000 people using GStack every day. That's wild. Yeah.

52:43And I don't know, like, I have a job. I guess, like, you know, what was it like to start Keras? And how did you keep maintaining it? What's a good maintainer? Like, what did you learn from that? I don't know. This might be a whole hour. Yeah, I mean, lots of learnings from growing Keras. So right now, I'm less involved with it. There's a big team at Google that's working on it, and they're doing an amazing job. So it is possible to not, you know, to put people together to like... It is possible to start something. It is possible to start something. That's a relief. And then get more people involved.

53:18And at some point it becomes its own thing. And just, you know, it used to be your baby, but now it's all grown up. It's all adult and going on with its own life. So if you ask me the factors that really made CARE successful, I mean, first of all, is that there was this big focus on making the API simple. and intuitive. There was this big focus on usability. And this was inspired by scikit-learn. Scikit-learn was sort of like the OG machine learning library for Python. And what made it successful was that it was so easy to get started with it. So at first I was like, okay, I'm going to package all this functionality I've created under really, really simple APIs.

54:00It's going to be like the scikit-learn API. That was like the big idea. The focus on usability It is not just making sure the API is simple, it's also making sure the entire onboarding experience is nice and easy. Like the docs should be very informative. The docs should be not just telling you about how to use this thing, they should actually be teaching you about the domain in the first place. Because the folks who land on your website, they're not going to be already deep learning experts. They're going to be people looking to maybe start using deep learning. And so you have to teach them not just how to use the tool, but what the tool is good for and the entire field around it.

54:40And then, you know, you have to put a lot of investment into community building. One thing we did a bit at Google, in fact, you know, Google made it kind of difficult. And I was sad about that is hire your power users, like hire your fans. This is a really, really good idea. Like find the most enthusiastic users from your community and just hire them on your team. Amazing. Yeah. And these are always the best people, right? All right. Time to start gstack.org, put in a bunch of my own money and then hire a bunch of people to work on it. That sounds good. I think you've been a leader and pioneer and we're so lucky to have you sit with us.

55:22there are people watching who are at the beginning of their, you know, adulthood even, like they're certainly their professional careers or actually like people just around the world. They're like trying to understand like, what does this mean as intelligence becomes broadly applicable? Like, what would you tell, you know, if you were 18 right now, what would you tell them? Yeah. I mean, there's a lot of people today who are very pessimistic, very negative takes about the rise in AI capabilities. They say, oh, you know, I'm going to be out of a job soon. That's going to be mass unemployment. AI is just going to take over completely.

56:01And my take is actually, you know, the more you know, the more expertise you have about things like programming, for instance, the better you're able to use and leverage these tools for your own benefit. And with the right kind of expertise, all this AI progress is actually empowerment. Like it's something that you can leverage for yourself. I mean, that's exactly what you did with your project, right? And yeah, more people should have this mindset of trying to learn as much as possible, not just about AI, but about the domain that they want to apply AI to, right? So that they should seek to turn this new development into an opportunity, into a tool they can use for themselves to improve their own lives.

56:46I think that's the right mindset because, you know, you're not going to stop AI progress. I think it's too late for that. And so the next question is, okay, like AI progress is here. It's actually going to keep accelerating. How do you make use of it? How do you leverage? How do you ride a wave? That's the question to ask. I wish we could keep going for a couple hours because I'm sure we could. Francois, thank you so much for spending time with us. Thanks so much for having me.

57:18you