AI’s Next Frontier: World Models Explained by Christian Keller

0:00So text is by definition already a compression of information. Information is lost. And so using other modalities helps get other information. And oftentimes when you get text and images or text and videos, with images you get the context of the text in an image, in a setting with the colors. When you add the video, you get the temporality of it that you would not have had before and after how things can change, how things can enter the screen or not. And so I think the first point is that LLMs alone, text-based LLMs, will never be able to learn like a human because they're missing so much information.

0:34We've removed by the sheer definition of putting it in words. Hi, my name is Christian Keller. I'm an angel investor, startup advisor, and currently work as a product lead at the Meta Superintelligence Lab. I've been building AI products and models for the past 15 years, and I'm really excited today to talk to you about AI research and building AI products. Okay, so one of the things, Christian, that Henrik and I love to do is invite our cool friends to introduce to each other. So Henrik has very little introduction. Why don't you tell Henrik and also listeners who may not know you, why they should be interested in this conversation today?

1:15Well, I've been working in AI for the past 15 years. So I'd like to say that it was like before it was cool. And I've taken a pretty deterministic approach to my experience in it, where I've started with building AI products myself, like thinking through problems that I knew and leveraging AI in order to apply them. And then I had some experience in infrastructure around my work on PyTorch and understanding what it takes to actually deploy AI at scale. And at the same time, how researchers also were using these tools. And lately, I've been doing a lot of work in AI research, working on Llama 3, Llama 4, world models also, which I think gave me kind of a good overview of the three kind of main categories I see of the AI ecosystem and what it takes to build successful products in it.

2:04So obviously your career spans long before kind of the chat GPT moment. Maybe we could start with that moment as an inflection point. What changed for you and for the teams that you lead who are building AI systems, what changed for you when ChatGPT came out? And I'm sure there's one element, which is people's awareness of what you do and understanding somewhat of what you do, but then also you and your team's abilities to do what you do in a different way. I'd love to hear about that. So at the time that ChatGPT launched, I was working on PyTorch. And so PyTorch is already quite known in the space of AI and was like...

2:46Could you just explain shortly what that is for people don't know. Oh, right. Yeah. So PyTorch is a framework that's used to build AI models. And so it's the language by which researchers are able to formulate what an AI model should be, what it should do. And it's a translation layer in a sense between these functions that the researchers had in mind with the hardware. And so in effect, when you code normally, you might use Python, right? So PyTorch is on top of it, an additional kind of layer that allows you to build AI models. So for example, it's an open source tool built at Meta initially, but it's used broadly now.

3:23It's part of a foundation. And ChatGPT was built leveraging PyTorch. So it's used in many, many other areas. So what happened to us while on PyTorch, I think what was interesting is that, first of all, people started talking to me about, you know, oh, is AI a thing? Like, have you guys seen it? And it's interesting because from the PyTorch standpoint, we saw a lot of what researchers were doing ahead of time. And language models were not something new. I think the technology had been there for some time. And there were many, many teams that were leveraging them using a key technology called transformers that had kind of changed the space quite a bit at the time.

4:00So we're seeing a lot of innovation already happening. I think the surprising part for a lot of people was how well packaged this technology was in ChatGPT. and how the team that developed ChatGPT was able to make it accessible to a lot of people at scale. So suddenly there was an awareness that wasn't there before of what this technology could do and how well it could be applied. And so I think there was a shift at that time in terms of how people were using it. And why was packaging not a consideration before? I mean, is it the classic engineers aren't thinking about end user consumption or what's going on there?

4:42I don't know, but I think my bet on this would be that oftentimes people are looking at things like assistants. And so you had the Google assistants, you had the Alexas, you had even assistants we're working on at Meta. But each time I think we're considering them more from the perspective of like one use case with one specific deployment. was chat GPT was a way to say, look, we've got this really cool technology. We don't know what people are going to do with it. Let's sort of create the most generic, the most accessible way to do it. And discussion chatting is literally like, you know, the group of society in a sense.

5:14Which probably is a little bit of like a big bet, right? Because I think as a product developer, you would normally kind of like tell people, you need to figure out like, who's the customer? What's the problem? And then kind of be very specific. Like niche down, niche down, yeah. Yeah. So in many ways, I mean, like, do you think they were very purposeful with that kind of understanding that we just this is such a foundational technology? This is more like electricity than TCP IP. So therefore, we just need to kind of like put it out there and basically throw spaghetti at the wall. There's two things that are going on.

5:45I think I don't know what they were thinking about at the time, but it's one of these problems where you've got a really nice hammer and you're looking for nails here. And that's the opposite approach that you want to usually take when you build products, right? You want to find the problem first and then develop the tool that's going to help solve it. So what do you do in this case? Here, I think what was very interesting in their approach is that the chat interface allowed for people to, in a sense, explore what use cases could be interesting. And you've seen some more or less formalization, I think, of some of these use cases in the last several years that were not quite clear initially.

6:23And so I think what they did is they created this learning engine by having all this data from all these users that they were able to kind of capture. And so today you can see use cases that are, I mean, the way I formalize use cases for Generative AI are like one could be UI interface type, something that helps you better communicate with a machine. So, for example, the voice interface, the chat interface. And it's a different way from like being explicit about clicking and typing things. the second thing I see is more around like text generation and so that's initially your chat that could be your your thought leader assistant that could be the editor that's going to help correct your write-up your newsletter or what you want to say so anything that has to do with like interactions text knowledge creation whether it's like with text and images and then there's the kind of automation which is like let's actually do actions and take tasks and entirely kind of not necessarily remove the person from it, but take some chunks of work away entirely that can be either done better or faster, or even if it's not better or faster, maybe cheaper and like, you know, free people up to do other things that are more valuable for them.

7:33You know, what you describe about what the field had been doing in creating narrow assistance versus what OpenAI did when they released JIGPT in creating a general technology is what I've heard called the bitter lesson. I don't know if you all are familiar with that, But it's what researchers call the bitter lesson. And it's been established time and time again that you can work to train a model very specifically. For example, Deep Blue, the model that beat Kasparov, had been trained by chess masters for the purpose of chess. The models that beat Deep Blue aren't trained on chess. They're just told, play chess a bunch of times.

8:08And the bitter lesson basically is the following. any attempt humans have made to train for a narrow task ultimately gets done better by scaling laws of a general model and that's that's been proven again and again that's called the bitter lesson which is to say in a way the it's bitter because there can be a sense of exasperation or fatalism why bother training it for this narrow thing because if scaling laws catch up eventually a generally trained model is going to be able to do this better than what I can train to do it now. It's a little bit also, as you know, I'm obsessed with Kenneth Stanley's book, Why Gradients Can't Be Planned.

8:50He's an AI researcher, turned, sold his company to Uber, then became head of AI Uber, and then went to OpenAI. And he has the same argument. Basically, this book is about how you get to AGI. And his point is that you can't define what it is. You have to have these systems of open-endedness. And then that becomes basically the stepping stones have leaked air and so i think i'm fascinated by that um i have a bunch of questions but one thing i'll say about the chess part is that chat gpt does not beat the current best players at chess so that's something interesting to see which is there is some level of like specialization in in the functions that some models can do and like the counterpoint i think here is that i think there is definitely your point proves that there is value in like building I think these frontier models are very general, both because I think they unlock new capabilities that we were not expecting, and that they allow, I think, for the discovery piece of use cases that we wouldn't even have known would be possible.

9:46But we know today how to take these models and to make them very specialized and even better in a more narrowly, more efficient kind of way to run, which is kind of the other side of the coin here. Is that because that generalism, And obviously the models are statistic kind of like averages in many ways. And often when you want to be kind of beating a chess master, you need elements of originality or differences. And the general models don't do that. And so when you go very purpose is because either you need to work a very specific workflow and you don't really want like the average making that up.

10:23You want something that you know is the best. Or when you sometimes need to have basically outlier answers, you need to kind of train the model to kind of look a little bit off the rapid regression to the mean? There's two things, actually, and you're making a really important point here. The first thing is that I think different model architectures and, you know, will kind of solve different types of problems in some cases. So like the way LLMs are built right now are very much focused on like language or like omni models, maybe various multimodalities, and they're good at doing that. They're not necessarily good at planning initially, right?

10:55That's not what they've been trained for. They've been trained for finding the next most likely token or the next most likely word. So there's a question around which models are best suited for what type of application. But the second point here is that there is something to be said about these frontier models, even when they're post-trained and where when they've actually gone through fine tuning and any other round or, you know, reinforcement learning or whatnot. We've reduced the entropy for them. We've kind of tailored them towards very specific types of answers and very kind of unique approaches or ways we want them to kind of answer some questions.

11:29And so when you take a pre-trained model, there's still a lot of variability in it. There's more variations of the type of answers that they can give you. And so that's an interesting point because you can take that pre-trained model and then specialize it in different ways based on your post-training that you're going to do to make it useful for, I don't know, Maybe knowledge about chess or knowledge about like math theorems proving or knowledge about... Is there a way that users that are just using the models as they are can use that insight to get better outputs of their generalized models?

12:05So, for example, with that thing, if I'm sitting and I'm being asked to make an agent that does HR stuff or brand stuff or whatever it is, obviously you use a rack, I would imagine, or like whatever context window to kind of do some of that. Are there other kind of tricks to make sure you achieve that non-generalizing-ness? I mean, I think it depends like what stage of your product development you're in. Like right now, I'd say most teams I've talked to, like in startups and whatnot, like they usually start with taking a generic model, like the best they can find. And they'll test some prompts and then they'll kind of generate some like system prompts in a sense that will somewhat like help narrow the model usage.

12:51But you're still starting from a model that's been posturing already, that's been kind of narrowed to following the instructions and to having certain languages. So you can move it, but you've got only so much like leeway to kind of move it around. So once you validate that, maybe with your product market fit and see that it's working and you want your tool to improve or your model to improve, then this is when you take a model that's been more like pre-trained with instruction full, for example, and then you fine tune it. And so if you're willing to invest in it, you have a whole strategy you have to figure out around benchmark definition, like data strategy acquisition in order to create models are going to be even better at doing these things.

13:27through that fine-tuning process. The question that I was keen to ask you before we got on today is, I saw a YouTube video of Ian LeCun, who is from Your World. He was making an argument that I kind of understood, but not really. And so I thought I would use the excuse to ask you. He was talking that these language models had built in basically limitations. And so as we're kind of like trying to raise towards AGI, whatever that is, He felt that we needed to kind of start to train on video models or other kind of multimodal models, I think is how I computed it. Is that, do you sound familiar? And can you explain to me why the language models might not be right, the text-based models, and why suddenly having video would make it better?

14:16So I am very much aligned with Jan, I think, on his message. So let me kind of summarize. There's two main points here. The first one is around the modality. So text is, by definition, already a compression of information, right? Like the example I usually give is like you can go and watch a game at the stadium. There's the crowds, there's the smells, there's the game you see maybe not as well because it's further out, you know. But you've got this whole emotion. You've got like a feeling there's a sense of community with the people around you. And then you can like watch it on TV. You get maybe that if you've got a few friends, but you're missing what's happening in the stadium.

14:53You're missing that experience. Then you can read an article about it like in the newspaper the next day. I can tell you you're going to have a very different emotional reaction between like reading the article and like actually experiencing the game in the stadium. So information got compressed. I even tell people like when you read Tolkien, there's like three pages long description of like one tree, right? Like you still haven't touched that tree. You still haven't smelt it. So information is lost. And so using other modalities helps get other information. And oftentimes when you get text and images or text and videos, with images you get the context of the text in an image in the setting with the colors when you add the video you get the temporality of it that you would have had before and after how things can change how things can enter the screen or not and so i think the first point is that is that llms alone text-based llms like will never be able to learn like a human because they're missing so much information that's already we've removed by the same sheer definition of like putting it in words.

15:52The second point, I think, is around, you didn't mention it, but his main point, I think, is around the fact that LLMs are hallucinate, right? They hallucinate by design. Literally, they're just trying to make predictions of most likely things that are coming next. So they're really good at it. And with post-training, we create an illusion of intelligence, right? Because it seems like that's what I would want to hear. And that it makes sense when we having conversation loud that LLM totally gets me, you know, like gets my problems and the like. And so that's actually fine for a lot of use cases.

16:30But when it comes to using LLM for super intelligence or AGI or however you want to call it these days, like you're going to want to solve like really hard problems. You're going to need robustness. And so there's a second concept of world models that Jan pushes for and that I'm a firm believer in, which is we need to create models that can predict a change of state in the world robustly. Not simply say, look, this is the next likely word that was going to be said, but I understand that if in a video I'm holding an apple and I stop the video and then I say, okay, the hand is going to open, what's going to happen?

17:05Well, the apple is going to fall, right? Because there's gravity. We need to understand that if it's easier in the space station, that's going to act differently also, right? And so you need to be able to build models that are going to help increasing this robustness, increasing this level of predictability of outcomes for the models to solve much bigger and harder problems in the world to get to superintelligence. And video is probably the thing that is most available to trade on. But there might be, remember when kind of like suddenly Google put cameras on cars and suddenly there was like street map and it was like mind blown that somebody would actually drive a car through all the streets in the world.

17:43Do you think almost we need to get a point where kind of like you need the Google street car of like the models to go around and kind of like sucking smells and sounds and everything in order to figure out how to map the world? Well, you know, like if you have an iPhone, your iPhone only takes pictures, like has a mini radar for distance in it, like in some cases, depending on the models and whatnot. So you're already getting some of that additional information beyond simply the image or the video that can start getting recorded. I don't know if anybody's using it. But we'll definitely need more.

18:12The videos, what it adds is the temporality. I think that's a really important one. And so it's better to kind of teach cause and effect in the sense if you can see things that happen before and after, for at least in our scale of the universe. But we will need, I think, more. But we don't know yet. I think we haven't pushed the research far enough yet around these JEPAs or world models and videos and others to understand yet if that might be enough to get to what we consider AGI. Okay, so I'm dying to know the other part of our first question, which you could basically describe our interviews as a series of rabbit trails, right?

18:49But going back to the first question, what's changed for you in developing AI models now that we have LLMs? And part of the reason I'm asking this question is I heard an interview with Zuck recently where he talked about one of the things Meta is doing is really using its LLMA models, etc., and tuning it for the development of AI. and I think there's kind of broad belief that in the developer community that using AI to build AI is kind of you know the way to achieve escape velocity right I'm curious kind of from an abstract existential point of view I'm curious to know for a practical point of view for you personally as a developer of this stuff what started changing and when did you feel like whoa all of a sudden I'm riding an electric bike not a regular bike or you know whatever the right Yeah, no, that's essentially what I think it's been continuous.

19:41I think every time AI improves, we find new ways to use it. But I mean, AI has been building AI for quite some time. The whole post-training process, in a way, has a component to it where we generate synthetic data, we generate data traces that are generated by AI models, right? And so initially, and this is all in the LAMA 3 paper, LAMA 4 papers, it's all very public, But you can see that initially what we use is like what we call like SFT, I think, for Find2Me, which is you have humans making annotations, say, OK, what should the answer be? Let's have a human write it because a human will know like what's the best answer there.

20:19And so you would do that. You train models, try to replicate a little bit like what that answer was going to be and kind of guide them towards that. But then what we realized is that, you know, you can scale generating a bunch of different solutions with AI itself much faster than with a human, right? And so we would generate a bunch of data points of outcomes with models. And then we say, well, the humans is still, you know, the better judge here. So the human would look at all these different answers and say, OK, this one's the favorite. Let's like try to train the model to kind of do that. And then we're like, wait a second.

20:53Wouldn't the AI be a better judge? Can we train it?

21:24So now... everything that they're doing right now, they end up having an epiphany where they go, wait, could AI help with that? Is it basically just kind of working its way through the progression of, you know, activities in the workflow? I'm lacking the vocabulary here, but you're getting my chance. Yeah, no, it's definitely that. I wouldn't say like it's that like for every single aspect of research, there's a lot of like... I didn't mean to be making comprehensive claims. I just meant to be describing kind of the way it works. Yeah. No, and definitely. And that's the thing is like, I think about it in my personal life flow, right?

21:55This is exactly how I use AI. And I'm like, wait a second. I'm not using AI for this. Would that be easier and better? And so I ask that constantly to myself to try to stop them. I say, no, I have to eat my own dog food in a sense, right? It's like, I have to keep using these things and understand what the use cases are and test them and researchers do that. What's the last use case you've done yourself where you went like, hey, wait a minute, that was smart. Wait, I can do that with AI? Yeah, I'm dying to know. What was the last one? I'll credit my wife for that one because she's learning French right now.

22:22And she was taking this like B1 level exam for her leveling. And she was like, wait a second, like I'm trying to do more tests, but I can't find these tests online. And so she just worked with various AI models like, you know, Lama, ChatGPT and Gemini to try to like create tests for herself and like learning programs. And so she would like, she didn't like typing. So she would even write everything by hand, just take a picture of the thing. And then it would grade it. But it would grade it because it had provided the model with the online reference to how these things should be graded. And she did some tests to figure out how known rating for some specific examples would come up based on her thing.

23:05And they came out exactly right every time, the exact same grade that were recommended. And so it was great because that allowed her to, one, scale herself, generate a lot of test data for herself to learn faster that wasn't available online. at least she couldn't find it. And so that's an interesting point, right? Because she found this way to do this. This is something she did with a chat interface. But a lot of innovation and products are happening right now where somebody realizes, look, I've done all this with a chat interface. Let me just package it into a product and make it simpler for people to just go and use it.

23:39I'll give you a fun example of that. I'm building this platform called Autos where we're trying to launch 100 ,000 companies a year. It's kind of like an entrepreneurial platform. And one of the things I saw the other day, it blew my mind. And that is the woman that is, she's American Chinese. And she was learning, I think, French too. And one of the things that she found out was that people who grew up in different cultures, they hear sounds differently. And so the tonality. And so she created this little kind of like, almost like guitar auto-tune kind of visualization. So if you want to say, I don't know, bonjour, she could actually go, And she could kind of find the tonality that she had to do to say it correctly in the accent that was correct.

24:22Because it just said to like, you know, phonetically or whatever, what she would kind of see and what she would say would seem right to her. But for a French person to be like, this is completely wrong. Fascinating. So Christian, what you're saying in both your case and your wife's case is there's this moment where you go, could it do that? You know, kind of a epiphany moment, which I love. And I often tell folks, I think sparking your own imagination is kind of the highest priority. And being in an environment where your imagination can be sparked is very important. Do you have practices or routines or rituals either for seeing new use cases and applications and or for auditing your own kind of existing workflows on some kind of a regularity to identify what's the next area of your life you should be inviting AI into?

25:16Or is it more kind of serendipitous and random? No, no, I do. I feel like two or three different things. The first one, I think, is any job that I'm doing that's taking me some time, I always will try to apply AI to it. And what I notice is initially it'll take me longer with AI. But if I commit to it, it'll usually make my life easier after a while once I've kind of narrowed down the kinks. The second one is, being an entrepreneur before, I think I always look for problem spaces. Because all these aha moments come from, this is part of my workflow. This is a problem that I'm solving right now.

25:51And I need to really make it better or I need to scale it. Like in the case of research, like, I mean, I didn't invent these things. Like this is what researchers do. But when they are these moments, it's like, look, I need to be able to scale more. Right. I need to be able to get more data. I need to be able to leverage better the humans in the process. I need to be able to do these things. And so it's having a keen eye on the problems because now we do have this armor, but we are looking for the nails everywhere. So looking for these problems constantly, I think, is an important one. But I think the pitfall long-term with this is that that helps you to optimize existing workflows and existing processes.