Ep 92: xAI Co-Founder Unpacks the Future of Model Development

0:00Welcome back to Unsupervised Learning. I'm Jacob Effron. We had an awesome episode today with Igor Babuschkin. Igor has been at all the right places at all the right times, leading a lot of really interesting AI work. He was at DeepMind, where he led a lot of the work around StarCraft as well as AlphaCode. He was at OpenAI doing the early work on reasoning. Then he was a co-founder of XAI, where he did some of the heroic work on Colossus, as well as helped get the models up to where they are today. It was awesome to get to talk to Igor about his new company, River AI, the bets they're making around personal AI for companies, for consumers, as well as local hardware.

0:38We talked about all his takes on where the ecosystem's headed, what's required to improve models beyond coding into non-verifiable domains, and when he thinks we'll get there. We talked about why he thinks the closed source model providers are actually in a really hard place right now as businesses. And we hit on his reflections on what AI models mean for the future of the world and the policy implications of this as well. Just awesome to talk to someone who's at the forefront of the space on all the questions that are top of mind. I think folks will really enjoy Igor's perspective. Without further ado, here he is.

1:11Igor, thanks so much for coming on the podcast. Really excited to have you here. Yeah, thank you for the invitation, Jacob. We're excited to chat with you. You've pioneered so much important research in your time at DeepMind, OpenAI, XAI. You were early to like so many of the things that I feel like define this era we have today, our fire-based reasoning, inference time search. And you recently left XAI to start your own company, River, really focused on personal AI. I feel like there's so many things that our listeners are gonna be excited about to get your take on the episode. I wanna make sure they understand River, they get your reflections on this, being at the epicenter of all these things that have happened in AI, as well as your predictions for kind of where we're going.

1:46As I was thinking about where to begin, one thing I found really compelling is that you wrote some really interesting fiction earlier this year. For those that haven't seen, you know, one piece was this dystopian look at how models ended up uncontrolled, running the world, and had this really interesting ending, right? Where your protagonist kind of realizes that they dreamed all of this and, you know, there's nothing, none of this has happened yet, but they still make the decision to kind of take that first step and begin. And then kind of interesting contrast, your second piece was really this like love letter to the machines that come next.

2:17And so maybe to start, I'm curious, what inspired you to write these? Yeah, I think that's something pretty big that happened to all of us software engineers, AI researchers, around November, December of last year, which is that the coding agents suddenly became so powerful that you can't ignore them. Previously, some people used them, some people didn't use them. But from that point onwards, that latest iteration of cloud Opus models, suddenly everybody agreed that, yes, these models are very powerful. yes, they make our jobs much, much easier. And also, wow, they're doing so much of the software engineering now that we used to do that we used to enjoy.

2:56So it was a real transformation for all of us. And like many folks, I spent a large part of December rewriting all of the coding projects that I've ever, you know, wanted to do in my life or have done using the coding agents. And you get this incredible sense of like, wow, anything is possible now and also these agents are going to continue to improve. They're going to continue to get more capable and what will happen with the world at large as these models are getting stronger and stronger and that kind of inspired the story. It's very much a modern version of the Sorcerer's Apprentice. So the Sorcerer's Apprentice is very fascinated with magic and starts to use it but then it kind of runs away from him.

3:44So kind of like in the Disney short or in the original poem. So it's very much a version of that in my mind. And I think we're all becoming the sorcerers apprentices now with LLMs and agents becoming more and more capable. And I think it's not going to stop at coding. So I'm very, very interested in figuring out where our agent is going to go next. What else is going to be transformed? And the more I think about it, the more it feels like everything will be transformed. And all of computing really is a potential, is up for grabs when it comes to agents, because the ability to deeply understand what's happening in your life, at your workplace, what's happening in the world at large, and then taking action based on that is something that you can apply to any problem, to any kind of thing you might want to do as a business, as a consumer.

4:36So that's why I'm so interested in personal AI, because I feel like that's the next iteration, that's the next manifestation of what the agents are going to be able to do for us. The impact will only grow from here. I mean, obviously, it feels like that December moment you talk about resonated so strongly with so many folks. And I think actually even a lot of researchers responded to your story being like, this really hits hard. We had that moment in November, December on the coding side. And I think a lot of folks are trying to figure out what's required to get agents to work really well in other domains.

5:05and kind of maybe asking, well, is there something special about coding and math or the most verifiable domains out there? Are we going to see the same kind of progress in other domains? How do you think about what needs to be done to kind of get to even more general purpose agents? Yeah, I think there are actually many next steps that we could be taking from here. There are many different directions that open up coding. Coding was sort of the central path, the obvious low-hanging fruit, because it's verifiable, because it touches so many different domains and unlocks so many different things that you can do with the agents.

5:39But from here, there are a few different options of where to go. I would say scientific discovery is a very, very clear one. We're also starting to see the first sparks of agents being able to solve real serious problems in math and the sciences. And that was a big inspiration behind XAI as well. So we were all really fascinated by this idea like, well, at the time, LLMs weren't really capable of solving hard reasoning problems, but we kind of felt that this was the next big breakthrough and that was a big inspiration behind XAI where we started it. So I think agents will continue to improve very rapidly on complex coding projects and also math problems, anything that's verifiable.

6:19So with math, you can even formalize your theorem and your proof using tools like Lean. And it gives you a good reward signal that helps you improve agents. It also helps you tell if you've actually found the proof as these get more and more complex. and seeing that actually work for real is really fascinating because we AI researchers, we've thought about this for some time before, like maybe formalization will be useful in the future and so on, but actually seeing it happen in real life is really quite something else. But beyond this, one huge bottleneck is data. So if you want to do great breakthroughs in the physical world, whether it's material science, whether it's new fundamental physics discoveries, whether it's building a better rocket engine, the type of thing that Elon might want to do with the agents.

7:09That's where we often need to run experiments in the real world. So you need to figure out a way to close that loop with the real world environment. So the agents, when they have an idea what kind of material to design, they need to get some feedback as to whether that was a good idea, did it work, did it not work. So that's a big frontier right now, I would say, in using agents for scientific discovery. So this kind of direction of science and discovery, I think that's a big thing that is happening at the moment. That's something a lot of people are focusing on with the agents. And there's a potential there that we'll get these highly super intelligent AIs that are vastly more capable than any human or even any collection of humans at creating scientific breakthroughs, at making decisions, maybe at predicting the future as well.

7:55So I would also broadly categorize that in that type of direction. And then I think there might be other directions as well. So what is the AI going to be that the everyday person is going to use? They probably are not going to use the super AI that costs a million dollars to run once on a question. So what is that AI that everybody will be using? So I think we might see a bit of a bifurcation there actually where there are other super AIs that only a few people have access to. But there's also the everyday common AI that you and I will be using all the time to be more productive or to help organize our lives better, just lead better lives, basically.

8:37So the reward function is how much to help the human feel better and get more things done, feel better about themselves. So I think there's also tremendous potential to improve things for humanity, improve our everyday lives. but those agents might not actually need maximum capability. They might not need to be able to prove the Raman hypothesis. I mean, it already feels like that bifurcation is happening, right, in terms of the most frontier models being really valuable for a lot of coding and mathematics tasks. But for many people that are ChatGPT users, it's not inherently clear that the more complex models have actually, the ones from a year ago were totally fine for most of what they wanted to do.

9:22Yeah, exactly. And I think it's important for the individual to also feel the benefits of AI, because increasingly it feels like there are a few large companies that create these powerful AI models that control access to them, that get to say how they will behave. And for me, it's just very important that everybody feels like they're actually personally able to get value out of AI. And that's a big inspiration behind where I started River recently, because to me, it feels like potentially a technology problem. So we've made certain assumptions about how AI should be built and distributed to people.

9:57You should do large pre-training runs. You should set up an API and charge people by the token. And that's sort of the model everybody knows about, everybody follows. But it doesn't have to be that way. So we can come up with new ways of building AI models, of customizing them, of distributing them to people that maybe will put them more in control of their experience. We'll put them more in control of what the AI is doing. And it's a research problem. It's an engineering problem. So it's something that I find very fascinating. So that's why we decided to start this new company. When did it kind of crystallize for you?

10:30Because obviously your previous experience had been at some of the largest closed source players with those kind of models. And so when did this kind of all come to be? Yeah, so when I left XAI in 2025, I decided to do a bit of angel investment in AI safety companies. So I got to meet a lot of different founders in the AI community and AI safety in particular. So people having new kinds of ideas around how we could help make AI more beneficial for humanity in the long run or to better control the downsides of AI or the risks. And there were a lot of amazing ideas out there. And I just, I found it all really fascinating.

11:10But at the end of the day, I'm just sitting around, you know, waiting for all these folks to make progress on their mission and to succeed and got really impatient. So I just felt that I wanted to start something new myself so I can help push on this direction. And for me, helping distribute the benefits of AI, helping distribute control over AI is a part of AI safety. It's maybe, for me, the most urgent thing that we need to do right now because power over AI is concentrating quite a bit. And especially in the US, you can feel that people aren't too happy about that. They don't necessarily trust the AI builders or feel like they have their best interests in mind.

11:51So I think it's important we show some ways in which people can feel the benefits, can feel like they're in control of the whole thing. Just wanted to take a quick break to say that if you're enjoying the conversation, the single best way to support the show is hitting follow or subscribe wherever you're listening. It makes a real difference in helping us get the best guests and helping others find the show. Now, back to Igor. Well, maybe it's an interesting time to just talk about your vision for the River AI as a company, the product, and how you hope to achieve that. Yeah, so when we started the company, we looked around, where are these technological opportunities?

12:27What are these ideas that have been under-invested into that could change the way that we use AI, that we build AI? And let's run this as a research project. Let's figure out if this is something that we can make work and then ship to everybody in the world. So we have really three different projects that we're working on at the moment. So one is we realized the tools to help companies take control of AI are pretty much already there. We just have to make it work really, really well. So that's why we started the River API, which is, you can think of it as a reinforcement learning and fine tuning service.

13:03Similar to other players like Thinker from Thinking Machines. But we have our own unique take on it. And one thing that we're really proud of is our engineering. So we really optimize this kind of platform as much as possible, try to make training as cheap and as reliable, as scalable as we can make it using all these skills that we've learned over the years. So that's the first bet that we're making on the River API. The second one is around personal AI and personalization. So we feel like there's an opportunity to align AI agents much, much better with the individual than what we're currently doing.

13:40So today we're training AI models to basically work well for the average user. So you're literally pooling all of the feedback, all of the results from all of your different users or during training and you're telling the model, behave the same way for everybody. And our idea is what if we kind of break that assumption and we allow the model to behave differently for each individual that it's serving. So your AI agent will speak differently to you than my AI agent speaks to me and they're going to have different preferences, different behaviors. This area of research is still pretty early. So there are a few different directions that we're going down here.

14:20Because obviously all the agents that are out there that are commercially successful, they're all trained in this sort of average population kind of way. But we feel like it's going to feel really amazing once the agent truly learns directly from you and gets better every time you interact with it. So something that I'm pretty excited by. And then the third big bet that we're making is on hardware. Because right now, the assumption is that the only way to get access to the most powerful agents is through access to a data center. So companies like OpenAI and Tropic, they bring up really, really large data centers.

14:56All of the inference, all of the training happens there. and as an individual that wants to use AI, you have to go through their APIs. There's no other way. They can decide how it's restricted, whether they get access or not. So really the ultimate move here is, can we bring the inference compute locally to the users? So what would it take to actually be able to run a frontier model in a small device that you can have in your office, that you can have at home? Very much a research project because today you're very memory limited there. So we're actually able to fit all of the weights of the model onto a single chip, onto a single device.

15:33So that's something that we want to figure out. Like, how do we bring the best models in the world to the individual so you can really feel like you've got control over it? It's in your home. Your data is protected as well. So those are the three bets that we're making. And we're very much running this the way that back in the day, opening up in DeepMind. So we're thinking very long term. where we're taking some riskier bets. We're assuming there's going to be research, there's going to be some new ideas that we'll be using. No, it's fascinating. I think each of those three bets is really interesting.

16:08And so maybe to kind of dig into them, we'll start with the last one maybe on the hardware side. Obviously, you have a really broad scope of things you're seeking to accomplish here. Is the motivation there on the hardware side? Obviously, ultimately, the most controllable AI is one you can run locally, right? No one can kind of dictate whether you have access or don't have access. Is that kind of the motivation or is there other kind of tangible benefits you see to end users that made you increase the complexity level even further by adding the hardware component to it? Exactly. So really the control aspect is the most important to us.

16:41So we'd like people to be able to take control of their AI experience and really feel like the model is theirs and not anybody else's. That's a big motivator. But then as we started to work on this, we also realized there are a few additional benefits. So having this box in your home locally with you could give you a much lower latency, for example, so it opens up a lot more possibilities when it comes to interacting with the agents via voice or via video input, video output. So there are actually some new possibilities for consumer experiences that open up once you go with this form factor. So that's something I'm very excited about because it might even make the whole experience much better than what we're getting from the data center world.

17:20Yeah, and I think just the privacy is also going to be important in the personal AI world. So these personal agents that know you deeply, that kind of observe everything that you're doing and then help you proactively, they're going to get access to incredible amounts of sensitive data about you. So basically all of your documents on your computer, everything that you're saying in the room could be stored by a personal agent because it just makes it more useful if it has that context for you, right? And sending all of that data out to the data center where there might be some security leaks or that your data might be trained on, even if you don't like that.

18:00That's really suboptimal. So we're also thinking of it from a privacy point of view. I think part of the vision of the company is everyone having their own model that's continually learning to improve and be customized to that person. As folks think about how these types of models may be built, there's different schools of thought. One of them is going into the weights themselves and updating the weights. and other people saying, hey, you might just be able to do this through a really good memory system and prompting. How do you think about that? And is that still like, are there a bunch of different research directions you all are pursuing, or do you feel pretty convicted in the approach?

18:34Yeah, so for us, it's pretty open which of those are going to be the key tools to use. But I'd say from my experience building AI models, the one thing that you really, really need is end-to-end training. So you actually want to train your agents to make use of their personal information, to make use of their memory systems if they have one. That's really crucial. And today, often what we do is we take off-the-shelf proprietary agents which have been trained a lot on coding, but not necessarily trained over long time horizons, helping an individual, using their memory to help them. So how do you maximize somebody's happiness as the reward function?

19:12That hasn't really been done yet. So for me, that is the key aspect, because I've kind of observed over time the models that were trained for a particular task have performed the best, and you really want to train end-to-end on the actual signal that you want to optimize, which would be the individual's happiness. Are you going to basically have the models running in the wild and hook it up to some self-reported happiness or a wearable, I guess, to close the loop? Yeah, that's really something we've got to figure out. But you could also just try to observe, did I end up helping the user? And the models are getting more intelligent, So they might be able to figure out, oh, actually, I made things worse today, negative reward, I made things better, positive reward.

19:53So the possibilities are growing there. It's moving into this non-verifiable domain that's still very challenging. So we need to use models to figure out how you're doing. But increasingly, with what was becoming more intelligent, it's getting possible. As you kind of envision the future state of River, what are the kinds of differences in models that you think of as canonical examples of if you and I had different models over time? Obviously, there's some set of capabilities that if the general models had, everybody would be happy, and then there's different things folks might want. What are the examples that you think of, Top of Mind?

20:27I think the examples are really endless. And it's just hard to imagine it because we haven't experienced it yet ourselves. But it could be really subtle things from maybe how much text the model writes when it replies to you, to what kind of vocabulary it uses. Is it more casual? Is it more formal? so all of these different aspects of the answer today we have one sort of fixed set of how the model is going to respond in one fixed style in which it will respond to you but in the future this can be untied and it can pretty much be endless because it will affect how it will help you throughout the day when should it reach out to you in which situations is it being too annoying or is it not helping enough it will calibrate itself to you It will know what your taste is in terms of books, movies, music, and it will be able to adapt your day to that.

21:22It's really, the possibilities are actually endless. To be a bit more inspired about what's possible, you can look at what recommendation systems are able to do today. So if you use an app like TikTok or YouTube, it figures out what your interests are and allows you to go really, really deep on what you would like to see. and the experience is totally different from giving the same set of videos to every single user. That's kind of what we want to create for the agent world and you can kind of think of it as a recommendation system agent hybrid which has never been done before so that's something we're pretty excited about.

22:04You could imagine obviously the parallels in the enterprise space when you said you're kind of starting the business with working with enterprises on models. There's obviously been a ton of dialogue in the last months about owning your own models and the importance of training your own models. It feels like today a lot of people are doing that really for cost and speed, right? Like finding ways to run smaller models. How do you think that changes over time? Do you think folks will actually try to push capabilities forward? Or even, obviously, there's a lot of parallels to what you talked about and people having different preferences, companies have different cultures, ways of doing things.

22:39How do you imagine the parallels there? Yeah, I think today a lot of people go to the proprietary models first. They have the strongest capabilities. They're used by everybody else. That's a big reason to start out with them. And then they might do custom models for speed reasons, cost reasons. Maybe privacy could also be an aspect if you actually have to run this on your own infrastructure. Because you have maybe other customers that have more sensitive data. So those are the main reasons that I'm seeing. But I think a lot of the reason why people go to their proprietary models is that the field is moving so quickly.

23:15If you wanted to learn how to train your own models internally at your company and sort of master deep learning, master reinforcement learning, and get the most out of your internal data that you have, you have to build up a team, you have to build up infrastructure, and it can be a many-month project to be able to surpass the proprietary models. But by that time, there's a new model. Right in time for the next one, yeah. Yeah, exactly. There's a new model that's even better. So you could have just gone with the proprietary model. And I think that's been the way things have been going for the last few years.

23:45But expect that to change now that open models are starting to become extremely capable. And we're sort of starting, we've sort of harnessed all the low-hanging fruits in terms of what you can do without deeply, deeply knowing basically the data and the problems that the company is solving. So if I'm an expert in a certain area, my business is building a certain SaaS tool, like an HR platform, for example, I can imagine that I have the best data, I have the best understanding of what the customers need. And I've thought about the domain quite a lot, much, much more than an entropic or open AI would think about the product that I've built.

24:26So really, those companies are in the best place to decide how their agents should be working, what they should and shouldn't do, how to best help the customers. And so far, that hasn't really hit yet because we're still riding the wave of stronger and stronger proprietary models. But as soon as that runs out, then you have a choice. Either you give all of your data to Entropic to OpenAI for them to post-train their models better, or you start to do it yourself to try to squeeze out additional capability. And I think a lot of people will opt to go for the second one because you're kind of giving away the whole foundation of your business.

25:05Otherwise, you're giving away the very thing that gives you an advantage over all your competitors. You think the proprietary models do slow down? Because obviously there's endless amounts of data budgets that seem to go purchase data across a bunch of different domains. I guess I can see a few different arguments, but one might be like, take a bank, for example. You can actually go hire a bunch of people that were ex-bankers and they can label a lot of data for you and do a lot of things in different environments and suddenly your model gets pretty good at finance. And I think it's an interesting question of like, one, do we expect the model progress to continue in these other domains?

25:40And two, how unique is actually an enterprise's data versus the army of data labelers that exist out there? Yeah, so training of these models actually has diminishing returns. So the stronger we want to make the pre-trained model and the model overall, the more GPUs we have to put together, the more high-quality data we have to gather. And AI companies have done an incredible job at scaling up through putting in so much more effort at gathering the data, tuning the models, improving them again and again, adding new innovative algorithms and capabilities into them. But eventually something will have to give.

26:17So at some point, you can't cover the entire earth with GPUs. So eventually there will be a bit of a slowdown in terms of what the AI companies are able to do in terms of the capabilities of their models. And I would argue we're starting to get close to that point. At the same time, there's another thing that's happening, which is these models are starting to get so capable at the frontier that you might not want to release them anymore. Or you might even get criticized, or there might be some regulatory action that prevents you from releasing the next generation of the model. At the same time, open models are getting stronger and stronger.

26:56So I think as a proprietary model builder, you're kind of starting to get squeezed in a little bit. You make the model too good. You're not allowed to release it. But then open source is just right behind you, getting better and better every month. So I think it's actually not the best place to be as a proprietary model builder. So I think the way out for OpenAI and Entropic is to innovate. So it's the same thing that they've done the past few years that's allowed them to be in the place where they are today. So they have to come up with some really new fundamental ideas about how to make the models more useful, how to make everybody feel the benefits of what they're building, exploring new domains, exploring new ways of helping impact, generating revenue.

27:38Because you think basically the capability improvements are slowing down. Yeah, exactly. And if they don't, then you might actually have to keep the model private because they're starting to cross this critical threshold Now you really have to think carefully about whether you can give anyone access to the model. And then on the post-training side, the effort that's being made is really incredible. So you're basically assembling all the world's experts on every single topic and then having them work with you to centralize, like all that knowledge flows into one set of weights and then gets delivered to all the customers.

28:15So scaling that operation up is also incredibly challenging. there are also some diminishing returns there. And I think that the way out there is to go distributed. So there are all these experts out there in the world already for their particular domains. So they're inside of the existing companies, they're at the universities, and so on and so forth. And I think we should give people an opportunity to basically independently create improvements to models and essentially post-train them locally for themselves, for their company, for their team, for the individual, rather than trying to capture all of human knowledge, essentially, and then sell it by the token.

28:57So that's my personal philosophy. And that's the type of thing that we want to do at River AI. So I figure out how do we allow people to do their own post-training, essentially, locally. It's interesting, because I feel like we're in this moment where you buy a lot of data and a domain and you focus on that domain and that then adds those capabilities and you're kind of going one by one. And it feels very different than the pre-training world where it's like you get to some certain threshold and then you start to see massive generalization across the board, right? And I think some people have posited that the same thing would happen in post-training in RL.

29:29Eventually, I don't know, you're adding domain 30 or 40 or 50 and suddenly it becomes like you get a lot of benefits from having everything in the same place versus just training on finance in this place or healthcare in this place. Do you think we see that generalization and that benefit of generalization down the line for RL? Or is it going to always make sense to just take some really strong pre-trained model and apply one domain specifically to that? I think we're seeing generalization in the sense that these models have such incredible knowledge of the world and they have such a wide set of skills that they can now go into any domain and bring in all the knowledge from everywhere else I think coding is a great example of that.

30:12And then I think whether there's a lot of synergy between all the different things that are being post-trained, I don't really know, to be honest. I feel like there are so many specialized domains, so many specialized skills out there that you only really need in one place. I wonder how much of that really is contributing to the overall intelligence of the model. A lot of the intelligence seems to come from coding and math training and things like that. at the moment. Yeah, this seems a really interesting question for how much it makes sense for enterprises to post-train their own models, right?

30:44It's like there's pharmaceutical companies that have data sets of drug trials that are pretty proprietary. There's chip design companies. There's certain things that you're like, yes, those companies have quite unique data that probably isn't out there. Maybe it's less clear to me if your average bank actually has data that isn't just going to be in, that eventually your closed-source model providers get access to? Yeah, we might see some of those businesses struggle if they're not able to retain their competitive advantage, if they're not able to leverage their own IP to make their own models better.

31:20So I think it really depends on the industry and that's kind of how you can guess where things might go and how the transformation might happen. I would say the worst place you can be in is if all of your knowledge is on your internet. if everything that makes a company unique and special is already out there and can be scraped, that's a very dangerous place to be. It kind of exactly ties into the overall thesis of River, which is keeping this intelligence local, the kind of ways you're interacting with and using these models. I'd say we have a choice, actually. I think as a company, you can choose to give your data to somebody like Anthropic and open it in a way through the APIs, through other means, for maybe selling your data to them, even if you want to turn a profit on it.

32:04Or you can start to leverage it to build your own models. So I see these two different paths. One obviously leads to more centralization of intelligence, and one leads to more distributed benefits of AI. Behind all the stuff you're building today, it seems like anyone that's trying to build their own models or fine-tuned models is using the cutting-edge Chinese open-source models. And obviously there's been a big discourse these past months about these models. Obviously, I imagine you think cutting-edge open-source models are really good for the world. What do you make of the fact that this ecosystem is all Chinese today, and then kind of the government talking about potentially banning these models?

32:40Yeah, I think first of all, it is good that we have powerful open models out there. So it really lowers the bar for anyone who wants to get into AI, who wants to customize models, who wants to use models to their own advantage, and really have that control of their own destiny. At the same time, it does put the Chinese labs in a kind of a preferred position where they also can exhibit some control. So for example, they might stop releasing their open weights in the future and everything relies on them. That's not a great thing for the US economy. They could also change the licenses to those models.

33:22So we often see that already, that there are certain times that if you're a company that has more than X revenue, then you need to enter into a partnership agreement where the rules might be a bit different. So that's another way people can exert a bit of pressure and stay in control of the open weights. And then one issue that's more theoretical, but is theoretically possible, is for the waste to somehow have backdoors or behaviors that we would find undesirable. How worried about that are you? Yeah, today I'm not very worried at all because my sense is that if these kinds of things existed, we would already have at least one example of it happening in the wild.

34:05We haven't seen any reports of it. Also, the technology that I'm aware of for kind of planting these behaviors, planting these backdoors into weights is not advanced enough today to really avoid detection and to not leave any hints. So I think we're kind of early on this. right now and I'm not too concerned, but in theory, the concern is there. And over the next two years, as this tech becomes more sophisticated and you can actually, and maybe LLMs and agents become more important as well in all kinds of domains, including the military, you know, that's, it is something that we should have on our radar.

34:44So I would say it's really critical that we have the best open models in the US. So that's something that I'm very supportive of. and I think we should all figure out how can we get a US team to train the very best open model, period. Not just the best US open model, but the best one worldwide. So yeah, definitely super supportive of that. Yeah, it seems to be an interesting business model question of how you end up doing it, but it's such a public good that maybe you'll find a bunch of folks. I mean, NVIDIA already is subsidizing this stuff quite dramatically to make it possible. And so maybe you'll see.

35:20Maybe it's going to be us as well. So we've had that experience with XAI of starting from scratch. And then within less than two years, we had Grog3 and then Grog4 as well, which were very much at the frontier. So I'd love to train the best US open model. But right now we're very interested in how to build a business on these open weights so that the company can be self-sustaining. And we're making good progress on that with the RL API, with the other products that we're building. Well, I'd be remiss if, you know, having you here not to ask a little bit about some of the past roles you've had, because I think you've obviously seen, you've had a front row seat and kind of led a lot of this stuff that's happened over the past years.

36:02You know, maybe to start, like, you've said AI progress took off, you know, faster than you expected. Is there a moment or two that, like, you know, really surprised you these past, like, you know, I don't know, three, four years? I think the crazy thing is I'm not too surprised by the progress. It's just crazy living for it. You know, it's totally different from, you know, when you imagine it, it's just so different from actually living it for real. It's almost like being in a dream. So when I started to get interested in AI, I was a physicist. I was working at the Large Hadron Collider, was doing a PhD there.

36:39And in physics, they teach you basically to really think ahead. I think some of that rubs off on the physicists as well. And they start to think about, you know, how will the future, you know, look like? And so when I realized that, for example, AlphaGo happened, so DeepMind was able to create a Go AI that's very, very powerful, then for me, that's when it clicked that amazing things are going to happen in AI and I better switch over. You and a lot of prominent researchers made the move from physics. It's very interesting how many folks started in that world. Yeah, I think it's just a great way to learn the ropes about how to think about these tough problems and how to solve problems from first principles.

37:21So it's a great preparation for doing AI work. And I decided to join DeepMind as a research engineer, which was really fun because I got to see both the algorithms side and how to solve these research problems, how to train the models, and also how to build distributed systems, how to improve efficiency of the training of inference, and so on. Then worked on text-to-speech generation with WaveNet, and then got really interested in reinforcement learning, and ended up being a tech lead for the StarCraft project at DeepMind. So it was a really challenging RL project. In some ways, it was ahead of its time.

38:01100%. I mean, that seems to be a much more complex environment than a lot of things we're optimizing RL on today. Exactly. We have to figure out how to deal with that complexity. And some of the strategies we've developed, we could immediately see how they could apply to language models as well down the line. So for example, with StarCraft, we did a lot of imitation learning. So we got all these replays of people playing StarCraft at various skill levels, from beginners to almost professional players. And then we started with imitation learning, and only then switch to RL. And the rewards here are verifiable.

38:38It's a two-player game where the one gets a positive reward. If you win, you get a negative reward. And the combination of those two, imitation and then sort of large-scale training for imitation and then reinforcement learning to further improve the policy, that's really, really powerful, we found. And that's still the way that these agents are trained today. So really after the StarCraft project at DeepMind, we actually realized that coding is going to be the next big thing. So it was a bit early at the time. So it was way before there were any commercial sort of capable commercial code models out there.

39:17And we kind of saw it as a natural extension of StarCraft. And so we started this project called AlphaCode at the time. and I had the strong feeling that models, that basically LLMs were not performing well on code at all at the time. And I thought the reason was that they're not able to think, we don't give them enough time to think about the problem, basically. So there's, when humans solve tough problems, they have to think for a moment or two and something magical happens that allows them to understand the problem much better and to solve it. and we didn't have a mechanism back then for LLMs to do that kind of reasoning.

39:58And so I really felt like we got to do some research on this and who's working on that. And I realized OpenAI has just started up a reasoning team exactly to figure out this problem. So I joined OpenAI, moved to California and yeah, I was both involved with the reasoning effort there but also helped out on some big pre-training runs together with Greg Brockman. So I kind of got to see the large-scale pre-training side there as well, which was really fun. A lot of this reasoning stuff seems to have worked post really large-scale pre-training. And so it seems like in the early days of the reasoning work at OpenAI, there was almost just this conviction, like this is going to work as models scale and get better, which is really impressive intuition.

40:44Yeah, exactly. And then at the time, we didn't have the solution just yet. The solution ended up being essentially the O1 model and the research that happened later at OpenAI to unlock the reasoning approach where the model has thinking tokens that it's allowed to generate before giving the final answer and then use reinforcement learning to basically improve its thinking process and thinking ability. So that ended up being really the key that we were looking for, that happened a few years later. And yeah, and then at OpenAI,

41:25DeepMind had a who's who of heavy hitters. I think almost everybody did a stint through their Google brain at some point. I think as people have reflected on it, it's like maybe there was more different kinds of bets made or more of a belief in reinforcement learning and less of a focus. I don't know, you lived through both of them. So I'm curious how you thought about that. It's really hard to say, honestly, because there were all these extraordinary thinkers at DeepMind, and I think some of the right ideas were kind of floating around for sure. But I think what helped OpenAid was that they were a smaller team and they were able to get great alignment within themselves, great conviction that this is the right approach.

42:07I think it requires kind of that critical mass of talent density and people thinking along the right lines, which they were able to do. So even as the underdog, they were able to make huge progress. And I think to us at the time, it was pretty clear that OpenAI was going to win in many ways. And then already at that point, I started to think like, wow, that's kind of unfortunate if there's only one AI company in the world that controls the most capable models that decides what gets done. And we would even think about how do we set up UBI for everybody so that humanity can be paid while OpenAI essentially controls the models, which just didn't sit right with me.

42:55I was always a big open source fan and big Linux users over the years, so I felt like that's the way to distribute the benefits. In the end, philosophically, if you think about it, the pre-training of the model happens on all of humanity's knowledge, all the texts that we've created out there on the web. It's really a commons. It's really something that humanity as a whole should own. The fact that nobody technically owns it, you know, and you're able to just download it basically allows you to compress the knowledge, to refine it, to turn it into something that's actionable, that's useful in the form of the model weights.

43:33And that's really what the AI model builders are doing, in my opinion. And to me, philosophically, it makes the most sense for those pre-trained checkpoints to go out and be free for people to use. Just thinking from first principles, what's fair here? But obviously, the AI labs are putting in tremendous effort in terms of in the form of compute, in the form of all these experts that figure out how to set up the algorithms, how to train the model, how to innovate in the space. So there's definitely a huge benefit that comes from having them involved. But I feel like we're not accounting for all the human ingenuity that went into the text on the web.

44:19But obviously the fact that all that data is open allowed you, in the case of XAI, in short order to kind of get back up to pretty close to the frontier. And I feel like that's been interesting to see that play out. How did that come about? Yeah, so basically I ran into Elon. I randomly happened to be at the Tesla office and talking with the team there about what they're up to. And he happened to be around. So I got a chance to talk to him, which was pretty fascinating because he's such a great thinker as well. So thinking very deeply about humanity, technology, and everything in between. And yeah, so I think we kind of saw eye to eye on all these things, like the idea that maybe there should be another competitor in the world, another AI lab that has a bit of a different mission, a different philosophy around how the models should be developed and basically how they should behave when they're out in the wild.

45:19So I think we kind of had that connection from then. And then later on, he wanted to start something new, he reached out to me, and that's kind of how XAI started. And yeah, and then from then on, I talked to some old friends that were looking for something new to do, and we managed to get a pretty good group of people together. And I think that's critical for any AI effort. You have to get that critical mass of talent, the right kinds of people that want to work pretty hard and do something great. That was really the number one. The question of whether we were able to do it would depend on whether we got the people together.

46:02And then from there, starting pretty much from scratch, we sort of slowly built up the company and trained the models. and within less than two years we were able to get to the frontier. We did some pretty crazy things famously like getting costs up and running I think in 120 days and I guess what lessons did you take away from just the speed at which you were able to do a lot of that stuff? Yeah, I think it just requires a tremendous focus and thinking outside of the box. That was really very much driven by Elon because he saw the need for us to bring more GPUs online very, very quickly to be more competitive, to train Grok3 in particular.

46:45And so he decided, you know, let's build our own data center. We don't want to depend on anybody else. And he's really the best person in the world to do that from what I've seen because he's able to see the big picture of what needs to be done, what are the individual components and steps, and then sort of a very focused manner figure out from first principles how to accelerate every aspect of the project and then able to go out and do things that other people might not actually be aware. Did you think it was crazy when it was first proposed on timeline perspective? I did, but I also knew Elon well enough at that point that I knew that we'd get it done.

47:29So I also had tremendous confidence. and really the key to making Colossus work was just questioning how everybody else was building data centers because out there when we talk to people how long would it take to build a new data center and put this many GPUs inside we would get quotes way above a year and the reason is because they had to bring up a new shell like a new data center building and had this like very, you know, waterfall-like process for building it. And you also often deal with subcontractors of subcontractors of subcontractors. So a lot of efficiency gets lost or can get lost in the process there.

48:17So Elon has a totally opposite mindset. We want to do it ourselves. We want to be the general contractor. And here are all these ways of solving our problems that other people haven't even realized are possible. So it's kind of how to find a glitch in the matrix on every single aspect of the project. So yeah, really fascinating working with him on this. I'm sure our listeners would be super curious. What is it like working with him? And obviously, how do you compare that environment to, obviously, you've worked in DeepMind, OpenAI, three probably very different environments and cultures. I'd say that the reason I was really interested in working with him after that initial chat is that I could sense the respect that he has for engineers.

49:02So as a research engineer, it's kind of jumping between research and engineering quite a bit. And yeah, I wouldn't say that there's a necessary double standard at these AI labs, but definitely, you know, you've got titles that kind of affect your research scientist, your research engineer. And I just felt like really respected by Elon and felt like we were seeing, I tell you, he was another team member that I was working with. He focuses a lot on engineering meetings. So really talking directly to the people during the actual work dives tremendously into the details of everything that the company does.

49:42And you can feel the respect for the engineers, the respect for the problem that they're working on. And yeah, that was a really fun aspect of it. And also he has this tremendous energy. and every day he comes to work with a smile and wants to tackle all the hardest problems once again from the beginning. And that energy, you can feel it in the team as well. And I think that's...

50:14They come in with tremendous energy and motivation. Since you left, the company's done some pretty interesting things, obviously getting acquired by SpaceX and then acquiring Cursor. What would you make of the Cursor acquisition? Yeah, I think it's an incredibly smart move. I think it's basically allowed him to jump way ahead on coding models. So Cursor both has a great team that's very familiar with coding. There's a product that a lot of people out there use. I've also used Cursor quite heavily. And there's a lot of data that you can use to kickstart your training and tremendously improves over the states where XAI was at.

50:55Yeah. And so is that coding data from the real world, from people using it, just that much more valuable than the stuff you get from data labelers and the other suppliers? I think the quantity of it is really, if you have it in quantity, that's really, really valuable. There's also a trend that's value in building the right kinds of RL environments. so basically setting up these verifiable tasks where the agent is tasked with doing something fixing a bug in the code base and then afterwards you check did it actually do it correctly and also did it do it in the right way and that's really what's been able to deliver a lot of improvements in coding models and that's kind of the other side of the coin you need kind of one both types of data in the process but obviously it's way easier to figure out what environments to build having seen these things deploy I'm sure way easier when it's actually your own model in the cursor harness, but even Claude or whatever in the cursor harness is probably still really valuable.

51:54Yeah, and obviously I don't have any insight into what kind of data the cursor team brought in and how they were able to make such a huge leap in model performance, but I'd say it's super impressive how quickly they managed to ramp up the coding abilities of the GROC models. And I'm sure we haven't even seen the biggest jumps yet, so there's some additional models that are coming out. What do you feel like the biggest constraint is on continued model progress right now? Is it data? Is it just getting all this expertise that lives within organizations that hasn't been fed in? How do you think about that?

52:29I think the biggest unlock would be just new ideas around how to set up the training such that it can handle much longer time horizons, such that it can handle non-verifiable rewards much more easily. and I think there's a big inertia and there's a big energy barrier to getting these things to work and it's kind of easier to continue to iterate on the coding regime. So I think one trap here would be just to continue to make the coding better and better but I think what people really want to do is figure out how to move into those longer term domains. So for example one quantity that can bottleneck your training is how long does it take to roll out your agent directories.

53:13So if you're in order to update the model, you first sort of do trial and error. You have your agent attempt many, many solutions. You collect the rewards and then you can use reinforcement learning to update the weights of the model. But if your rollout takes 24 hours, you're going to have a really hard time training the model, right? Because every single training step will take very long. So we have to figure out ways of slicing up the work that the rollouts, the work that the agents are doing. And so being able to incrementally train them, maybe agents have to guess, do they do well, do they not do well, based on incremental progress.

53:49And then moving into all these domains where we don't have the rewards, where we can't have a unit test, or we can't have a proof, or a verification of a proof. So that's really where people should be pushing today if they want to improve agents. How close do you feel like we are to cracking some of the non-verifiable domain problem, I guess? Maybe back to your AI scientists and the ability to have a novel physics discovery or something like that. Do you have a gut intuition as to how close we are to some of that stuff? I think that the models are so capable now that they can often make amazing judgments about whether something has worked or not.

54:30But we still haven't seen great examples of people implementing that at scale to improve the models further. So I think that's something that we might see in the next few months, next 12 months or so. People actually pulling off these LLM judges, the approaches with LLM judges where really, there is no verifiable reward, but we're still able to learn a lot from the reward interactions. Yeah, but you think the recipe is kind of known and it's just literally about running that experiment? Yeah, I think people maybe haven't figured out all the details of the recipe yet, how to optimally do these things.

55:04But the thing that's been helping is just the models themselves becoming so strong. Now that you can often trust their judgment, they're often very reliable at determining what's happened. So that's kind of something that lifts the whole thing up. Yeah. I'm curious, you've obviously, it seems like, had conviction early in a lot of ideas that proved correct and obviously super early to reasoning and a bunch of these things. In the last few years, or maybe in the last year, anything you've changed your mind on that you kind of actually felt pretty strongly would happen in the AI world and has gone in a different way?

55:39Well, the thing is, I think nobody could predict how transformative the coding agents would be once they crossed a certain threshold of ability. So I was expecting kind of smooth progress on coding models. They just continuously get better, and eventually they'll be used more and more. But this sudden explosion of usage, that was really something else. And now I'm starting to think, And maybe we'll see more of these transformations in the future. Yeah, there's just a step change moment. I feel like everyone felt that around November, December. And it's even hard to articulate even what it is. But it just starts working so much better.

56:20And you can imagine, yeah, in other domains, you'll reach some sort of point where that becomes apparent. But it's kind of hard to predict. Yeah, I think what's fascinating is even the folks closest to those models. It was hard to predict a priori. That's when the breakthrough was going to happen. I was thinking about where to end. And one thing, maybe to tie it back to where we started, which was really around you writing this fiction and thinking about where the world's headed. And obviously these are questions that motivate River and your founding there. I feel like you've written the question, obviously keeping many people up at night, is if machines can do all this work, what's left for people?

56:55I feel like the future, if we want to stay relevant in the world as humans, we want to figure out the right kind of symbiosis. with the machines. So today there is a bit of symbiosis going on. If I'm using a coding agent, it takes care of a lot of the low-level details, takes care of tasks that I kind of understand at a high level, but I might not want to type out the code myself, and I still get to do the architecture, the planning. And I think that's a great mode to be in where the human and machine are working together, but there's a big risk that the balance will shift over time. So as the coding agents and as agents in general become more capable, the incentive is always to let them take more control, let them do more and more.

57:45And this is kind of what the story is about, because once you let the agent make all your decisions for you, you're not controlling your life anymore. You're not controlling the situation anymore. And so I think what we should try to do is figure out how to keep the symbiosis going. And for me, that requires new advances in alignment. So the kinds of alignment that we do for the models are often now around kind of learning from human preferences. So a lot of this group of humans, they like this kind of behavior, they don't like this kind of behavior, and we modify the model to comply with that.

58:22But I think the future, in order for humans to stay relevant, it will need a much deeper alignment, much deeper integration between human and machine. And ultimately, maybe there'll be something like Neuralink that allows you to control large amounts of intelligence just by thinking. But I think that's still some time away. So in the meantime, we can align the models better by changing the way we train them, by innovating on the algorithms, on the technology. And that's kind of what River AI is trying to do. So I think that's something I'd encourage anyone, anyone who's in AI research also

59:03our intentions and helping us rather than taking over and replacing us. Yeah, and it seems like in this context, helping us is still giving us a role, even if it's maybe suboptimal in some way of maximizing whatever paperclip objective there is, it is maximizing at least human flourishing. Exactly, and we should train the models to maximize human flourishing. That's another takeaway. You know, there was obviously this letter over the last 24 hours that many researchers signed, you know, I think it was released yesterday, calling for the government to kind of consider slowing down AI. And there's all this kind of concern around just how fast is all moving.

59:42Like, is that something that resonates with you? Like, you know, how do you think about that? Yeah, I think it's incredibly difficult to slow down at this point. AI has become such a big thing. You know, the U.S. economy is very reliant on AI making further progress. as well. There are international competitors out there. China is quickly catching up in terms of AI capabilities and they might or might not decide to slow down as well. So I think it is really, really difficult to get people to slow down. I don't know how realistic that kind of the latter really is. Obviously, I would love to be surprised if we do manage to slow down a little bit, do more work on alignment.

1:00:24I think that would be a good thing. But if we're not able to slow down, I think we should accelerate on technologies that can improve alignment, the AI models that can improve safety, because that's going to quickly become very, very relevant for the models that we deploy. And is the best way to do that just more open source models and more effort within the closed source providers? Or if you had a magic wand, what would help increase that amount of work? I think open models that are close to the threshold of where they might be starting to become dangerous are really, really valuable because because it allows anybody in the world to try out their ideas for how to align these models, control them.

1:01:02So they're not so powerful that you can do real harm with them, but they exhibit some of these things that will become dangerous in the future. Cyber security capabilities are one of the biggest ones right now. So I feel like we don't want to just have the few thousand people at the big AI labs thinking about these problems. We should have as many people as possible trying to figure out What are the algorithms? What are the tools that we need to build to make these things viable? I mean, you've obviously been closer to this stuff than anyone. What's your current probability that this all ends up okay?

1:01:34That's a good question. I mean, the question is okay for whom at this point? Because potentially AI could amplify inequality in the world. And there could be people who benefit greatly from it. And then there could be people who kind of feel left behind. I think that's the most immediate, urgent risk. And then, yes, there are risks in the future. Can the AI models take over? Can they take control away, even from the people that have built them, that are controlling them today? And I think that's also a risk, but it's so hard to make predictions about that. I feel like that's going to be in the further future and the more immediate problem that I can see is, are we able to bring everybody else along on the journey?

1:02:20Yeah. Well, last question for you. You had in that piece you wrote, The Prayer to Machine God, you had this beautiful line, I think, that was like the tears you didn't expect when the model first spoke back. And so I have to ask, was that from a personal experience? Have you had something that felt like that? Yeah, I put that in there because I still remember how magical it felt to first get these models to work. And back then, it was very, very modest compared to today. I think this is something that all the early folks in AI tell us that when the models first started to do some pretty simple things, but things that you couldn't do with any other machine before, it was just really, really magical.

1:03:01So whether it was just writing a Python function that can generate prime numbers or something like that, just felt like this is unreal. So yeah, I wanted to capture that feeling. I'm sure that's something that a lot of folks that were early in AI felt. But now looking back, it all seems so unreal because the models have advanced tremendously. They're really affecting the entire world at this point. And it's gone so far beyond those little experiments that we used to do when AI was more of a hobby. Oh, totally. Well, I think that's a perfect place to end. It's been fascinating. Thank you so much for coming on the podcast and sharing all this.

1:03:41Thank you, Jacob. Appreciate it. I'm Jacob Efron, and this has been Unsupervised Learning, a podcast where I get to talk to the smartest people in AI and ask them tons of questions about what's happening with models and what it means for businesses in the world. As I hope is clear, I have a ton of fun doing this. It's a nights and weekends project. in addition to my day job as an investor at Redpoint. But our ability to get these incredible guests on really comes from folks like you subscribing to the podcast, sharing it with friends. It's really what ultimately makes this whole thing work. And so please consider doing that.