1. Podcasts
  2. Training Data
  3. David Silver and Lasse Espeholt of Ineffable Intelligence: A Superlearner for the Age of Experience
Training Data

13 Oct 2026 · 1h 8m · 28 chapters

Episode artwork

David Silver and Lasse Espeholt of Ineffable Intelligence: A Superlearner for the Age of Experience

In short

Ineffable Intelligence’s “super learner” goal: build reinforcement-learning agents that autonomously discover knowledge indefinitely, with depth (from nothing to beyond human understanding) and breadth (learn from any stream of experience). They contrast this with today’s language-model + human-data approaches and argue for “crucibles” (emergent, complex environments) plus simple objectives that yield emergent complexity.

Guests and backgrounds

  • David Silver: 2009 PhD on reinforcement learning for Go; led work demonstrating superhuman performance in Go (AlphaGo), inspiring modern RL/deep learning.
  • Lasse Espeholt: worked at DeepMind scaling reinforcement learning on GPUs; contributed early scalable RL methods (parallelization/architectures); also worked at Google Brain; co-founder of Ineffable.

Key claims

  • Superintelligence requires learning from experience without heavy human priors; removing human data can outperform human-in-the-loop (AlphaGo Zero vs AlphaGo).
  • Crucibles can be Turing-complete/sufficiently complex for computation, language, math, and science discovery in simulation.
  • Alignment should be approached via the science of superintelligence in simulation; language-based monitoring will become unreliable.

Notable examples

  • AlphaGo (human-data, years of work) vs AlphaGo Zero (hours/days, no human data).
  • Go’s simple rules but infinite emergent complexity with multiple agents.
  • Evolution and human history as analogies for emergent complexity from simple optimization signals.

Written by AI. May contain mistakes.

Chapters

0:00

The Essence of Collective Discovery

Explore the idea of collective human discovery as a foundation for AI development.

“Collectively, we started with this kind of gray mush in our head.”
1:12

Reflections on Reinforcement Learning

Dave Silver shares insights from his PhD work and its impact on AI.

“Your 2009 PhD was on reinforcement learning applied to the devilishly difficult game of Go.”
3:40

The Decision to Found Ineffable

Dave discusses his motivations for creating Ineffable Intelligence.

“Okay, we're going to come back to that later.”
6:01

The Meaning Behind 'Ineffable'

Explore the significance of the company name and its implications for knowledge.

“And it's going to be something which is profoundly amazing and perhaps so profound that we can't even express it in human language.”
7:05

Léze's Journey to Ineffable

Léze Espeholt shares his background and reasons for joining Ineffable.

“Well, you were in a different group than Dave in the deep learning group at DeepMind, and you spent years scaling up reinforcement learning on GPUs.”
8:36

Defining a Super Learner

The concept of a super learner is elaborated and its significance explained.

“I knew Wojtek and Chris from back in the day, and I thought the team was great.”
9:50

Super Learner vs. Current AI Models

A discussion on the differences between super learners and current AI models like LLMs.

“So to me, a super learner, it's something which is, it's got incredible capacity, both in its depth of what it can learn and in its breadth.”
13:00

Building the Future of AI

Insights into the practical steps taken to build a super learner.

“What I mean by a super learner is something which, first of all, discovers completely autonomously.”
14:02

Vision and Practical Implementation

Learn about the ambitious vision and day-to-day activities driving the project.

“And I'd love to understand, you have a big, big, big vision.”
16:46

Emergence of Complex Discoveries

Explore how simple objectives can lead to complex discoveries in AI.

“And after kind of following that trail, we're kind of pulling the agent by its nose towards a particular desired outcome.”
Show all 28 chapters
19:58

Crafting the Right Environments

Understand how the design of environments affects learning and discovery.

“So if we can get that process right, you know, things are going to take off and lead to, you know, unbelievable things.”
24:08

The Role of Human Language and Priors

Discuss the limitations of human language and priors in AI development.

“Yeah, like the end goal is to have something that is grounded in the real world, right?”
27:40

Betting on Removing Human Influence

Learn about the strategy of removing human priors to enhance AI intelligence.

“that anyone can point to of what happens when you remove human priors.”
28:00

The Challenge of Research Questions

Understanding the nature of compelling research questions and the importance of uncertainty in innovation.

“where I think you ask research questions which are like on a knife edge where it's like 50-50.”
29:08

Building a Super Learner: Lessons from AlphaGo

Exploring the transition from human data reliance to developing a self-discovering algorithm in AI.

“And it comes back to this, what does it mean to build a super learner?”
31:04

The Implications of Human Priors in AI

Discussing the role of human biases in AI development and the necessity to minimize them for progress.

“Um, so in the, in the game of chess, it took four hours to beat the world champion, the best ever AI that had ever been created in the game of go.”
33:17

Confidence Levels and Timing in AI Development

Examining the importance of timing in achieving breakthroughs in AI and the confidence in creating super learners.

“You mentioned that you like, philosophically, that you're drawn to research that's more on a knife's edge, 50-50.”
36:29

The Role of Compute in AI Progress

Understanding how computational power drives advancements in AI and the opportunities it presents.

“And, you know, that's what it means to take risks and do something exciting.”
38:06

Simulation and Its Potential for Superintelligence

Debating the limits of simulation in AI and its capacity to foster superintelligence safely and effectively.

“So yeah, we can really test this out in a way that we couldn't do at DeepMind, for instance.”
42:01

Exploring Simulations and Science

Learn how simulations can help in understanding scientific concepts and achieve faster superintelligence.

“Science in some sense is the study of our world.”
43:45

The Challenge of Alignment

Understand the difficulties in aligning AI with human values and the need for new approaches.

“I'd love to talk about alignment and maybe less.”
46:51

The Future of Monitoring AI

Discuss the limitations of language-based monitoring for superintelligent agents and explore alternative monitoring methods.

“So I think these are two things that I believe we can do.”
50:27

Pacing AI Development

Debate the need for pacing the development of AI in the real world while accelerating simulations.

“But the ones which do support the flourishing of humanity will will end up kind of supporting us and protecting us against the ones which aren't acting on our best interests.”
53:05

Compute Infrastructure for AI

Learn about the compute challenges and architecture necessary for training and simulating superintelligent agents.

“This is going to be an incredibly compute expensive journey and you are pushing the limits of what compute can do.”
56:01

Exploring Scaling Laws in Reinforcement Learning

Learn about the scaling laws related to reinforcement learning and their implications.

“And maybe in the last few years, also inference, but less about simulation, right?”
58:08

Foundational Work in Reinforcement Learning Algorithms

Discover foundational algorithms like Impala and Seed that shaped modern reinforcement learning.

“What are those, and how do they relate to the work that you're doing at Ineffable?”
1:00:05

Building a High-Performing Team at Ineffable

Understand the philosophy behind team building at Ineffable and the importance of culture.

“I think it's actually kind of fun to see the reinforcement learning space now versus like 10 years ago, or like, let's say like eight years ago, maybe.”
1:02:33

Research-Led Culture and Future Predictions

Explore how Ineffable fosters a research-led culture and hear predictions for future advancements.

“So we're not like fighting for resources from day one.”
A man walking through an airport, asking his phone a question

Ask about this episode

Get the gist, explore an idea or ask about something said.

Claude

On Training Data, what are the key points of this episode?

Searching your listening

Transcript

Automatic transcript. May contain errors.

0:00Collectively, we started with this kind of gray mush in our head. We somehow started out knowing nothing at all and somehow discovered everything that humans know today was this process of collective discovery of languages and technology and science and mathematics and all the rest of it. Everything we learned, we discovered it for ourselves. And that's an incredible thing that no one has cracked yet in AI. And so to me, that is the essence of what it will take to get to super intelligence, like a system that can just keep learning and discovering knowledge forever.

0:49I'm thrilled to have Dave Silver and Léze Espeholt with us, two of the co-founders of ineffable intelligence. Great name, by the way. Fun fact, Dave, you were the first person I cold emailed to come on Training Data when we started the show 100-ish episodes ago, and it feels like a full circle moment to finally have you on the show. So thank you for joining us. Thank you. Dave, let's start with you. Your 2009 PhD was on reinforcement learning applied to the devilishly difficult game of Go. You demonstrated for the world for the first time that superhuman in one domain, machine intelligence, was possible.

1:29And I think you were really the figure that inspired the entire current generation of reinforcement learning and deep learning researchers. And so thank you for your contributions to where the field is. I'm curious, thinking back to that point in time, what did you learn? What did you have conviction on? And what has remained the same? And what has changed as AI has changed over the last five years or so? Yeah, thank you. And pleasure to be here. And I'm glad I finally made it after these 100 episodes. It's a pleasure to be here. So I suppose I always had this conviction, starting my PhD and going into the game of Go, that the right approach to solve AI was through learning from experience, through systems that discovered things for themselves, that were learning through reinforcement learning to get better and better and better, and that that would be the thing which would lead to the emergence of intelligence and ultimately superintelligence.

2:30And so I suppose we were able to show that in the game of Go, but it's a conviction that has held true throughout my career. And here we are now applying that conviction, ineffable, trying to show that indeed superintelligence can emerge all the way through systems that learn from experience. And in terms of things which are different, although we are not pursuing the human data path that the LLM world has pursued, I do think there's lots of interesting lessons to learn from that. In particular, it was definitely a surprise to me how a system that was trained on a massive amount of data could then be very quickly fine-tuned on a very small amount of data to kind of completely change its behavior.

3:16That was something which I think was something which sort of changed my mindset in terms of, you know, just suggesting that if you get things at the right scale, that a lot of things which seem like incredibly hard or like they might be very slow, that once you've kind of experienced enough data, that after that, the things that were slow can become very fast. I think that's a lesson that wasn't obvious to me in advance until it happened. Okay, we're going to come back to that later. You could have run this program inside DeepMind, where you could have probably had more access to compute than just about anybody.

3:51You decided the right thing to do was to start a new lab. Why did you do that? Yeah. So, look, first of all, I had a fantastic time at DeepMind. I loved my years there. and I have a huge amount of respect for the people there and still got on with them very well. But at the end of the day, I have a really strong belief about how we can go beyond where we are today with AI and all the way to super intelligence. And the path that I believe in doesn't actually go through LLM-based products as an intermediate point. And when I really thought about that, I thought about it really hard. I realized that I felt that so passionately that this deserved to be more than just the corner of an organization that's focused on LLM-based products and deserved to be represented by an elite AI lab that really just gives this approach its all and represents it 100%.

4:52And, you know, once I realized that, it felt very natural to create this company. I suppose another way to look at it is that the world already has tens of thousands, if not hundreds of thousands of really talented researchers all pursuing exactly the same approach, really just kind of doubling down on LLMs with pre-training, post-training, coding, and that kind of LLM path. Am I going to be the 100 ,001 person that's pursuing that approach? approach? Is that the most effective use of my time and my energy? And in the end, I realized that there's this other lamppost that right now no one is looking under and where if I actually put my effort there, there's a much higher chance of really having a massive impact on the field.

5:41Because if this works, it's going to change the whole of AI. So I think that kind of realization also meant that this was something which just means that it's the best way to have impact upon the field through this approach. Plus, you had a company name that couldn't go to waste. So that's right. Yeah. Do you want to say where the mind's called ineffable? Yeah. So I suppose that the name ineffable, it came from this idea that, you know, there's really just knowledge could be so incredible, like the things which superintelligence will one day discover, like this knowledge is going to be so unbelievable, incredible.

6:18And it's going to be something which is profoundly amazing and perhaps so profound that we can't even express it in human language. And so really, it's like, you know, in line with this other path that doesn't go through human language based approaches, but instead build systems that discover things for themselves and has the possibility to discover knowledge that's unbelievably profound and ultimately ineffable. I should say there was also a secret second reason that we used, which was that, you know, it's this idea. It really it has to work. Like if you really just double down on learning from experience, it's an approach that just has to work.

6:59And so you could say that I can't possibly F up. So it's ineffable. So there you go. There's the secret reason as well. Amazing. Best lab name. Lese, what about you? You were at Google. Well, you were in a different group than Dave in the deep learning group at DeepMind, and you spent years scaling up reinforcement learning on GPUs. What made you decide to leave DeepMind and what drew you to co-found Ineffable? Right. Yeah, so I joined DeepMind in January 15, and actually I started working a bit with Dave around 16, where I was doing this work on scaling up reinforcement learning. So as you said, I was part of the deep learning team and he was leading a reinforcement learning team.

7:42But we were working on an internal project together. I actually left DeepMind twice. So I left DeepMind in summer 18 and went to Google Brain. Then eventually Google Brain and DeepMind merged and then I was back again. But at the end of the day, I left DeepMind again like a year before Dave left. And the first time I heard about Ineffable was actually through one of our eventual investors. it's like reaching out to me asking what i thought about dave well my first like reaction was damn i missed out of on an opportunity i should have you know reached out to him but i was you know i recall like davis more often than not being one of the smartest people in the room and you know shortly after that uh sync with that investor um they actually reached out to me so i was like great But yeah, so I really love to work with Dave.

8:36I knew Wojtek and Chris from back in the day, and I thought the team was great. And I also thought it was amazing to have an opportunity like this in Europe. It should be said as well. We worked together on some of the early approaches to combining reinforcement learning with deep learning and trying to scale things up. And it should be said that Lasse, who, by the way, is very modest, But he actually worked on some of the original approaches that were the first attempts to actually scale up reinforcement learning and show that it could be done at large scale and ways to parallelize reinforcement learning and ways to kind of build out architectures that were scalable.

9:14And if you look today at the way the field's gone, like actually all of the ideas that people use today kind of sort of hail back to that original work that Lassa did back in the day. That's where all the roots come from of scaled up RL. So he was the first to figure it out. Wonderful. We should have you guys introduce each other instead of you. You're both too humble to brag about yourselves. Okay, let's talk about Ineffable and what you're building. You have a beautiful mission statement as a company. You're going to build a super learner and make first contact with super intelligence. Can you define both of those things for us?

9:50What is a super learner and what does making first contact look like? Yeah. So to me, a super learner, it's something which is, it's got incredible capacity, both in its depth of what it can learn and in its breadth. So in the depth dimension, it should be something that can start from not knowing anything at all and can go all the way through to discovering things that go way beyond what humans know. So an analogy to this would be the way that human collectively, we started with this kind of gray mush in our head. We somehow started out knowing nothing at all and somehow discovered everything that humans know today was like this process of collective discovery, like of languages and technology and science and mathematics and all the rest of it.

10:38Everything we learned, we discovered it for ourselves. It wasn't that someone came down and gave us like a blueprint of how this should work. And it wasn't that someone came down and gave like a trillion element data set of like, oh, this is what these things should look like. We actually figured it out for ourselves. And that's an incredible thing that no one has cracked yet in AI. And so to me, that is the essence of what it will take to get to superintelligence, like a system that can just keep learning and discovering knowledge forever. So the second element is breadth, that we want something which can do this incredible discovery from all the way to incredible things.

11:12And it should be possible to drop it into any stream of experience. So whether it's like a simulated world or a digital world or a physical world, whatever it's dropped into, it should be something that can just learn and learn and learn. So that's the dream. I think if we crack this, then to me, it gives rise to a superintelligence that will be like encountering magic. It will be something which is so profoundly amazing that it will be able to do wonderful things. What will it feel like when we first encounter it? Or is it one of these things where you can't predict ahead of time? I think it will be amazing.

11:48I think we should have an experience of wonderment. And the only thing I can say from my own experience is what happened when we got to this point in an admittedly much narrower field. But when we got there in board games, you know, when we got to the point of superhuman play in the game of Go, like it brought wonderment to the field. Like the Go players who saw like AlphaGo playing, you know, they were amazed. And like it was it was like a joyful experience for them to kind of witness this kind of Go playing. Same happened when we got to superhuman play in chess. And, you know, the chess players, they were so excited.

12:25And just imagine like going far beyond that, but like in, you know, things which are discovering incredible things, new fields of science, new types of technology that we've never even dreamed of. I mean, it will be it will be incredible. So the current paradigm, these large language models with reinforcement learning seem to be, you know, at least knocking out millennium problems and making some sorts of scientific discoveries. Does that count as a super learner or what's the difference between something that can make novel scientific discoveries versus a super learner in your sense of the term?

12:57Yeah. So, look, I think LLMs are doing a lot of great things and I have a lot of respect for what's being achieved. What I mean by a super learner is something which, first of all, discovers completely autonomously. And I think what we've seen so far, at least with these kind of approaches that have been taken, is there's a lot of human in the loop effort that's required to to make these breakthroughs. It's not fully autonomous yet. And it's also something which should be applicable to kind of real world embodied domains where the agent can learn from its physical experience. And we're a long, long, long way off that one.

13:32It's not at all clear that LLMs and language based approaches are even the right vehicle to achieve that kind of knowledge. Thinking about these things in terms of language might not even be the right paradigm. And it might just be that all of this human data that's being used in LLMs is kind of weighing things down. So a lot of respect for some of these, the progress that's made, but I don't think we're anywhere close to this like true super learner of something which can be like autonomously just discover everything for itself. What does building this look like in practice for you? And, you know, we backed Yanis, who was on your original AlphaGo team, and he told us that, you know, one of the things that's distinct about you is you can have these incredibly ambitious visions, but you also ground the team in very practical research milestones.

14:18And I'd love to understand, you have a big, big, big vision. What does this look like in the day-to-day and what are you implementing? How do you spend your days right now? Yeah, great question. So day-to-day, I suppose there's a lot of hard technical work by people like Lassa to kind of build the whole stack. That means we've got a state-of-the-art reinforcement learning stack that allows us to run things at massive scale, inhale data at the optimal way, be able to kind of learn and have systems that keep learning and learning and learning. And then on top of that, so we've spoken about the super learner.

14:56The other aspect which we're building is we're building out environments which are capable of super learners like discovering everything, like environments in which simulated environments in which a super learner can discover everything that we mean by intelligence. We call those crucibles. So they're environments in which it's possible for agents to make incredible discoveries in which it's possible to express and learn everything that we think of as intelligence. And so we have been putting a certain amount of effort into creating these environments. And so it's a combination of building out the super learner, doing the research to make super learners incredible and beat the state of the art of what's possible with reinforcement learning, both in terms of the engineering and the research that happens, and creating and placing the super learners into these crucibles to kind of see the emergence of incredible behaviors.

15:50And in terms of the grounding, you asked about how to ground this in the day-to-day. So we ground it in milestones, and the milestones are kind of analogous to, you know, what happens with the human journey where we discovered incredible things. And so those become natural milestones for us to for our agents to rediscover the great moments that humans went through in our discovery of knowledge. What do you think the agents will rediscover first when that's going to be your first moment of oh this is working? I imagine we're going to be surprised like I think one thing which is different about the way we take to doing reinforcement learning to a lot of the rest of the field is that we are not breadcrumbe trailing the particular objective to be learned about.

16:37A lot of the way things people do RL with LLMs at the moment is to kind of say, oh, you know, the agent should be learning about this problem and then this problem and this problem. And after kind of following that trail, we're kind of pulling the agent by its nose towards a particular desired outcome. We're not taking that approach. We're giving the agent some simple objectives and asking it to, you know, discover things that will surprise us and amaze us. So, you know, the agent is in principle capable of discovering all kinds of things. And we don't necessarily know in advance what those discoveries will be.

17:10And it's kind of just going to be exciting to witness that journey. So I hope to be surprised, I think is the answer to your question. Interesting. So what you're saying is instead of explicitly giving the agent the goal to go rediscover language or build a calculator or build a computer, you're going to give it other more open-ended goals and the agent may discover that the most efficient way to actually, or the best way to actually accomplish that goal, maybe to build some of these tools that we as humans have already built. Is that one way of putting it? That's one way of putting it. I think another way to say it is people often think in RL, oh my God, how do I come up with a reward function that kind of perfectly specifies this really complicated thing that I want to discover?

17:53Like, you know, how can I specify exactly what I mean by science? Or how can I specify exactly what I mean by interesting maths. But there's an alternative. And the alternative is that to give the agent something really straightforward to optimize, but in a really complex environment where there's a lot to learn about. And so it turns out that if you look at the way the world has worked in nature, the pursuit and optimization of a very simple, easy to describe quantities can often give rise to incredibly hard to describe complexity and emergence. So examples of that would be evolution, which kind of just optimize a kind of survival of the fittest, some kind of fitness signal.

18:41And the pursuit of fitness gave rise to all of the complexity that we see in life. In the human journey, the pursuit of relatively simple kind of goals that we as humans have gave rise to all of the complex behaviors that humans ultimately acquired and all of the knowledge that we eventually ended up with. So the kind of emergent complexity is much greater than the kind of starting point that was given. So it wasn't like evolution said, look, hey, humans, here's this idea of science and you need to kind of get to the most complex science that's out there. It was really something much simpler. So a way to describe this maybe would be, you know, perhaps that humans in the pursuit of our goals started coordinating a bit more.

19:26And then in the process of coordinating, we discovered that it was helpful to start communicating. And in the process of communicating, we started trading. And then in the process of trading, we discovered that, you know, basic arithmetic was useful. And then we started to discover that, you know, tools were useful. And then we discovered that some basic science was required to help understand how the tools would work. And that gave rise to more complex technology and understanding the technology required more complex maths and so forth and so forth. So all in the pursuit of relatively simple goals, this incredible complexity emerged.

19:58So if we can get that process right, you know, things are going to take off and lead to, you know, unbelievable things. It's fascinating. Are the rewards for your agents kind of similar to like human biology and evolution then? Like, you know, try to make it to the next generation. What does a reward function look like for you all? Yeah, I mean, we've been experimenting with quite a number of different rewards. So it's not like we have one answer to that question. But I think one thing we're not doing is the kind of, you know, crafting a reward that exactly describes what we mean by science, or exactly what we mean by technology, or exactly what we mean by language.

20:36We expect our agents to discover things. And if we can't crack that, you know, like, if we can't crack this problem as AI scientists, we're never going to have agents that discover whole new fields of knowledge that we don't even know about. There are vast fields that will exist that are out there that which we don't even know have the potential to exist at this point in time. You know, it's like imagine, imagine you're back in, you know, the days of Newtonian physics, and no one even knew that like quantum physics was a thing or that, you know, relativity was a thing, right? We should be in that process of open minded discovery where our agents are free to kind of figure things out for themselves.

21:12And I think that's going to lead to much more exciting outcomes. What do these crucibles look like in practice? And you're contrasting it to today when people say environments, oftentimes it's a set of tasks and what the agent should have done on the task. It sounds like your environments are, your crucibles are very, very different. What do they look like in practice? So we are building crucibles, which are, without giving too many details away have the capacity for a lot of emergent complexity like the key thing about them is it's like it's not like we've kind of handcrafted a whole load of puzzles for the agents to to to work with um there's just a huge amount of emergent complexity there some of that emergent complexity can come from different places it can come from from like physics engines that give rise to kind of complexity as you've got you know millions of interacting objects with with like a complex physics engine.

22:08You can have millions of interacting agents and the world becomes much more complex because it suddenly now includes the complexity of all of those other agents within the world that the agent's building. So there's ways to build worlds that are vastly more complex than the kind of standard simulators like Atari 57 or whatever it is that people have been working with in the past, and which are certainly more complex than the kind of standard benchmark kind of tasks that people are using. And if you get that right, you have a world in which there's the possibility for discovery by the agent. There's the possibility for the agents to discover everything from technology to science, to mathematics, to languages, to everything else, you know, just the same way are humans all the way through from more basic things like, you know, motor control or coordination.

23:00You know, the whole spectrum is possible. Like if you look at like a game like Go, right, which is like very simple to represent is just like a bitmap. The simulation is extremely simple. But you still see this like emerging behavior of something complex. Right. It's like super interesting to us. So like, of course, that goes like way simpler than we can imagine doing. But at the same time, going all the way to the content present realistic physics will be like very computational expensive. Right. So we want to like create this range of crucibles where where we have simple ones that can execute quickly and we can iterate on, but where we can still see very complex behavior emerging.

23:40Interesting. Yeah. It's almost like what you choose to represent is tricky, right? Because there's, for example, I can imagine a lot of economics arises from game theory, and most of that is completely unrelated to physics. And so there, a crucible that represents physics is probably not that helpful. But if you have information theory represented, which agents know what, then you probably do end up rediscovering quite a bit of economics. Is that a fair way to put it? Like the primitives you choose to capture in the environment kind of give rise to the type of learning that emerges. Yeah, like the end goal is to have something that is grounded in the real world, right?

24:19But stepping stones. I want to come back to the Go example because I really like this example. You'll be surprised to hear that I found Go a pretty good example. But what I find fascinating, which I think is what Lass is talking to here, is that Go is just like a 19 by 19 grid where you're trying to kind of maximize the territory. And if this was a single agent game, it would be like the most boring game you could imagine. It would be, you know, you just place down stones on a grid until you filled up the whole grid and there'll be no strategy whatsoever. The moment you put a second agent into the environment, you get this emergent complexity, which is suddenly because the agent contains like one agent has to reason about all the possible behaviors of the other agent.

25:03Suddenly it kind of massively increases the complexity of the environment. And to the best of our knowledge, you know, certainly all of the compute which we sank into it when we were working on it, there was just like this endless learning curve that where without end, you could continue to discover new things in this game. None of this is to say that Go is like the end point. Of course, it's still just a board game and quite narrow in a certain sense, but it shows that very simple environment dynamics can lead to kind of infinite, like an infinite onion of layers of complexity that kind of never stops.

25:37Infinite to the capacity of what our computers can do, of course. Do you think that the agents will end up rediscovering language? And will that language look like ours or will it look different? I mean, my personal belief is that anyone who thinks that human language is the end point of like, or the best way to reason, that perspective is a little naive. I think that it must be the case that there are far more sophisticated approaches to reasoning, maybe using paralyzable, rich, complex approaches that neural reasoning or whatever it is, these approaches will ultimately be far more expressive and far more capable than approaches which are based on human language.

26:30I would also say that starting from human language is like the approach for reasoning is somewhat problematic because it pulls in all of our biases about how we think agents should think. So for example, a nice way to think about this is to imagine, you know, what if we were to step back, you know, 500 years or 1000 years or 2000 years, and only like, we're able to use all of the language that that humans used at that time. So maybe our agents would believe that the world was flat, or that the diseases were caused by the four humors, or there'll be all kinds of, you know, strange beliefs that would be embedded in that will be very hard to kind of remove because they're so fundamental to the kind of language and the processes of thinking that humans have used.

27:21So I do think that going beyond this is a really important goal. I'd love to talk more about this concept of removing human priors. And I think your experience with AlphaGo and then AlphaGo Zero is maybe the best example that anyone can point to of what happens when you remove human priors. Maybe if you take us back to AlphaGo and then AlphaZero, was it surprising to you that removing human priors would result in even smarter intelligence? I wasn't sure, honestly. So I remember I always like to do things in research where I think you ask research questions which are like on a knife edge where it's like 50-50.

28:07I think it's not very interesting to do research where you already know the answer and you're like 95 % certain this thing's going to work. It's kind of like boring, honestly, to do those things. So, you know, I think research, good research is about asking questions where you're really not sure what the answer is in advance. And in this case, I was about 50-50. And I thought that maybe there was a good chance that it might, but I wasn't confident. And it was very exciting to see it. I should say that you asked about human priors. So we are making a bet, but ineffable. and the bet we're making is that we can go further by giving up like on on the prize that everyone's relying on today like the human data like the whole internet of knowledge that everyone else is is building their systems upon that's a uh an important choice and so we should probably talk about like why we're doing that and i think the reason for me of course it's you know a short-term sacrifice like we we give up a certain amount of performance um in order to access something which is potentially far greater.

29:08And it comes back to this, what does it mean to build a super learner? To me, building a super learner again is about building something that has this potential for full depth and breadth. So in the depth direction, we need to get to something which is capable of discovering things, which goes so far beyond human knowledge that it goes just far beyond the prior. And the more we lean on human data, the less able we are to actually discover the algorithms that are really going to take us all the way to something that's going to, for example, discover a whole new field of knowledge. And then in the breadth direction, we need to have algorithms that can be placed down and discover things in environments for which language and human data that we started with is just not a good fit.

29:50I mean, there are just many, many areas, different kinds of sensors, for example, where human language data or human data in general is not a good fit for some of this description. And so I think by letting go of the human data, we put ourselves in a position where we can actually discover something which can go far, far beyond it. So if we come back to the AlphaGo example, what we found there was that the moment we let go of the human data, which we'd spent, by the way, we'd spent about two years building AlphaGo, somewhat similar to the LLM approach, where we built every level of this, you know, we kind of hand put together the next level, we kind of put in a lot of human data.

30:36And then we were kind of human in the loop, like improving it, where we would kind of take the previous generation and choose how we were going to improve it to give the next generation. And so we were right in the loop of this process. It took us two years to get to AlphaGo and beat LisaDol. When we got AlphaZero right, you know, we gave up on the human data and finally focused on what would it be like to build an algorithm that could discover for itself. So once we got it right, that algorithm, it took just a few hours. Like it took a few hours to like outperform the two years of work, which we'd done as with human in the loop.

31:07Um, so in the, in the game of chess, it took four hours to beat the world champion, the best ever AI that had ever been created in the game of go. It took, um, you know, like, uh, I think it was a couple of days or something, but it was incredible progress. Like, and, and, and so you have to let go of the human data to allow yourself to discover these things that can then shoot past the competition and discover the things that just keep learning and learning and learning forever. So, you know, it has to be an active choice. And if you don't make that active choice, you know, what I see in a lot of the kind of LLM companies today is they're basically, you know, building semi-autonomous human in the loop kind of learning algorithms with thousands of people in the loop of the learning progress, which is, you know, it works and a lot of respect for how far it gets.

31:52But, And until we step back from that and really discover the process that's going to learn for itself, I don't believe we'll get all the way to something which is truly a super learner and unleashes the potential that we've talked about so far. That's very profound. As far as I know it, there are humans that are ineffable and some of you are creating environments or crucibles. And what does it look like to actually try and make sure that human priors aren't kind of leaking into what you're doing? yeah well so when we design like environments um and like rewards functions and so on like obviously there's a human bias in there but we don't want to as Dave was also mentioning earlier we don't want to like bias it towards uh building a machine based on like the von Neumann architecture something like this right we want to like but if you're stating like um some problem where like a machine would be useful it would be very interesting to see what type of machine it comes up with like what computer like machine it comes up with so i totally agree with this i would also say that we're not religious about this it's not like a religion that you can't have any human prior anywhere um it's more like a pragmatic position that you know until we give up on on like relying on the trillion like item data sets of human knowledge that we're not going to make the progress that we that the field deserves to see so it's more like a actually a practical position.

33:17And so in practice, of course, there's going to be a certain amount of priors, which, like, as Lessa says, there'll be priors which apply to many different, like, worlds, as opposed to, like, you know, trying to put in these very strong priors that exactly encode how all the knowledge we know about this one particular problem or this one particular world. So kind of weak environment agnostic priors seem perfectly reasonable, but kind of these super strong priors with, like, trillion item data sets, we're trying to avoid that because we think pragmatically it's going to lead to better outcomes.

33:48Yeah, absolutely. You mentioned that you like, philosophically, that you're drawn to research that's more on a knife's edge, 50-50. I'm curious what your confidence levels are for ineffable at the broadest level. Will a super learner emerge? And then, I guess, at the more pragmatic level in terms of the things that you're trying to accomplish right now, are you 50-50 on these as well? So first of all, I would say I strongly believe that the approach will be feasible at some point. In fact, it's the only thing for which we have an example in our brains. It's the only existence proof of something which has discovered an incredible intelligence that's very general and works in an embodied world and discovers how to do all kinds of different things.

34:36So we have this incredible example of the human brain that we can look at. So then, you know, do I think we can achieve it? Well, I think like many great like moments in science, it's about timing. So, you know, whether this succeeds or fails, I think is about timing. Like when you create a startup, you want to be at the right time. You don't want to be too early or too late. You know, if you're too late, everyone's done it. You know, it's not interesting, like a thousand other people have already done it. And if you're too early, you know, You might be banging on the door, but not have the right kind of, you know, maybe the field isn't advanced enough or there isn't enough compute to be able to succeed yet.

35:17So to me, it's all about timing. For example, with Computer Go, you know, I made a prediction, I think, 10 years before Computer Go was solved and I got it right to the year. got it right for StarCraft, which I also, we had a bet and made a prediction. Same for alpha proof when we were predicting when the IMO was going to fall. So you're a super forecaster of sorts. So, so far, so far, I've done very well at this. And I do have a very strong sense that this is the right time for this to fall, that super learners are possible. Like there's been enormous progress in the level of technology, the ability to accelerate research using kind of LLM coding methods or auto research methods like we've produced in the past.

36:05The ability and understanding of scaling laws and the sophistication of technology in the stack. There's a lot of things that fall into place that I think make this the right time. So that's my bet that this is the right timing. But yeah, of course, this is research. Research is never guaranteed. And And that's what makes it exciting. If you get the timing right, you can be the first. You can be the ones to change the whole field of AI. And, you know, that's what it means to take risks and do something exciting. For an affable and also all the previous things that you correctly forecasted, is the biggest driver of timing compute?

36:42I think it's a very big driver. I think it's an enormous driver. I think, yeah, I mean, we're in a great position with the compute that we've managed to get hold of. And I think it gives an opportunity to do things which are, you know, orders of magnitude more sophisticated than approaches that have tried to kind of do self-discovered RL in the past. So I think, yeah, if we compare to, I don't know, something like hide and seek from OpenAI, which was like an early example where agents discovered kind of some basic tool use, I think, you know, we could probably run experiments that like use 100 ,000 times more compute.

37:24And so it's kind of amazing that people haven't come back and really pushed on these types of approaches given the availability of compute. And of course, hide and seek, I don't want to pretend that that's the end goal. I think it's very basic. but i think it just it's suggestive of of what could be done like if we really you know so i think compute is a huge driver but it's not the only one yeah i think we're in a very fortunate situation that we have been able to secure like a very large cluster right so a very beautiful cluster at that yes so um um so this is like really like an opportunity for us right like we really have like the cars on our hands like to test this out um very few other people are doing something similar.

38:06So yeah, we can really test this out in a way that we couldn't do at DeepMind, for instance. Absolutely. It seems like one of the maybe determinants of how far this can go is how far you can push simulation. And these ultimately these crucibles are, you know, they're digital, right? And I think simulation has probably come quite far in the last handful of years, but it's not perfect, right? We recently had your friend, Rich Sutton, on the show and I asked him about simulation and he said he was very dismissive of simulation. I'm curious how you guys think about this and like I say, you've been on both sides of this.

38:49I think you obviously did a lot in the digital world but you also built MetNet which touches real world atmospheric data. How do you all think about where is the ceiling on simulation and what is when is the right time to bring an applicable into the real world so i guess mednet was actually like an example so there's like a weather forecasting model we did in the google brain amsterdam office together with google research where like we were doing like a machine learning model to predict short term like weather forecasts and there's like way better than the physics-based simulations right like so the physics simulations were like accumulating errors over time.

39:32So there's like a point to that, that physics simulation is not perfect. I do think there's a lot of relatively simple environments that we can build where we can see this like emerging behavior of let's say like technology and so on, where we don't need like exact physics. I actually do think it's interesting to see how much we can build these days. We've got like a very small team in terms of these simulations and so on. So there's definitely like a ceiling. Is there a ceiling? Actually, I'm curious about this. Yeah. Well, I guess quantum physics and whatnot. Right. But but so there's like a ceiling somewhere.

40:10But but we can do like very interesting things without real physics, I would say. So I think the right question to ask is, can a super intelligence be built in simulation? So if it is possible for it to be built in simulation, I believe it will happen 10 times or 100 times faster just because simulation can be infinitely paralyzed. You get perfect reproducibility, has a lot of really nice properties. It's also the safest place for a superintelligence to be built. So it has these really nice properties. Then we need to ask, is it possible? And I think for the reasons we discussed earlier, there are very clear approaches that we can add into the simulation that allow it to be sufficiently complex to represent arbitrarily interesting things.

41:03So for example, let's say you wanted to build a computer in simulation. Well, all you require is that your simulation is Turing-complete. If you have a Turing-complete simulation, suddenly everything that we mean by computation, like provably everything we mean by computation can be expressed completely inside simulation. And indeed, humans have built incredibly complex computers or related things inside simulated worlds. The same is true for various other ideas, like if you want to build an economy, money can be built out of coins or out of paper or out of bits. And as long as you have one of those, some representation of money, everything that we mean by economics can be expressed inside a simulation.

41:48The same is true for language. If you've got multiple agents and they've got some ability to express a channel of communication between the agents, suddenly everything that we mean by language is expressible in simulation. Mathematics we know is expressible in any universe. Science in some sense is the study of our world. So will a simulation be sufficient to study what we mean by science? Well, there are some aspects of science which are like expressible in many things like, you know, energy or entropy or evolution. But also, it's true that, you know, the agent will discover its own science, like whatever the simulator is, if it's got this emergent complexity to its environment, the agent will discover its own science about whatever that world is.

42:32It's very, like, provably possible to build environments in which it's possible to express and represent enormous complexity of all of these things. Well, I agree with LASO, there might be some things which aren't well expressed. You know, there will be things which can only be expressed in the real world, like quantum computation is a good example. The real point is that if it's possible to get all the way to a superintelligence in simulation, it will happen faster and it will happen safer. And so for those reasons, it's something that we choose to put a lot of energy into. you. I actually agree with Rich Sutton that I think his point was something along the lines that you shouldn't expect that the things that you learn in simulation to transfer sim to real into the real world, because the real world is just vastly more complex and in very interesting ways that are hard to simulate.

43:21But I think what we can do is we can do all of the science to discover the super learner. And once we have that super learner, we will have the answers and we'll know how to build something which that same algorithm should be applicable to the real world. And we can also make our simulations more and more high fidelity towards the real world and check that they still work before putting them into the world. Great answer. Thank you. I'd love to talk about alignment and maybe less. I know there's a politically charged topic right now, and so I'm less interested. I'm zero interested in having the politics or policy discussion around alignment, but I am very interested in the scientific discussion around it.

44:00It seems like so much of the misalignment that's emerged has arisen from, you know, scaling reinforcement learning to the extent that it's been scaled. Do you think that the super learner you build will naturally be aligned to human interests and have human values? Is it important for it to have? alignment with human values? So this is a big topic. So let me start by saying a few things. So where we are today with alignment, we are essentially relying very heavily on alignment approaches that are based on the language that the agents use and priors that are based on on human data. That has been reasonably been very effective up to this point, but we're starting to see that that approach is reaching its limits.

44:55And by itself, I think it's going to be difficult to see a path where as agents become inevitably more and more powerful, we can already see this happening as agents are trained more and more, they are kind of going to start to overwhelm their priors, they're going to start to learn to say one thing and act in a different way. And it's going to be increasingly difficult to align to the things that humans want them to do. So my belief is that we need something more and that this by itself is not sufficient. The most important single thing that we can do as a company is to actually understand the science of superintelligence.

45:37intelligence. So this means to say that if we actually can, in simulation, really build a superintelligent agent and start to see these super learners go all the way, we can start to actually understand what it takes for a superintelligence to actually interact peacefully. So we can answer really important questions. These are fundamentally important questions, like under what conditions, like what are the conditions, what are the structures, What are the kind of leverages that we need to put into place to make sure that like a society of super intelligences actually interacts peacefully and in particular interacts peacefully with other lesser intelligences that are also going to be in their world.

46:19And I think if we get this right, we actually can get to some answers, like to a real science that will help to understand what the right things to do are in the real world. Because we're hurtling towards that real world outcome where there are going to be a whole load of different interacting super intelligences. And we don't quite know how that's going to play out. And I think the faster that we can actually get to these answers in simulation and publicly share that, the better situation we'll be in. So that's my strong belief. I also believe that we can use this to kind of gate the actual decision making of like what we do in the real world.

46:57So, you know, one of our commitments is that we will gate anything we do in the real world on having sufficient evidence from this kind of science of superintelligence that we are really confident that things are going to go well and we'll only push forward to the next level of what we do in the real world once we are absolutely confident that that can lead to a good outcome. So I think these are two things that I believe we can do. We're in this world where a lot of things are happening very quickly. We're hurtling towards an outcome with lots of interacting super intelligences. I personally believe that this will lead to a good outcome for humanity, but it's happening very quickly.

47:34And I think that if we are able to actually understand this and run ahead in simulation, we'll be in a much better place to actually do the things that ensure that this goes really well. Fascinating. And it seems like one of the primary things that current safety researchers rely on is being able to monitor the agent's chains of thought in English. And so when they start speaking in a release, that scares people because we can no longer understand the monitor as well. In your case, you have your agents kind of going straight to rediscovering language. What do you think that means for monitoring?

48:10or is the point that if you get the reward right and if you get the environments right, then you shouldn't need to monitor? Yeah, I think monitoring is something which is still really helpful. I think that, like it or not, I believe that the approach of monitoring the human language that these systems use is going to be increasingly difficult as agents get more powerful. and we're already seeing that, that the monitoring agents are, maybe they're producing like a tome of a million words of reasoning, or maybe they're actually actively using subterfuge to kind of think in one way and then act in another.

48:55It's increasingly difficult to rely 100 % on that. So I do think we need other elements in our armory anyway. So I think it's actually really important that some people are looking at how to get this right. I think there are ways to monitor agents that don't use language. I think we can do things like start to, for example, you can use shadow actions where you start to have actions taken where you can look at what they would actually do if they were allowed to act in the world, but you don't actually let them take that action yet. You can start to have other super intelligences which are acting as the monitor.

49:34So, for example, you might have the idea of like an incredible scientist agent that sort of treads lightly upon the earth. And then you have another super intelligent that's kind of monitoring the kind of magnitude of its actions and regularizing it to make sure it can't do things which are kind of affecting the world too much. I think there are lots of ideas. I would say that there's a lot of research to be done here. I'm not claiming to have all of the answers, but I think it's something that deserves a lot of investigation. and if people aren't looking at this because they're only looking at the language-based approach, I think we're in trouble because I think that language-based approach is not enough in itself to actually get to where we want.

50:14I should say in terms of having said in trouble, I am optimistic about where the world is going. So by trouble, I mean that we're not going to have agents that do the things that we want them to do. I do think we're going to end up in a world with lots of complex, interacting, super intelligent beings And I'm optimistic that that will lead to a kind of a balance of of of actors, some of which are supporting the flourishing of humanity and some of which may not be. But the ones which do support the flourishing of humanity will will end up kind of supporting us and protecting us against the ones which aren't acting on our best interests.

50:54So so that's the world I think where we can work towards. Yeah. Well, given your super forecasting abilities, I'm happy to hear that. What do you think about all the conversation around pacing the frontier? I'm actually very much supportive of the idea that there should be pacing of what happens for the frontier of AI in the real world. At the same time, I think we should move as fast as possible in simulation. Now the reason is that I think in simulation we can actually develop a lot of things like the science of super intelligence that will actually help us to find solutions that help things to go well in the real world.

51:36And I think that kind of work should happen as fast as possible to make sure that when things really happen in the real world, that things have the best possible opportunity to go well and lead to human flourishing. So I think that would be my thought. But yeah, in general, I'm supportive of the suggestions, like the kind of direction that that people like Demis and Daria and so forth are suggesting. I think it's generally a good idea. Maybe part of that is that I think that things are moving really quickly. And I do believe that when we accelerate AI and get to superintelligence, it will lead to a really great future for humanity.

52:21But I also think that it's something that the speed at which things are moving is not something that we are capable of adapting to that fast. So I think just by pacing a little bit, I think it gives us the opportunity, A, for our governments and our organizations and our emotional responses to adapt to the changes that are coming. And also for this work to continue apace, like really figuring out in simulation how to, you know, the real science of doing this right. So accelerate in simulation so we can understand what's happening and do science around it and then careful in the real world. Yeah, I think that would be my summary.

53:04I'd love to transition to talking a little bit about compute. This is going to be an incredibly compute expensive journey and you are pushing the limits of what compute can do. I think both Google and NVIDIA have been great partners for you. Can you talk a little bit about your cluster? You're not simply renting spot compute from NeoClouds like many other NeoLabs. You're actually doing some really serious work on your cluster and the different other forms of compute that you have. Can you just talk about your compute setup? Right. So, yeah, as I mentioned earlier, we are very fortunate with the compute that we have.

53:43We are the first customers of the largest real-grouping cluster on GCP by the end of the year. Well, actually started like one month now. So we are like extremely fortunate in that. And I actually feel quite bad about like the startups that are like getting compute now because it's becoming like harder and harder. But yeah, so we have like this like very group and cluster, which is like based on GPUs, right? Like NVIDIA GPUs, which is really fortunate for us because it means that we can easily do simulation and like training and inference, right? They are quite like general chips. So that's like amazing.

54:20I think the Rubens specifically have this architecture where one rack can basically conceal one machine. There's a very high bandwidth between the nodes and one rack, which is very important for reinforcement learning where you have really fast iteration cycles compared to LLMs that are relatively speaking quite slow. So that's quite fortunate for us. I think some of the things we did eight, 10 years ago at Google, where we basically had abundant CPUs at Google that doesn't scale anymore. So we really need to make sure that everything we do is on the GPUs and we don't need millions of CPUs beyond these GPUs.

55:02So our simulations can run on the GPU inference and training and so on. So we have some good architectures for that. I think the LM companies now have the problems that we had many years ago, where the need like sandboxes running on CPUs, assimilating computers and so on. And we tried to not put ourselves in that situation. All the compute, is it roughly fungible between the different things that you're doing? I imagine you're building crucibles and then you're doing these big rollouts where the agents are actually learning. Is that the rough buckets of what compute is being used for? Yeah, you could split it up into training, inference and simulation.

55:44Is the same compute equally good at training, inference and simulation? No. Okay. So I think it's pretty clear what has happened with GPUs over the last many years, or like since deep learning took off, is that there have been more and more tailors towards towards training, right? And maybe in the last few years, also inference, but less about simulation, right? So, but we are like similarly to the LLM companies. We also want to be in the case where like scaling laws matters. Like we know that roughly like if we add this amount of compute and experiences, then we get a certain amount of quality out of that.

56:26So I guess if GPUs aren't great for scaling simulation, then what's the solution? So there are many different solutions here, right? So there's like model-based reinforcement learning where like the model becomes much more expensive compared to the simulation. So there's many different tricks you can do. There's like various approaches for like replay buffers or like reusing data many times. So yeah, there are ways around that. And all of that, we do spend quite some effort on just making the simulations as fast as possible and make the best use of the GPUs. Are you seeing scaling laws yet or is it too early to say?

57:05So for reinforcement learning, scaling laws are slightly different. So scaling laws is the component of how much compute you put in, how much data you put in, and the model size. Reinforcement learning is a little bit different because the problem is non-stationary. So as you progress in the problem, the data changes, the data distribution changes. So they are a little bit different. It's like a very interesting research direction. I understand the difference, but are you seeing some sort of scaling law where more compute in gets you closer to achieving the goals? Or is it hard to say right now?

57:42Yeah, but it's something we are actively pushing towards. And especially as we are transitioning from a smaller cluster to the bigger cluster, we want to be ready for that. Yeah. Lese, at the beginning, when David introduced you, he mentioned that you had done some very foundational work to compute algorithms for reinforcement learning, upon which the field really relies on today. Maybe talk about some of that work, and in particular, your work around Impala and Seed. What are those, and how do they relate to the work that you're doing at Ineffable? Right. So when I joined the deep learning team at DeepMind, most people were running single machine, CPU-based reinforcement learning.

58:24So the scale was quite small. So what I did together with colleagues was to make an agent, reinforcement learning agent, that really utilized GPUs. I could scale linear with compute. I think partially because it was very efficient. It was very good at scaling. and like the third thing was it was very easy to extend and like modify and do whatever you wanted with it and contributed to it being like the most popular reinforcement agent at the company and at i think at one point um most our team used it or like derivatives of it um so that was quite successful and quite a nice project to work on then like later on like we we got tv use right we got like racks of these two views so we need to scale beyond that so some of my work at google brain after i left deep mind to found the google brain amsterdam office was to scale to like racks of tvs and that probably was called like seed rl and so there was like really about like utilizing the tvs both for like inference and training in the best possible way while keeping um the environments as flexible as you wanted to.

59:37It could be like anything from go to very expensive physics simulations. And how's that being used at Ineffable? Yeah, so well we actually implemented a few of those algorithms, also like derivatives of these algorithms. So that is like what we are using today. We have like depending on like the environments we use like different algorithms, right? There's like pros and cons between them. So yeah, so we talked about like we have some very simple and fast to simulate environments and also some that are more expensive. So for those who are working on large scale RL architectures today and haven't read those original papers, it's definitely worth a look because I think you'll discover that, you know, many of the ideas that are currently in use, like kind of trace back to those early work that that Lasse kind of did and figured out, which was a little bit prescient, I would say.

1:00:34I think it's actually kind of fun to see the reinforcement learning space now versus like 10 years ago, or like, let's say like eight years ago, maybe. Like the LLM community is also using reinforcement learning in particular last two years. And like a lot of the things that they're using are being rediscovered approaches that we used many years ago. So it's kind of interesting to see with a lot of novelty as well on top of it. Can you talk a little bit about your team? How big is the team right now? We have intentionally kept the team relatively small. So we're not in a hurry to build a giant organization.

1:01:16I think you can do an awful lot nowadays with a small number of incredibly talented people. So we have hired 20 people in total. And yeah, the people we've hired, it's about finding some of the exceptionally talented people. But it's not just about talent for us. It's also about bringing together people who are kind, collaborative human beings. That's really important to the kind of culture we're trying to build. And people who are really well aligned with the kind of mission that we're following. and people that sort of fit the needs that we have. So we try and build the team with the right set of skills as well.

1:01:59So all of those four things are really important. And yeah, we kind of have very high standards and very high expectations of the people that come in. So we're very lucky to get some amazing, really amazing people. And so just to tie it to the compute as well, right? Like this relatively small amount of people so far means that we have a lot of compute per person. Right. And like these days you can you can do a lot of experiments just like one person. So it's really also to keep the compute to person ratio quite high. So we're not like fighting for resources from day one. I think three of the co-founders worked together on AlphaStar.

1:02:41Many of you, many of you did work together. Are you assembling the Avengers from, especially given the recent changes from DeepMind, are you assembling the Avengers of folks that you've worked with in the past? Have you hired new people? And if so, kind of what are the attributes that make somebody new to you likely to be a fit and an appable? So it's a mixture. So, I mean, AlphaStar was an exceptional team. It was sort of intentionally assembled within DeepMind by choosing the kind of best of the best to work on that project. So in some sense, it's not a surprise that I think actually four of our co-founders come from that team.

1:03:17But we are not exclusively drawing from that team or indeed from DeepMind. I mean, the other three founders all came from very successful startups. and we hire people who are brilliant, but also kind and nice people to have around who are going to work together well, who are aligned with the mission and who fit the things that we need. But in terms of finding those people, we put a lot of effort into getting the right people. So as uncomfortable as it is, I've had to say no to a lot of people I've worked with in the past just because our bar is so super high yeah i've worked i've been fortunate to work with like some amazing people and so you know i've been choosing amongst uh like a a very exceptional crop and bringing in yeah what i consider to be a yeah superstar team um and so yeah it's it's um it's it's not easy to get into ineffable but uh the people who we have are wonderful human beings and yeah, very, very proud of the people we brought in.

1:04:29I can see why people are drawn to working at Enoughable, especially at a time when, you know, the biggest clusters, the biggest labs are kind of, you know, they're locked in this prisoner's dilemma race and, you know, there's a lot of dollars at stake and so, you know, decisions get made differently. I can imagine for a set of people that, especially that really, really deeply believes in the scientific mission of what you want to accomplish. This must feel like a breath of fresh air. Yeah, I think that's right. And we've tried to establish a research-led culture. So it's not a company that's led by building an incremental next product v17.1 on the thing that's come before.

1:05:15It's a company that's built on researchers pursuing their passion and trying to produce the breakthroughs that they believe are possible. and we center the company around that from the beginning and from the outset to create like a haven for research you know where where we push back on on on product pressure because we believe that if we make these breakthroughs it will lead to you know far bigger products downstream and far bigger successes than if we kind of start off from day one on like you know oh you know here's here's the product we're doing from the beginning so yeah so it's research led we're creating um uh i think a really special environment.

1:05:52And yeah, people are loving it. I'm loving it. Wonderful. Wonderful. I'd love to close with some predictions, maybe especially since you're such a super forecaster. What's your guess for when you'll have the first strong evidence that the approach is working? And then what's your forecast for when we'll achieve the mission that you've set out to achieve this first contact with super intelligence? Yeah, so these are fair questions. So there are going to be some reasonable bars of uncertainty on this. You're being recorded, so we're going to hold you to it. Exactly. Yeah. So I think we will see some extremely strong evidence that we're on the right track within a year.

1:06:40And I think we will achieve something which is a true super intelligence, like according to the definition I gave earlier of something which you know discovers things which are far beyond what humans can achieve in across you know everything from science mathematics technology languages etc etc I originally thought within three to five years um my timelines have accelerated a little bit so I'm going to say three years because of um yeah what can be done with um with like um coding agents to accelerate things um I think um yeah I think things are going to move fast Buckle up. It's going to be a journey.

1:07:16Wonderful. Let's say, David, this is a wonderful, wonderful conversation. I'm so inspired by what you're building. And thank you for taking the time to sit down and share with our audience what you're building today. Thank you, Sonia. It's been a pleasure.

1:07:43Thank you.

From the publisher

David Silver spent two years building AlphaGo on human data. When he removed human data altogether building AlphaGo Zero, the AI surpassed AlphaGo in under two days. He left DeepMind to run that same bet in pursuit of superintelligence. David and Lasse Espeholt, co-founders of Ineffable Intelligence, explain their plan to build a “superlearner”—an agent that starts knowing nothing and discovers science, mathematics, and language for itself by direct experience instead of through a pre-training run. Silver argues that human-language reasoning is a ceiling, not a foundation, that alignment via chain-of-thought monitoring is already reaching its limits, and that the science of how superintelligences coexist should be worked out in simulation before anything touches the real world. Espeholt describes the GPU cluster and the lineage from IMPALA and SEED RL that makes the experiment possible. Silver’s forecast: strong evidence within a year, superintelligence in three.

Hosted by Sonya Huang, Sequoia Capital

0:00 – Introduction

1:09 – What held true from Go to today, and what the LLM era changed

3:35 – Why leave DeepMind to start a new lab

5:58 – Why "Ineffable" (and the secret second reason)

7:03 – Lasse's path: scaling RL at DeepMind and Google Brain

9:31 – Defining a super learner and first contact

12:35 – Super learners vs. LLMs that make discoveries

13:59 – Day to day: the RL stack, crucibles and human-journey milestones

16:13 – Don't breadcrumb the objective: simple rewards, emergent complexity

21:16 – What a crucible looks like

24:14 – Go: one agent is boring, two is infinite

25:41 – Will agents rediscover language?

27:22 – Removing human priors: AlphaGo to AlphaZero

32:05 – Keeping human bias out of the crucibles

33:35 – Confidence, timing and super forecasting

36:24 – Is compute the driver?

38:13 – The ceiling on simulation and Rich Sutton's objection

43:44 – Alignment and the science of superintelligence

47:45 – Monitoring agents that don't think in English

51:01 – Pacing the frontier

52:54 – The compute setup: Vera Rubin on GCP

55:18 – Training, inference, simulation and RL scaling laws

57:40 – IMPALA and SEED RL

1:00:58 – 20 people and the compute-per-person ratio

1:04:49 – A research-led culture

1:05:47 – Predictions

More from Training Data

All episodes