DeepMind Gemini 3 Lead: What Comes After "Infinite Data"

27:09So from that perspective, if you squint enough, you will recognize a lot of the original transformer paper pieces in that. Can you describe for people to make this educational what an MOE architecture is? At a high level, the transformer kind of has two blocks. So there's an attention block, which is responsible for mixing the information across times, across different tokens. And then there's the feedforward block, which is more about giving the memory, but also the compute power for the model to make these inferences. And those operate on a single token at the time. So they operate in parallel.

27:48So in the original transformer architecture, this is just a single hidden layer neural network. So it's a dense computation where the input gets linearly transformed into a hidden dimension. You apply some activation function and that one gets linearly transformed again into the output of the dense block. So that's the original paper. And then there's a lot of work before transformers as well on mixed-gen experts. And here the idea is you kind of decouple the amount of compute you use with how large the parameter is to use that. And so you dynamically route effectively to which expert you want the computational power to be used on rather than having that coupled.

28:30Gemini is natively multimodal. In practical terms, what does that actually mean for the model to think about text, images, or videos? Yeah, what this means is that there's no specific model trained to handle images and a different model trained to handle audio, a different model trained to handle text. It's the same model, the same neural network that processes all these different modalities together. Presumably, there is a cost aspect to this. Does being natively multimodal mean you're more expensive from a token perspective? Yeah, this is a really good question. There's kind of two costs to this.

29:17I would say that the benefits largely outweigh the cost here. And this is why we train these models. But the first cost is maybe less obvious to people, but it's this complexity cost and this research that I was talking about. Because you're doing a lot of more things, and especially different modalities interact in some ways, this can interact with different parts of the research and it has a complexity cost. So we have to spend time thinking about these things. The second cost is, yes, images are often larger in terms of input size than pure text. And so the actual computational cost is, if you do it naively, is higher.

29:58But of course, then there's interesting research to be done on how you make these things efficient. All right, let's talk about pre-training, since it's the area that you cover in particular. So starting with the high-level question, we mentioned, of course, the term scaling laws towards the beginning of this conversation. We talked about chinchilla a few minutes ago as well. In 2025, there was this much-discussed theme of the death of scaling laws, particularly for pre-training. Is Gemini 3 the answer that shows that all of this is not true and that indeed the scaling laws are continuing? Yeah, the discussions there to me were always slightly strange because my experience didn't match those.

30:50I think what we've seen is scale is a very important aspect in pre-training specifically and how we make models better. I think what's been the case, though, is that people overvalued that aspect. So it is a very important aspect, but it's not the only aspect. So scale will help to make your model better. And what's nice about scale, it does so fairly predictably. And that's kind of what the scaling laws tell us. As you scale the model, how much better will the model actually be? But this is only one part. The other parts are architecture and data innovation. These also play a really, really important part in the performance of pre-training and probably even more so than pure scale these days.

31:33But scaling is still an important factor as well. Right. And we're talking about pre-training specifically, right? Because this year we seem to have scaled RL in post-training and scaled test time compute, all the things. But for pre-training, you're seeing not only scaling loss, It's not slowing down, but you see some acceleration. Do I understand this correctly due to data and different architectures? I think the way to put this is these all compound. So scale is one axis, but this model and data also will make the actual performance better. And yes, sometimes the innovation part outweighs the benefits of scaling more.

32:18And sometimes just raw scaling is the right answer to make the model better. So that's on the pre-training side. And yes, on the RL and RL scaling side, I think we're seeing a lot of the same things we're seeing in pre-training or we saw in pre-training. What's interesting here is because we have the experience of pre-training, a lot of the lessons apply and we can reapply some of that knowledge to RL scaling as well. Speaking of data, so what is the pre-training data mix on Gemini 3? I think you guys had a model card out for a bit that talked about some of this. So what went into it? Yeah, it's a mix of different things.

32:58So the data is multimodal from the ground up. And yeah, there's many different sources that go into this. Another classic question in this whole discussion is, are we about to run out of data? So there's always the, do we have not enough compute? And the other question is, do we not have enough data? Clearly, there's been a rise in the usage of synthetic data this year. In your day-to-day work, or perhaps in general, where do you think synthetic data helps and where does it not help? Yeah, so synthetic data is interesting. You have to be very careful in how you use it because it's quite easy to use it in the wrong way.

33:44And what's often the case as well with synthetic data is you use a strong model to generate the synthetic data, and then you run smaller scale ablations to validate the effect of the synthetic data. But one of the really interesting questions is, can you actually generate synthetic data to make a model that you want to train in the future, which will actually be better than the model that generated the synthetic data in the first place? Can you actually make that one better as well? And so we spend a lot of time thinking about this and doing research in this direction. The other part of your question, are we running out of data?

34:19I don't think so. There's more. We're definitely working on that as well. But more than that, I think what might be happening instead is kind of a shift in paradigm where before we were kind of scaling in the data unlimited regime where data would scale as much as we would like. And we're kind of shifting more to a data limited regime, which actually changes a lot of the research and how we think about problems. But one good analogy of this is before LLMs, a lot of people were working on ImageNet and other benchmarks. And there was very in a very, very data limited regime as well. So a lot of techniques from that time start to become interesting as well.

35:00And perhaps that's one of those. And I don't know to which extent you can talk about it, if not talk about it in general. But there is this concept throughout the industry of training models based on reasoning traces. So basically forcing the model to show its work, how it got to a certain outcome, and then taking that to train the next model. Is that something that you do or that you think is interesting or a future direction? What is your perspective? Yeah, unfortunately, I can't comment on the specifics. This is how I'm asking the right questions. But maybe in general, is that something that people in industry do?

35:38I believe so. And this also falls into the previous question around synthetic data you were asking and kind of our approach to that is similar. And perhaps with that, taking this into a futuristic conversation, but like another big question and theme seems to be indeed, how can models learn from less data, which I think is what you were alluding to, talking about a data limited regime. Again, whether at DeepMind or in general, are you seeing interesting approaches to use the famous analogy, a model can learn like a child does? Just to maybe clarify what I said earlier, in a data-limited regime, I didn't necessarily mean with less data, but rather with a finite amount of data.

36:24So the paradigm shift is more from like we have infinite data to we have a finite amount of data. The second point is, in some sense, model architecture research is exactly what you mentioned. So when you make an improvement on the model architecture side, what it typically means is you get a better result if you use the same amount of data to train the model. but equivalently, you could get the same result as the previous model by training on less data. So that's kind of the first aspect of that. But it is true in terms of the volume of data needed today. We're still orders of magnitude higher than what the human has available to.

36:59Of course, there's the whole evolution process as well, which I find these high-level discussions quite hard to understand or follow because you have to make so many assumptions to convert that amount of data into what is today's pre-training data. but at least at first order, it does seem like we're using a lot more data than humans do. What other directions in overall pre-training progress are you excited about throughout the industry? Yeah, I think the one thing is in Gemini 1.5, I think we had a really good leap in the long context capabilities of the model. And I think that's really enabling the ability of models and agents today to do this work where you have maybe a code base and you do a lot of work on it, so your context length really grows.

37:45I think there's going to be a lot more innovation on that side in the next year or so to make long context more efficient, but also just to extend the context length of models themselves. So that's on the capabilities front, I think it's something where pre-training specifically has a lot to offer and is very interesting. Relatedly, I think for us, at least on the attention side, we've made some really interesting discoveries recently that I think will shape a lot of the research we do in the next few months. And I'm personally very excited about that. Again, I think I want to emphasize the point I made towards the beginning, but the way things work is it's really a culmination of many different things.

38:25So there's a lot of small, medium-sized things that we can already see coming up where I think we fixed this issue, we fixed this bug. This is an interesting research that shows promising things. And all of these things coupled, I think, will drive a lot of the progress again. It's interesting, you know, thinking about Retro that we talked about a bit earlier. You know, you're the co-author of Retro, which was about efficiency and like smaller models doing more. And now you are in the world of Gemini 3, which is like massive amounts of data and training in a very long context windows. Do you think that this paradigm of having, again, larger models, large context windows effectively obviates the need for kind of drag and search and that everything gets folded into the model?

39:16I mean, obviously, there's a corporate data part, but in general? There's some interesting questions here. So first of all, I think Retro was really about retrieving information rather than storing it, not necessarily about making models smaller. So it's about how we can use the model to do more reasoning already in a pre-training sense of reasoning rather than just store the knowledge. So this is still very much the aspect today. The interesting part is the iteration cycle maybe of pre-training used to be a lot slower than that of post-training until fairly recently. And so making these large changes on the pre-training side is quite costly in terms of risk and how long it takes.

40:00And then you have approaches like RAG or SEARCH, which you can do during post-training and iterate much more quickly on, which gives very strong performance as well. I think deep down, I do believe that the long-term answer is to learn this differentiable end-to-end way, which means probably doing pre-training or whatever that looks like in the future, learn to retrieve as part of the training and learn how to do search as part of the large part of training. And I think that's kind of RL scaling maybe starts that process, but I think there's a lot more to do also on the architecture side. But this is something that we'll see in the next few years and not immediately, I would say.

40:37The one thing I want to highlight is people often talk about model architecture, and that's definitely one part of what makes pre-training better. But there's other parts as well, infra and data and evals specifically, that don't always get the same mention. Evals specifically is extremely hard, and it's even harder in pre-training, I would say, because it kind of has these two gaps you need to close. close. So on the one side, the evals we use or the models we train regularly are much smaller and less powerful than when we scale up. So that means the eval has to be predictive of what the performance or have to still work for the large model and point in the right direction.

41:17So it has to be a good proxy on that side. And then there's a second gap as well, which is when we evaluate pre-training models, there's a post-training gap as well. So the way the models get used is they don't just get used after pre-training, there's more training happening after. And so the evals we use in pre-training or on pre-trained models have to be good proxies of what happens after as well. And so making progress on evals is really important and quite hard and has also driven a lot of the progress we have in terms of being able to measure what an actual improvement is on the model or on the data side.

41:49And evals are the mind that's all internally built, like you have your own set of evals? Yes, to a large extent and more and more so because what we found is that external benchmarks, then you can use them for a little while, but very quickly they become contaminated. So they start to be replicated on different forms or different parts of the web. And then if we end up training on those, it's really hard basically to detect leaked evals. So the only way you really have to protect against cheating yourself and thinking you're doing better than you are is by actually creating held out ELLs and not really keeping them held out.

42:27In the same vein, is alignment a part of what you all think a lot about at the pre-training level? Or is that more of a post-training kind of conversation or both? It's a majority of post-training, I would say, but there's definitely some parts of it which are relevant of pre-training. I can't go into too many details here, but some parts are relevant to pre-training and we do think about that as well. And at a very simplistic level, I always wonder, again, in the context of Gemini or otherwise, if the core data set is the internet, there's a lot of terrible things on the internet. Is alignment 101 that there's stuff that you just do not include in the model?

43:06This is an interesting question and I don't think I have a definitive answer, but you don't want the model to do these terrible things. So at a fundamental level, you do need the model to know about those things. So you have to train a bit at least on those so that they know what their things are and know to stay away from those. Otherwise, when a user would mention something terrible, the model wouldn't even know what it's talking about. And they might not be able to say this is something terrible. Let's talk about DeepThink, the thinking model that was released a few days after Gemini 3. So first of all, is that a different model or is that part of the same model?

43:43How should one think about it? I'm not allowed to. I can't comment too much. What happens when the model thinks and you wait for 10 seconds or 20 seconds or whatever time? What happens behind the scenes? I think this has been covered quite a bit in some of your previous podcasts as well. It's about generating thoughts. And so rather than just doing compute in the depth or in the model side, you also do compute and allow the model to think more on the sequence length side of things. So the model actually starts to form hypotheses, test hypotheses, invoke some tools to validate the hypotheses, do search calls, etc.

44:28And then at the end, be able to view the thought process to provide a definite answer to the user. The industry has normalized around that paradigm of chain of thought. That's for, yeah. Can you talk a little bit about the agentic part of this and Google Antigravity? What do you find interesting about it? What should people know about it? Yeah, this is, I guess, what I was mentioning before around my own work, especially. I think that's interesting. A lot of the work we do on a day-to-day basis is more execution-based, babysitting experiments, etc. And I think this is where I at least see the most impact from those.

45:04Bringing it back to the topics of pre-training, I think that the perception and vision side is very important for this because now you're asking models to interact with computer screens. So being able to do screen understanding really, really well is critical. And so that's an important part on the pre-training side, at least. And in anti-gravity, there's a whole vibe coding aspect, truly vibes in that you don't even really see what happens when you ask. Is Vibes, same question, is that a pre-training thing? Is that just a post-training thing? How do you build Vibes into a model? Yeah, this is interesting.

45:42I think you can probably ask five different researchers and you'll get five different answers. There's also this notion of large model feel. People call this, especially I think GPT 4.5, historically had some of this presumably where larger models maybe feel differently. I wouldn't actually just, I wouldn't put it in these terms specifically, but I think Vibes comes down to this and actually pre-training probably plays a larger role today in some of that and how the model feels and in general than post-training. I think this is, yeah, this is in general for Vibes coding specifically, I think that's maybe more of an RL scaling and post-training thing where you can actually get quite a lot of data and train the model to do that really well.

46:26So zooming out a little bit, maybe for the last part of this, conversation i'm curious about where things are going in in general there was a uh a key theme discussed at neurops uh this year around continual learning and i'm curious about your perspective especially from a pre-training perspective right because we are in this paradigm where like every few months or years we and by we i mean you train a uh a very large new base model first of all what is continual learning and two how does that impact uh retraining if continual learning becomes a thing yeah i guess continual learning is about um updating the the model with new knowledge as as new knowledge is discovered right let's say a new scientific breakthrough is made tomorrow the base model we trained yesterday you wouldn't actually know about it in its pre-training first i think a lot of progress has been made on this front since in the last few years i think this is mostly around post-training, around search, use search tools and make search calls, then they would have access to that new information.

47:33In some sense, this is also what Retro that we talked about was doing by retrieving data and then trying to externalize the knowledge corpus with the reasoning part. So that's the first part. I think the second part is on the pre-training side specifically is what I was mentioning on long context as well. And one way of doing this is if you can keep expanding the context of the user, the model keeps getting more and more information in that context. And so you kind of have this continual learning aspect part of that. But then, of course, there's more of a paradigm shift. Maybe this is what people discuss is, can you change the training algorithm such that you can continuously train them on a stream of data coming from the world, basically?

48:17Beyond the continual learning, what do you think is hot slash interesting or intriguing in current research today? Yeah, there's a lot of, again, there's a lot of small things right now that accumulate. So that's kind of the first thought that comes to my mind. And that historically has really driven progress. I wouldn't just bet against that continuing to drive progress. The things I mentioned before around the long context architecture and long context research is one aspect. I think on the attention mechanism as well on the pre-training side. And then this paradigm shift from infinite data to the limited data or finite data regime is something as well, I think, where a lot of things will change.

49:01And there's a lot of interesting research. That's kind of on the pre-training alone side. The other side, which is quite interesting today, is these models become the amount of people using these models is growing quite rapidly. And so more and more what we have to think about on the pre-training side as well is how expensive is the model to use to serve and have really deployed at a large scale. And what things on the pre-training side specifically can we do to make this model have better quality and maybe be cheaper to serve and consume fewer resources during inference. For any student or like PhD student listening to this, if they want to become you in a few years, what problems do you think they should think about or focus on that's not, you know, like a year or two out, but like more interesting sort of a few years out?

49:55One thing that's becoming increasingly important is being able to do research, but being aware of the system side of things. So we're building these fairly complicated systems now. So being able to understand how the stack works all the way down from TPUs to research is kind of a superpower because then you're able to kind of find these gaps in between different layers that other people weren't necessarily able to see, but also to reason through the implication of your research idea all the way down to the TPU stack. And people that can do that well, I think have a lot of impact in general. So in terms of like specialization, it's really thinking about this research engineering and systems aspects of the model research and not just the pure model architecture research.

50:43That's one. I think personally, I still have a lot of interest in kind of this retrieval research as well that we started with retro. And I think it wasn't quite ripe until now, but the things are changing. And I just think it's not unreasonable to think in the next few years, something like that might actually become viable for a leading model like Gemini. And why was it not ripe and why may that change? I think that's around the complexity side of things I was mentioning. And also the fact that all the capabilities it brings, you can iterate much more quickly in post-training. So what I was saying with search and post-training data, you can give very similar capabilities to the model in a much simpler way.

51:28And as post-training grows and RL scaling grows as well, maybe that shifts again towards more on the pre-training side. Do you think there are areas of AI right now that are over-invested in, where there's a disconnect between what makes sense and where the industry is actually going and investing dollars in? I think it's got a lot better. I think maybe two years ago, what I was seeing is people were still trying to very much create specialized models to solve tasks that were maybe within half a year or a year of reach of generalist models. And I think people have caught up to that much more. And now I believe that for generalist tasks or tasks which don't require extreme specialized models, trying to use a generalist model and maybe not the current version, but the next version might be able to do that.

52:20and then so what that means is research in terms of how you use models and the harness etc is becoming increasingly important and also how you make models and these harnesses more robust to making errors and recover from such errors. Yeah in that vein do you have any advice or recommendation for startups right so seen from the perspective of a founder or the the VCs who love them. There is this feeling that the base models are becoming ever so powerful and then trained on multiple data sets. So it used to be the model is able to converse, but now it's able to do financial work and cap tables and that kind of thing which seems to shrink the area of possibility for startups.

53:07Do you have thoughts on that? Yeah, I think so maybe you have looked at what models were able to do a year or a year and a half ago and then look at what models are able to do today and try to extrapolate that. I think the areas where the models are improving, I think will continue to improve. And then there's maybe some areas where there's not been that much progress and that might be more interesting areas to do research. I don't really have a specific example in mind right now, but that would be the general advice. What are you excited about for the next year or two in terms of your personal journey?

53:40What I like very much about my day-to-day is working with many people and being able to learn from a lot of researchers. So that's what drives me to a large extent. Every day I come to work and I talk to really, really brilliant people and they teach me things that I didn't know before. And so I really like that part of my job. What I was saying multiple times at this point, but there are just so many different things that will compound and different things where there's headroom to improve. I'm really curious because right now I don't really see an end in sight for that kind of line of work to continue giving us progress so actually being able to see this through and see how far this can take us is really interesting at least for the next year or so I don't see this slowing down in any way Great, well that feels like a wonderful place to live it Sebastian thank you so much for being on the pod really appreciate it that was fantastic thank you Thank you very much Hi, it's Matt Turk again thanks for listening to this episode of the MAD podcast.

54:40If you enjoyed it, we'd be very grateful if you would consider subscribing if you haven't already or leaving a positive review or comment on whichever platform you're watching this or listening to this episode from. This really helps us build a podcast and get great guests. Thanks and see you at the next episode.