Deep Dive into Long Context

0:00I'm really impressed by the work of our inference team. I've got a bunch of spicy rag versus long context questions for you. You can rely on context caching to make it both cheaper and faster to answer. What's the limitation of continuing to scale up beyond one to two million? This thing is going to be incredible for coding applications. We will have lots more exciting long context stuff to share with folks.

0:51Welcome back to Release Notes, everyone. How's it going? Today, we're joined by Nikolai Savinov, who's a staff research scientist at Google DeepMind and one of the co-leads for long contacts pre-training. Nikolai, how are you? Yeah, hi. Thanks for inviting me. Let's start off with, at the most foundational level, and we'll sort of build up from that, what is a token and how folks should think about that? So the way you should think about a token, it's basically slightly less than one word in case of text. So a token could be a word, part of a word, or it could be things like punctuation, like commas, full stops, etc.

1:39And for images and audio, it's slightly different, but for text, just think of it as slightly less than one word. Yeah. And why do we need tokens? Like, humans generally are sort of familiar with characters. Why does AI and LLMs have this special concept of a token? What's, like, what does it actually enable? Well, this is a great question, and actually many researchers ask this question themselves. So there were actually quite some papers trying to get rid of tokens and just rely on character-level generation. But the thing is, while there are some benefits of doing that, there are also some drawbacks.

2:23And the most important drawback is, well, the generation is going to be slower. Because you generate roughly one token at a time. And if you are generating a word in one go, it's going to be much faster than generating every character separately. separately. So those efforts, they, I would say they didn't really succeed and we are still using tokens. Yeah. For folks who haven't spent a bunch of time thinking about tokens, there's a bunch of good, uh, Andre Karpathy videos and tweets and stuff like that of how like tokenizers are the root of all like weirdness and complexity and LLMs, like all these weird edge cases that you run into.

3:04It's like most of them are rooted in the fact that the model is not looking at things from a character level, it's looking at it from a token level. And actually, like, the pertinent example, which folks love to go to these days, is, like, counting the characters in a single word. Like, how many R's are there in strawberry is, like, a weird problem to solve. My understanding is because tokenizers, like, break the word into different parts, it's not actually looking at the word at, like, the individual character level. Is that an apt description? Yeah, I think that's a pretty good description of the problem.

3:38And one thing you should realize is that those models, due to tokenization, they view the world very differently from how humans view the world. And when you see a strawberry, you see a sequence of letters. But what the model says, it could be even one token. and then you ask like, hey, count number of R letters in this token. But this is a pretty hard, it's pretty hard to get this knowledge from pre-training because you would need to associate the R letter token that you encountered somewhere on the web with the word strawberry, which is also one token. So if you think about the mental load of doing that, it's not such a trivial task, I would say.

4:33Although obviously when the model can't do it, we start complaining, hey, like if it's AGI, how come it can't count number of R letters in strawberry? That's like a child could do that. Yeah, it is super weird. And actually another interesting thing is that if you watch some of the Carpathi videos, then there are a lot of problems with the white space. So this is an interesting point because normally most tokens are prefixed with the white space. And then some really weird effects might happen because you might encounter problems on the boundaries when you think you are concatenating something, but this concatenation is very unusual for the model to see.

5:27Interesting. That is super interesting. I think this actually takes me to just like generally talking about context windows and I think there's a lot of discussion or obviously we're talking about long context which sort of assumes you know what a context window is but can you give the lay of the land of like how folks should thinking about like what a context window actually is? Why do I, as a user of LLMs or somebody who's building with AI models, why do I need to care about the context window? So context window is, those are basically exactly this context tokens that we are feeding into LLM.

6:02And it could be the current prompt or the previous interactions with the user. It could be the files that the user uploaded, like videos or PDFs. And when you supply context into the model, the model actually has knowledge from two sources. So one source is what I would call in-weight or pre-training memory. So this is a knowledge that, well, the LLM was trained on a slice of the internet, and it learned something from there. it doesn't need additional knowledge to be supplied into context to remember some of those facts. So there is already, even without context, there is some kind of memory present in the model.

6:50But another kind of memory is this explicit in-context memory that you are supplying to the model. And it's pretty important to understand the distinction between those two, because in-context memory is much, much easier to modify and update than in-weight memory. So for some kinds of knowledge, in-weight memory might be just fine. Like if you need to memorize some simple facts that the objects fall down and not up. This is a very basic common fact. It's fine if this knowledge comes from pre-training. but there are some facts which are true at the time of pre-training but then they become obsolete at the time of inference and you would need to update those facts somehow and the context provides you a mechanism for to do this update and it's not only about the up-to-date knowledge there are also different kinds of knowledge like private information The network doesn't know anything about you personally, and it can't read your mind.

8:02So if you want it to be really helpful for you, you should be able to supply your private information into context, and then it will be able to personalize. Without this personalization, it's going to give you generic answers it would give to any human instead of answers tailored to you. And the final category of knowledge which need to be inserted in context is some rare facts. So basically some knowledge which was encountered very sparingly on the internet. And I must say, I suspect this category of knowledge, it might go extinct with time. Maybe future models will just learn the whole slice of the internet by heart.

8:50and we will not need to worry about those. But the reality at this point is that if something is mentioned once or twice on the whole internet, the models are actually unlikely to remember those facts and they are going to hallucinate the answers. So you might want to insert those explicitly into context. And the kind of trade-off we are dealing with is for the short context models, you have limited ability to provide additional context. Basically, you would have a competition between knowledge sources. And if the context is really large, then you can be less picky about what you insert and you can have higher recall and coverage of relevant knowledge.

9:41And if you have higher coverage in context, That means, well, you're going to alleviate all those problems with in-weight memory. Yeah, I think there are so many angles to push on. That was a great description. One of the follow-ups from this is we talked about sort of in-weight memory. We talked about in-context memory or just in-context in general. The sort of third class is around how to bring context in that, like, through RAG systems, retrieval augmented generation. Can you sort of give like a high level description of RAG? And then I've got a bunch of spicy RAG versus long context questions for you.

10:25Yeah, sure. So what RAG does is, well, it's a simple engineering technique. It's an additional step before you pack the information into LLM context. So imagine you have a knowledge corpus and you chunk this knowledge corpus into small textual chunks. And then you use some special embedding model to turn every chunk into a real valued vector. And then based on those real valued vectors, if you get the query at the test time, you can embed the query as well. and then you can compare this real valued vector for query to those chunks from the corpus. And for the chunks which are close to the query, you're gonna say, hey, like I found something relevant, so I'm gonna pack those chunks into context and now I'm running LLM on this.

11:27So that's how RAG works. And this is maybe a silly question. RAG, my sense has always been like, lets you obviously there's like very hard limits on context that you can pass to the model we have 1 million we have 2 million that's awesome but like actually if you look at like internet scale you know wikipedia has you know many trillions of tokens or whatever or maybe not trillions maybe billions of tokens whatever it is um why is like rag as this notion of like bringing in the right context to the model not just like baked into the model itself like is it just that to the point of the conversation, the model just not working well for like, it's just like the wrong research direction to go in or like, why don't you think we build that mechanism in?

12:12Because my face value perspective is like, it seems like that would like kind of be useful. The model could just like do rag. And if I could pass a billion tokens and then let the model sort of figure out heuristically or through the, you know, whatever mechanism, what the right tokens are or is that just like a problem somewhere else in the stack that should be solved and the model shouldn't have to think about that? Well, one thing I want to say is that after we released 1.5 Pro model, there were a lot of debates on social media like is RAG becoming obsolete? And well, from my perspective, not really like, say, enterprise knowledge bases, they constitute billions of tokens and not millions.

12:58And so for this use case, for this scale, you still need RAC. What I think is going to happen in practice is that it's not like RAC is going to be eliminated right now, but rather long context and RAC are going to work together. and the benefit of long context for RAC is that you will be able to retrieve more relevant needles from the context by using RAC and by doing that you're going to increase the recall of the useful information so if previously you were setting some rather conservative threshold and cutting out many potentially irrelevant chunks then now you're going to say hey I have a long context so So I'm going to be more generous.

13:47So I'm going to pull more facts. And so I think there's a pretty good synergy between those. And the real limitation is the latency requirements of your application. So if you need real-time interactions, then, well, you'll have to use shorter context. But if you can afford to wait a little bit more, then, yeah, you're going to use long context just because you can increase the recall by doing that. Why is 1 million just like a marketing number or like is there something like intrinsic that like after a million or 2 million like is there actually like something technically happening around the like million token mark for from a long context perspective or is it literally just we found a number that sounds good and then made the technology work from a from a research perspective?

14:39Well, when I started working on Long Context, the competition at the time, I think it was about 128k or maybe 200k tokens at most. So I was thinking how to set the goals for Long Context projects. And well, it was at the time, it was a small, small part of Gemini. And I originally And I thought, well, I mean, just matching competitors doesn't sound very exciting. So I thought, let's set an ambitious bar. So I thought, well, one million is an ambitious enough step forward. It was like compared to 200K, that's like 5X. And very soon after we released one million, we also actually shipped two million, which was about 10X larger.

15:34and I guess one order of magnitude larger than the previous state of the art that's that's a good goal that's what makes it exciting for people to work on yeah I love that and how my my follow-up spicer question from that is like we shipped 1 million we shipped 2 million rapidly after that like what's the limitation of like continuing to scale up beyond 1 to 2 million Is it like from like a serving perspective, it's too costly or too expensive? Or is it just like the architecture that makes one to two million work, like just like fundamentally breaks down when you go larger than that? Or like how how come we haven't seen the frontier for long context continue to push?

16:19Yeah. So when we released the 1.5 Pro model, we actually ran some inference tests at 10 million. And we got some quality numbers as well, and for, say, single-needle retrieval, it was almost perfect for the whole 10 million context. We could have shipped this model, but it's pretty expensive to run this inference. So I guess we weren't sure if people are ready to pay a lot of money for this. So we started, you know, more, you know, with something more reasonable. like in terms of the price but uh i think in terms of quality that's also a good question because uh it's it was so expensive to run we didn't run many tests and so just you know bringing up this server again it's also quite costly so unless we want to you know ship it to a lot of customers right now it's you know we don't have chips to do that yeah do you think that that will continue to hold that like the it's like this like matt i don't know if it's like an exponential increase in capacity that's needed as we do more long contact stuff but like do you have an intuition that like that will like do we need like fundamental breakthroughs from a research perspective for that to change to make it so that we can actually keep scaling up beyond or is it like one to two million is going to be what we stick with and if you want more than that do rag and then like be really smart about bringing context in and out of the of the context window from the model perspective so my feeling is that we actually need more innovations so it's not just a matter of brute force scaling to actually have close to perfect 10 million context we need to learn more innovations but then in terms of the rack and which paradigm will be more powerful going into the future.

18:21I think that the cost of those models is going to decrease over time. And we're going to try to pack more and more context retrieved with drag into those models. And because the quality is also going to increase, then it's going to be more and more beneficial to do that. Yeah, that makes sense. Can you take us back to when we originally landed long context? My understanding of the story is for 1.5 Pro, we didn't, it wasn't like it had been built for a long context to begin with. I think you had obviously tried to kick off that work stream with others inside of DeepMind. And it ended up just being that the pace of research progress was super fast.

19:07And I think my loose understanding of the story is we had the breakthroughs, we realized it worked, and then it was shortly thereafter, it ended up actually landing in the model side. or like what was the like timeline from the the effort starting to like actually landing it into a model that was available externally to the world oh i think that was uh that was indeed pretty quick and just to clarify we didn't really like we were wishing to to go long and achieve say one million or two million context but we kind of didn't expect uh ourselves to get there that fast and when it actually happened then we thought like hey like this is this is really impressive like we actually made some strides on this task so now we need to ship it and then we actually managed to assemble a pretty awesome team very quickly and the team worked really hard.

20:10Like, to be honest, in my life, I've never seen people working this hard. I was really impressed. I love that. That's awesome. And that was for the original 1.5 Pro series. We landed it for 1.5 Flash as well. We now have it for 2.0 Flash. We have 2.5 Pro. And I think the, can you sort of give us the lay of the land of like, what's, what's been happening from from a long context perspective, from that original launch when we know long context is possible, we released the technical report for 1.5 Pro, which showed the needle in the haystack results, a bunch of stuff like that, to today, where I think a lot of what's actually making this 2.5 Pro model blow people's mind is actually how strong it is at long context, which has been awesome for coding use cases and stuff like that.

21:01So what's happened in the long context world from original launch to today? Yeah, so I think the biggest improvement was actually the quality. And we made strides both on the quality at, say, 128k context and also at 1 million context. So if we look at the benchmark results for 2.5 Pro model, we observe that it's better compared to many strong baselines, like GPT 4.5, CLOT 3.7, and also O3 MiniHi, and some of the DeepSeq models. So the quality, like, to actually compare it to those models, we had to run the evals at 128k so that they all comparable and we saw quite a big improvement for 2.5 pro and now in terms of 1 million context we compared it with 1.5 pro and we also saw some significant advantages this is maybe a weird question but like does the quality like ebb and flow at different context sizes?

22:17Like, do you see like, is it like, like almost like linear quality, like on a, you know, 100 ,000 token input versus like 128 ,000 or like a 50 ,000 versus 100? Like, is it like pretty consistent across? Or is there like weird, like, I'm trying to imagine, maybe the, it all generalizes when you make it into the final model, and there's no there's no difference. But like, is there any like, nuance in that perspective? Have we done evals that show anything like that? Yeah, internally we looked at some of those evals. I guess maybe your question goes into these effects that people observed in the past, like a very popular one was lost in the middle effect.

22:57And to answer your question, the lost in the middle effect where you have a deep in the middle of the context, we don't really observe this with our models. But what we do observe is that if it's a hard task, not like a single needle, but some task with the hard distractors, then the quality slightly decreases with the increasing context. And that's something we want to improve on. Yeah. And just for my own mental model, does like when I think about putting 100 ,000 tokens into the context window of the model, Should the should like from a developer perspective or a user of who's like actually using the long context functionality.

23:43Should I assume that the model is like actually attending to all of the different context? I know it could definitely do the like the one needle. It can pull that out. But like, is it actually reasoning over all those tokens like in the in the context window? I have just a bad mental model of what's happening behind the scenes when you have that much context in the context window of the model. Yeah, I think that's a good question. So one thing you need to keep in mind is that attention, in principle, has a bit of a drawback because there's a competition happening between tokens. so if one token gets more attention then other tokens will get less attention the thing is if you have hard distractors then one of the distractors might look very similar to the information that you are looking for and it might attract a lot of attention and now the piece of information that you are actually looking for it's going to receive less attention and the more tokens you have the harder competition becomes so it it depends on the hardness of uh destructors and also also on the context size yeah this is another silly follow-up question but like is the is the amount of attention always fixed like there's no like is it possible to have more attention or is it just like whatever it's like a value of one and then there's like you know spread across all of the tokens in the context window and like so the more tokens you have like literally the less attention there is there's no way for that to change normally that's that's the case that um like the whole pool of attention is um is limited yeah from that example you gave about like the like hard distractors like causing the model to do a lot more work and sort of split the attention has the t has your team explored or other teams on the applied side explored like like pre-filtering mechanisms, any of that type of stuff.

25:50So you want long contacts to work really well in production. It sounds like the best outcome is you have very dissimilar data that's in the context window. If there's a lot of similar data, and you're asking a question that could be relative to all of it, you'd expect the performance to be worse in general in that use case. So is that just something that developers or the world needs to figure out from that perspective? Or do you have any suggestions of how folks should approach that problem for me as a researcher i think it's it would kind of be a move in the wrong direction i think we should work more on improving the quality and robustness instead of coming up with some hacks for filtering one practical recommendation though is of course uh try not to include totally irrelevant context like if you know that something is not useful then what's the goal of including it into the context because in the very minimum it's going to be more expensive so why would you do it yeah that's it's interesting because i feel like um in some sense that like goes against the like core way that people use long time like i think the examples i see online it's just like people being like oh just take all this random data and throw it in the context window of the model and have it sort of figure out what's useful for me So you'd almost expect the model to do, given how important it sounds like it is to remove some of that stuff, the model to do the pre-filtering itself almost to only include the relevant.

27:22Because I think that, not that humans are lazy, but I feel like that's been one of the selling points. It's like, I don't need to think about what data I'm putting into the context window. So do you think there's a world where the model, it's like a multi-part system or something like that, where the model is actually doing some version of the model is doing some of that? eliminating the extraneous data based on what the user's query is and then like making sure when the context actually goes to the model it's a little bit easier over time as the models get better quality and they get cheaper you'll just you'll not need to think about this anymore i'm just talking about the like the current reality is like if you want to make a good use of use of it right now then well let's be realistic uh just don't put irrelevant context but Also, I agree with your point that if you spend too much time manually filtering it or like hand crafting which things to put into context, that's annoying.

28:21So I guess there should be a good balance between those. Yeah. I think the point of context is to simplify your life and make it more automatic, not to make it more time consuming or, you know, make you spend time hand crafting things. Yeah. I've got to follow up on this around evals and the evals that you're thinking about from a long context quality perspective. Needle in a Haystack, obviously, that was the original one that we put into the 1.5 technical report. And for folks who aren't familiar, Needle in a Haystack is just asking the model to find one piece of context in 2 million, 1 million, 10 million tokens of context.

29:00The models are extremely good at this. How do you think about the other set of long context? Like, is there like a set of like standard, I think like folks like generally, I feel like needle and haystack gets talked about a little bit, but are there like another set of like standard benchmarks that you're thinking about from a long context perspective? So let's see, I think the evaluation is pretty much the cornerstone of the LLM research. And especially if you have a large team, evaluation provides a way for the whole team to align and push in the common direction. So the same applies to long context.

29:39If you want to make progress, you need to have great evaluations. Now, single needle in a haystack, it's a solved problem, especially with easy distractors. So if it's like Paul Graham's essays and you put a phrase, here is my magic number for the city of Barcelona is 37. Give me the magic number for the city of Barcelona. This is really a solved problem. But now the frontier of capabilities is handling hard distractors. If you, for example, packed your whole context with a magic number for CTX is Y, and you pack, say, the whole million contexts with these key value pairs, that's a much harder task because then distractors are actually looking very similar to what you want to retrieve.

30:39another thing which is hard for llms is retrieving multiple needles so i i feel like these two things the hardness of destructors and multiple needles they are the frontier but also there are there are additional considerations for the evals one consideration you might have is oh well like those new in the haystack evals even with hard destructors they're pretty artificial so maybe I want something more realistic. This is a valid argument, but the thing you need to keep in mind is that once you increase the realism of the eval, you might actually lose the ability to measure the core long context capability.

31:21For example, if you are asking a question to a very large codebase, and the question is basically can be answered by just one file in this codebase, and then the task is to actually implement something complicated, then you're not really going to be exercising the long context capability. Instead, you are going to be exercising the coding capability. And then it will give you a wrong signal for hill climbing. It will basically hill climb on coding instead of long context. Yeah. So that's one consideration. Another consideration is that something which people call retrieval versus synthesis evolves.

31:58So theoretically, if you need to just retrieve one needle from the haystack, that can be solved by Rack as well. But the tasks that we should really be interested in are the tasks which integrate information over the whole context. And for example, So, well, summarization is one such task, and Rack would have a hard time dealing with this. But now these tasks, it sounds nice and the right direction to go, but they're actually not so easy to use for automatic evaluation. For example, the metrics for summarization, we know that they are like metrics like rouge, etc. They are imperfect. And if you're doing hill climbing, then you're better off using something which is more,

32:56how do I say, less gameable metrics. And just a quick follow-up, what makes them less useful? Like, for summarization as an example, is it just that it's more subjective of what a good summary is versus what isn't it? It doesn't have a ground truth, source of truth? Or what makes that use case hard? Yeah, those of us are going to be pretty noisier because there will be a relatively low agreement even between the human raters. Of course, this is not to give an impression that we shouldn't work on summarization and we shouldn't measure summarization. These are important tasks. I'm just talking about my personal preferences as a researcher is to hill climb on something which has a very strong signal.

33:42Yeah, that makes sense. How do you see sort of as long context, especially for Gemini, is just like a core part of the capability story that we're telling the world. It's like a core differentiator for Gemini. And yet at the same time, it feels like the long context has always been like an independent work stream of like everything isn't long context. Do you think there's a world where like, you know, we have, there's a ton of other teams hill climbing on a bunch of other random stuff, factuality, whatever it is, reasoning, et cetera, et cetera. Do you think the directional, from a research perspective, from a modeling perspective, is that long context is just fused into every other work stream?

34:21Or do you think there's still, it needs to be an independent work stream because it's just fundamentally different in how you get the model to do useful stuff with long context versus reasoning as a corollary example, perhaps? So I guess my answer will be twofold. First of all, I find it helpful to have an owner for every important capability. But second, I think it's important for the work stream to also provide tools for people outside of this work stream to contribute. Yeah, that makes a ton of sense. I have another follow up around reasoning stuff, and I'm curious how the interplay between reasoning and long context.

35:07We had Jack Ray on and we were both at dinner with Jack last night, and we were talking about random reasoning stuff. Have you been surprised by how much it feels like, and you can correct me if this is wrong, the reasoning capability actually makes long context much more useful? Is that just a normal expected outcome just because the model is spending more time thinking? Or is there some inherent deep connection between reasoning capabilities and long context to make it much more effective? I would say there's a deeper connection. And the connection is that if the next token prediction task improves with increasing context length, then you can interpret this in two ways.