#447 – Cursor Team: Future of Programming with AI

28:21And so the perfect fit for that is using a sparse model, meaning an MLE model. So that was one breakthrough we made that substantially improved performance at longer context. The other being a variant of speculative coding that we kind of built out called speculative edits. These are two, I think, important pieces of what make it quite high quality and very fast. Okay, so M .O .E. makes your vx -words the input is huge, the output is small. Yeah. Okay, so what else can you say about how to make it like scashing play a role in this play? Oh, caching plays a huge role because you're dealing with this many input tokens.

29:02If every single keystroke that you're typing in a given line, you had to rerun the model on all those tokens passed in, you're just going to, one, significantly to great latency, two, you're going to kill your GPUs with load. So you need to design the actual prompts to use for the models such that they're cash, cashing aware. And then yeah, you need to reuse the KV cash across requests, just so that you're spending less work, less compute. Again, what are the things that Tab is supposed to be able to do kind of in the near term, just to like sort of linger on that. Generate code, like fill empty space, also edit code across multiple lines.

29:47And then jump to different locations at the same file. And then like launch. Hopefully jump to different files also. So if you make an edit in one file and maybe you have to maybe you have to go to another file to finish your thought, it should go to the second file also. And then the full generalization is like like next action prediction. Like sometimes you need to run a command in the terminal and it should be able to suggest the command based on the code that you wrote to. Or sometimes you actually need to, like it suggests something, but it's hard for you to know if it's correct because you actually need some more information to learn like you need to know the type to be able to verify that it's correct.

30:29And so maybe it should actually take you to a place that's like the definition of something and then take you back so that you have all the requisite knowledge to be able to accept the next completion. So providing the human the knowledge. Yes. Right. Yeah. Can you integrate like I just gone to know a guy named Prime Gen who I believe has an SS, you can order coffee via SSH. Oh yeah. Oh, we did that. We did that. So can that also the model do that like to eat you and my age and provide you with caffeine. Okay, so that's the general framework. Yeah, and the magic moment would be if programming is this weird discipline where sometimes the next five minutes, not always, but sometimes the next five minutes what you're going to do is actually predictable from the stuff you've done recently.

31:20And so can you get to a world where that next five minutes either happens by you just engaging in it, taking you through, or maybe a little bit more of just you seeing next step what it's going to do. And you're like, okay, that's good. That's good. that's good, that's good. And you can just sort of tap, tap, tap through these big changes. As we're talking about this, as you mentioned, one of the really cool and noticeable things about cursor is that there's this whole diff interface situation going on. So like the model suggests with the red and the green of like here's how we're going to modify the code.

31:50And in the chat window, you can apply and it shows you the diff and you can accept the diff. So maybe can you speak to whatever direction of that. We'll probably have four or five different kinds of diffs. So we have optimized the diff for the autocomplete, so that has a different diff interface than when you're reviewing larger blocks of code. And then we're trying to optimize another diff thing for when you're doing multiple different files. And at a high level, the difference differences for when you're doing auto complete, it should be really, really fast to read. Actually, it should be really fast to read in all situations, but in auto complete, it's sort of you're really like your eyes focused on one area.

32:37You can't be in too many. The humans can't look in too many different places. So you're talking about on the interface side. On the interface side. So it currently has this box on the side. So we have the current box. and if you try to delete code in some place and try to add other code, it tries to show you a box on the side. You may be sure if we pull it up and cursor .com. This is what we're talking about. So that box, it was like three or four different attempts at trying to make this thing work, where first the attempt was like these blue crossed out lines. So before it was a box on the side, It's used to show you the code to delete by showing you like like Google Doc style.

33:19You would see like a line through it. Then you would see the new code. That was super distracting. And then we tried many different, you know, there was sort of deletions. There was trying to read highlight. Then the next iteration of it, which is sort of funny, would you would hold the on Mac the option button. So it would it would sort of highlight a region of code to show you that there might be something coming. So maybe in this example, like the input and the value would get would all get blue. And the blue would highlight that the AI had a suggestion for you. So instead of directly showing you the thing, it would show you that the AI it would just hint that the AI had a suggestion.

34:02And if you really wanted to see it, you would hold the option button. and then you would see the new suggestion. Then if you release the option button, you would then see your original code. So that's by the way, that's pretty nice, but you have to know to hold the option button. Yeah. By the way, it was not a Mac user, but I got it. It was it's a button, I guess, you people have. It's again, it's just not intuitive. I think that's the key thing. And there's a chance this is also not the final version of it. I am personally very excited for making a lot of improvements in this area. We often talk about it as the verification problem, where these dips are great for small edits, for large edits, or when it's multiple files or something.

34:52It's actually a little bit prohibitive to review these dips. And so there are like a couple of different ideas here. Like one idea that we have is, okay, you know, like parts of the diffs are important. They have a lot of information. And then parts of the diff are just very low entropy. They're like the same thing over and over again. And so maybe you can highlight the important pieces and then gray out the not so important pieces. Or maybe you can have a model that looks at the diff and sees, oh, there's a likely bug here. I will like mark this with a little red squiggly and say like you should probably like review this part of the diff and Ideas in that vein.

35:36I think are exciting. Yeah, that's a really fascinating space of like UX design engineering Yeah, so you basically trying to guide the human programmer through all the things they need to read and nothing more Yeah, like optimally Yeah, and you want an intelligent model to do it like currently diff algorithms are, they're like normal algorithms. There is no intelligence. There's intelligence that went into designing the algorithm, but then there is no, you don't care if it's about this thing or this thing, as you want a model to do this. So I think the general question is like, Matt, these models are going to get much smarter.

36:19as the models get much more, the changes they will be able to propose are much bigger. So the apps that changes gets bigger and bigger and bigger. The humans have to do more and more and more verification work. It gets more and more hard. Like, you need you need to help them out. It's sort of, I don't want to spend all my time reviewing code.

36:41Can you say a little more across multiple files div? Yeah, I mean, so GitHub tries to sell this, right, with CodeReview. When you're doing CodeReview, you're reviewing multiple deaths across multiple files. But like Arvid said earlier, I think you can do much better than CodeReview. CodeReview kind of sucks. You spend a lot of time trying to grok this code that's often quite unfamiliar to you. And it often doesn't even actually catch that many bugs. And I think you can significantly improve their review experience using language models. For example, using the kinds of tricks that ARP had described, maybe pointing you towards the regions that actually matter.

37:27I think also if the code is produced by these language models and it's not produced by someone else. Like the code review experience is designed for both the reviewer and the person that produced the code. In the case where the person that produced the code is a language model, you don't have to care that much about their experience. You can design the entire thing around the reviewers, such that the reviewers' job is as fun, as easy, as productive as possible. I think that feels like the issue with just kind of naively trying to make these things look like code review. I think you can be a lot more creative and push the boundary in what's possible.

38:09Just one idea there is I think ordering matters. Generally, when you review a PR, you have this list of files and you're reviewing them from top to bottom, but actually you actually want to understand this part first because that came logically first. And then you want to understand the next part. And you don't want to have to figure out that yourself. You want a model to guide you through the thing. And is the step of creation going to be more and more in natural language? Is the goal versus with actual right? I think sometimes I don't think it's going to be the case that all of programming will be natural language and The reason for that is you know if I'm per programming with swallowing swallowing is that the computer in the keyboard and Sometimes if I'm like driving I want to say to swallow a hey Like implement this function and that that works and then sometimes it's just so annoying to explain to swallow of what I want him to do.

39:04And so I actually take over the keyboard, and I show him, I write part of the example, and then it makes sense. And that's the easiest way to communicate. And so I think that's also the case for AI. Sometimes the easiest way to communicate with AI will be to show an example, and then it goes and does the thing everywhere else. Or sometimes if you're making a website, for example, the easiest way to show to the AI what you want does not to tell it what to do, but drag things around or draw things and yeah and like maybe eventually we will get to like brain machine interfaces or whatever and it kind of like understand what you're thinking.

39:39And so I think natural language will have a place. I think it will not definitely not be the way most people program most of the time. I'm really feeling the age guy with this editor. It feels like there's a lot of machine learning going on underneath. Tell me about some of the ML stuff that makes it all work. Recursor really works via this ensemble of custom models that we've trained alongside the frontier models that are fantastic at the reasoning intense things. So cursor tab, for example, is a great example of where you can specialize this model to be even better than even frontier models if you look at evels on the task we set it at.

40:16The other domain, which it's kind of surprising that requires custom models, but it's kind of necessary and works quite well, is in apply. So I think these models are like the frontier models are quite good at sketching out plans for code and generating like rough sketches of like the change, but actually creating deaths is quite hard for frontier models for when you're training models. like you try to do this with Sonnet, with 01, any frontier model. And it really messes up stupid things like counting line numbers, especially in super, super large files. And so what we've done to alleviate this is we let the model kind of sketch out this rough code block that indicates what the change will be.

41:04And we train a model to then apply that change to the file. And we should say that apply is the model looks at your code. It gives you a really damn good suggestion of what new things to do and the seemingly for human trivial step of combining it to you're saying is not so trivial. Contrary to popular perception, it is not a deterministic algorithm. Yeah. I think like you see shallow copies of apply elsewhere. and it just breaks most of the time because you think you can kind of try to do some deterministic matching and then it fills, you know, at least 40 % of the time and that just results in a terrible product experience.

41:49I think in general this regime of you are going to get smarter and smarter models. And like so one other thing that Applied lets you do is it lets you use fewer tokens with the most intelligent models. This is both expensive in terms of latency for generating all these tokens and cost. So you can give this very, very rough sketch and then have your model models go and implement it because it's a much easier task to implement. This very, very sketched out code. I think this regime will continue where you can use smarter and smarter models to the planning and then maybe the implementation details can be handled by the less intelligent ones.

42:29perhaps you'll have maybe a one, maybe it'll be even more capable models, giving an even higher level plan that is kind of recursively applied by Sonna and an Epoil model. Maybe we should talk about how to make it fast. Yeah. If you like to. Yeah. Fast is always an interesting detail. Fast and good. Yeah. How do you make it fast? Yeah. So one big component of making it fast is speculative edits. So speculative edits are a variant of of speculative decoding. And maybe be helpful to briefly describe speculative decoding. With speculative decoding, what you do is you can take advantage of the fact that most of the time, and I'll add the caveat that it would be when you're memory bound in language model generation.

43:17If you process multiple tokens at once, it is faster than generating one token at a time. So this is the same reason why, if you look at tokens for seconds with prompt tokens versus generated tokens, it's much, much faster for prompt tokens. So what we do is instead of using what speculative decoding normally does, which is using a really small model to predict these draft tokens, state your larger model will then go in and verify. With code edits, we have a very strong prior of what the existing code will look like, and that prior is literally the same exact code. So you can do is you can just feed chunks of the original code back into the model.

44:00And then the model will just pretty much agree most of the time that, OK, I'm just going to spit this code back out. And so you can process all of those lines in parallel. And you just do this with sufficiently many chunks. And then eventually, you'll reach a point of disagreement where the model will now predict text that is different from the ground truth original code. It'll generate those tokens. And then we will decide after enough tokens match the original code to restart speculating a chunks of code. What this actually ends up looking like is just a much faster version of normal editing code.

44:35It looks like a much faster version of the model rewriting all the code. We can use the same exact interface that we use for DIFs, but it will just stream down a lot faster. And then the advantages that wireless streaming, you can also be reviewing, start reviewing, viewing the code before it's done. So there's no big loading screen. So maybe that is part of the advantage. So the human can start reading before the thing is done. I think the interesting riff here is something like speculation is a fairly common idea now it is. It's like not only in language models, I mean, there's obviously speculation and CPUs and there's speculation for databases and speculation all over the place.

45:19Let me ask the ridiculous question of which LLM is better at coding. GPT, Claude, who wins in the context of programming? And I'm sure the answer is much more nuanced because it sounds like every single part of this involves a different model. Yeah, I think there's no model that Prado dominates others, meaning it is better in all categories that we think matter. The categories being speed, ability to edit code, ability to process lots of code, long context, you know, a couple of other things and kind of coding capabilities. The one that I'd say right now is just kind of net best is on it. I think this is a consensus opinion.

46:07Our one's really interesting and it's really good at reasoning. So if you give it really hard programming interview style problems or lead code problems. It can do quite, quite well in them, but it doesn't feel like it kind of understands your rough intent as well as so on it does. Like if you look at a lot of the other frontier models, one quam I have is it feels like they're not necessarily over, I'm not saying they train on benchmarks, but they perform really well in benchmarks, relative to kind of everything that's kind of in the middle. So if you try it on all these benchmarks and things that are in the distribution of the benchmarks they're evaluated on, you know, they'll do really well.

46:49But when you push them a little bit outside of that, so on it's I think the one that kind of does best at kind of maintaining that same capability. Like you kind of have the same capability in the benchmark as when you try to instruct it to do anything with coding. What another day goes question is the difference between the normal programming experience versus what benchmarks represent. Where do benchmarks fall short, do you think when we're evaluating these models? By the way, that's like a really, really hard. It's like, critically important detail. How difference like benchmarks are versus real coding.

47:25We're real coding. It's not interview style coding. You're doing these humans are saying half broken English sometimes and sometimes you're saying like, oh, do what I did before. Sometimes you're saying, you know, go add this thing and then do this other thing for me and then make this UI element. And then, you know, it's just like a lot of things are sort of context dependent. You really want to like understand the human and do what the human wants. As opposed to sort of this, maybe the way to put it is sort of abstractly is. The interview problems are very well specified. Dave lean a lot on specification, while the human stuff is less specified.

48:16Yeah. I think that this vent work question is both complicated by what? So I just mentioned. And then also to what a mon was getting into is that even if you like, there's this problem of like the skew between what can you actually model in a benchmark Mark versus real programming. And that can be sometimes hard to encapsulate because it's like real programming is very messy and sometimes things aren't super well specified what's correct or what isn't. But then it's also doubly hard because of this public benchmark problem. And that's both because public benchmarks are sometimes kind of fill climbed on.

48:48Then it's like really, really hard to also get the data from the public benchmarks out of the models. And so for instance, like one of the most popular are like agent benchmarks, sweetbench, is really, really contaminated in the training data of these foundation models. So if you ask these foundation models to do a sweetbench problem, you actually don't give them the context of a code base. They can hallucinate the right file pass, they can hallucinate the right function names. So it's also just the public aspect of these things is tricky. Yeah, like in that case, it could be trained on the literal issues or pull requests themselves.

49:24And maybe the labs will start to do a better job, or they've already done a good job at decontaminating those things, but they're not going to emit the actual training data of the repository itself. These are all some of the most popular Python repositories, like Sympi is one example. I don't think they're going to handicap their models on Sympi and all these popular Python repositories in order to get true evaluation scores in these benchmarks. I think that given the dearth in benchmarks, there have been a few interesting crutches that build systems with these models or build these models actually use, see how they're going in the right direction or not.

50:03And in a lot of places, people will actually just have humans play with the things and give qualitative feedback on these. Like, one or two of the foundation model companies, they have people who, that's a big part of their role. and internally we also qualitatively assess these models and actually lean on that a lot in addition to like private e -vails that we have. It's like the vibe. Yeah, the vibe. The vibe benchmark, human benchmark. When you pull in the humans to do a vibe check. Yeah. Okay. I mean, that's kind of what I do, like just like reading online forums and Reddit and X, Just like, well, I don't know how to properly load in people's opinions, because they'll say things like, I feel like Claude or GPT's gotten dumber or something.

50:51They'll say, I feel like, and then I sometimes feel like that too, but I wonder if it's the model's problem or mine. You know, with Claude, there's an interesting take I heard where I think AWS has different chips and I suspect they have slightly different numerics than Nvidia GPUs and someone speculated that clause degraded performance had to do with maybe using the quantized version that existed on AWS bedrock versus whatever was running on Anthropics GPUs. I interview a bunch of people that have conspiracy theories. I'm glad to spoke to this conspiracy theory. It's not like conspiracy theory as much as it is.

51:37They're like, you know, humans are humans and there's these details. And you know, you're doing like these queasy monoflops and, you know, chips are messy and man, you can just have bugs like bugs are, it's hard to overstate how hard bugs are to avoid. What's the role of a good prompt in all this? See, you mentioned that benchmarks have really structured well -formulated prompts. What should a human be doing to maximize success and what's the importance of what the human's Euro -Doblock post called it prompt design? Yeah, I think it depends on which model you're using and all of them are slightly different and they respond differently to different prompts.

52:26But I think the original GP -D4 and the original sort of breedable models last year, they were quite sensitive to the prompts. They also had a very small context window. And so we have all of these pieces of information around the codebase that would maybe be relevant in the prompt. Like you have the docs, you have the files that you add, you have the conversation history. And then there's a problem like how do you decide what you actually put in the prompt and when you have a limited space and even for today's models, even when you have long context, filling out the entire context window means that it's slower and means that sometimes the model actually gets confused and some models get more confused and others.

53:09And we have this one system internally that we call preempt which helps us with that a little bit. And I think it was built for the era before where we had 8 ,000 token contacts windows. And it's a little bit similar to when you're making a website, you sort of, you wanted to work on mobile, you wanted to work on a desktop screen. And you have this dynamic information, which you don't have, for example, if you're making like designing a print magazine, you know exactly where you can put stuff. But when you have a website or when you have a prompt, you have these inputs. And then you need to format them to always work.

53:54Even if the input is really big, then you might have to cut something down. And so the idea was, okay, let's take some inspiration. What's the best way to design websites? Well, the thing that we really like is react and the declarative approach where you you use JSX in JavaScript. And then you declare this is what I want. And I think this has higher priority or like this has higher Z index than something else. And then you have this rendering engine in web design, it's like Chrome and in our cases that pre -empt renderer, which then fits everything onto the page. And as you declare it, we decide what you want and then it figures out what you want.

54:36And as we have found that to be quite helpful, and I think the role of it has sort of shifted over time, where initially it was to fit to these small context windows. Now it's really useful because it helps us with splitting up the data that goes into the prompt and the actual rendering of it. And so it's easier to debug because you can change the rendering of the prompt and then have the raw data that went into their prompt. Then you can see, did my change actually improve it for this entire e -bilt set? So do you literally prompt with JSX? Yes. So it kind of looks like React. There are components.

55:17We have one component that's a file component. And it takes in the cursor. Usually there is one line where the cursor is in your file. And that's probably the most important line because that's one you're looking at. And so then you can give priorities. So that line has the highest priority. and then you subtract one for every line that is farther away. And then eventually when it's rendered, it figure out how many lines can actually fit in a center surround that thing. That's amazing. And you can do other fancy things where if you have lots of code blocks from the entire code base, you could use retrieval and things like embedding and re -ranking scores to add priorities for each of these components.

55:57So should humans, when they ask questions, also try to use something like that? like would it be beneficial to RIGSX in the problem? The whole idea is this should be loose and messy. I think our goal is kind of that you should just do whatever is the most natural thing for you. And then we our job is to figure out how do we actually like retrieve the relative event things so that you're thinking actually makes sense. Well, this is the discussion I had with Arvand of Proplexity. It's like his whole idea is like you should let the person be as lazy. Yeah, but like yeah, that's a beautiful thing, but I feel like you're allowed to ask more of programmers, right?

56:40So like if you say just do what you want, I mean humans are lazy. There's a kind of tension between just being lazy versus like provide more as be prompted almost like the system pressuring you or inspiring you to be articulate. Not in terms of the grammar of the sentences, but in terms of the depth of thoughts that you convey inside the other problems. I think even as a system gets closer to some level of perfection, often when you ask the model for something, you just are not, not enough intent is conveyed to know what to do. And there are like a few ways to resolve that intent. One is the simple thing of having models just ask you, I'm not sure how to do these part space in your query, could you clarify that?

57:30I think the other could be maybe if you, there are five or six possible generations given the uncertainty present in your query so far, why don't we just actually show you all those and let you pick them? How hard is it to, for the model, to choose to talk back, sort of versus, it's hard to sort of like how to deal with the uncertainty. Do I choose to ask for more information to reduce the ambiguity? So I mean, one of the things we do is, it's like a recent addition is try to suggest files that you can add. So while you're typing, one can guess what the uncertainty is and maybe suggest that like, maybe you're writing your API, and we can guess using the commits that you've made previously in the same file that the client and the server is super useful.

58:35And there's like a hard technical problem of how do you resolve it across all commits which files are the most important, given your current prompt. And we're still sort of initial versions rolled out and I'm sure we can make it much more accurate. It's very experimental. But then the idea is we show you, like do you just want to add this file, this file, this file also to tell the model to edit those files for you? Because if you're maybe you're making the API, like you should also edit the client and the server that is using the API and the other one is only the API. So that will be kind of cool as both there's the phase where you're writing the prompt and there's before you even click enter, maybe we can help resolve some of the uncertainty.

59:20to what degrees agentic approaches. How do you survive agents? We think agents are really, really cool. I think agents is like, it's like resembles sort of like a human. It's sort of like the things, like you can kind of feel that it, like you're getting closer to AGI, because you see a demo where it acts as a human would. And it's really, really cool. I think agents are not yet super useful for many things. They, I think, we're getting close to where they will actually be useful. And so, I think there are certain types of tasks we're having an agent would be really nice. Like, I would love to have an agent.

1:00:07For example, if, like, we have a bug where you sometimes can't can't come and see and come and be inside our chat input box. And that's a task that's super well specified. I just want to say like in two sentences, this does not work, please fix it. And then I would love to have an agent that just goes off, does it? And then a day later, I come back and I reviewed the thing. You mean it goes, finds the right file? Yeah, it finds the right files. It tries to reproduce the bug, it fixes the bug, and then it verifies that it's correct. And this could be a process that takes a long time. And so I think I would love to have that.

1:00:43And then I think a lot of programming, like there is often this belief that agents will take over all of programming. I don't think we think that that's the case, because a lot of programming, a lot of the value, is in iterating, or you don't actually want to specify something upfront, because you don't really know what you want until you've seen an initial version, and then you want to iterate on that. and then you provide more information. And so for a lot of programming, I think you actually want a system that's instant that gives you an initial version instantly back and then you can iterate super, super quickly.

1:01:18What about something like that RISN came out, a replete agent that does also like setting up the development environment and solving software packages, configuring everything, configing the databases and actually deploying the app? Is that also in the set of things you dream about? I think that would be really cool. For certain types of programming, it would be really cool. Is that within scope of cursor? Yeah, we aren't actively working on it right now, but it's definitely like we want to make the programmers life easier and more fun. And some things are just really tedious and you need to go through a bunch of steps and you want to delegate that to an agent.

1:02:00And then some things you can actually have an agent in the background while you're working, like let's say you have a PR that's both back end and front end and you're working in the front end and then you can have a background agent that doesn't work and figure out kind of what you're doing and then when you get to the back end part of your PR then you have some like initial piece of code that you can iterate on and so that would also be really cool. One of the things we already talked about is speed. But I wonder if we can just linger on that some more in the various places that the technical details involved in making this thing really fast.

1:02:38So every single aspect of cursor, most aspects of cursor feel really fast. Like I mentioned the applies probably the slowest thing and for me from the pain. It's a pain that we're feeling and we're working on fixing it. Yeah, I mean, it says something that something that feels, I don't know what it is, like one second or two seconds, that feels slow. That means that's actually shows that everything else is just really, really fast. So is there some technical details about how to make some of these models, how to make the chat fast, how to make the divs fast? Is there something that just jumps to mine?

1:03:15Yeah, I mean, so we can go over a lot of the strategies that we use. One interesting thing is cache warming. And so what you can do is if, as the user is typing, you're probably going to use some piece of context. And you can know that before the user is done typing. So as we discussed before reusing the kvcache, results in lower latency, lower cost, cross request. So as the user starts typing, you can immediately warm the cache with, let's say, the current file contents. And then when they press enter, there's very few tokens it actually has to pre -fill and compute before starting the generation.

1:03:55This will significantly lower GTFD. Can you explain how KVCash works? Yeah. So, the way Transformers work. I like it. I mean, one of the mechanisms that allow Transformers to not just independently, like the mechanism that allows Transformers to not just independently look at each token, but see previous tokens. are the keys and values to tension. And generally the way attention works is you have, at your current token, some query, and then you've all the keys and values of all your previous tokens, which are some kind of representation that the model stores internally of all the previous tokens in the prompt.

1:04:35And like by default, when you're doing a chat, the model has to, for every single token, do this for pass through the entire model. That's a lot of matrix multiplies that happen and that is really really slow. Instead, if you have already done that and you stored the keys and values and you keep that in the GPU, then when I'm, let's say I have sorted for the last end tokens, if I now want to compute the output token for the n plus one token, I don't need to pass those first end tokens through the entire model because I already have all those keys and values. And so you just need to do the forward pass through that last token.

1:05:16And then when you're doing attention, you're reusing those keys and values that have been computed, which is the only kind of sequential part or sequentially dependent part of the transformer. Is there like higher level caching of like caching of the prompts or that kind of stuff to help? Yeah, that there's other types of caching you can kind of do. one interesting thing that you can do for a cursor tab is you can basically predict the head as if the user would have accepted the suggestion and then trigger another request. And so then you've done the speculative, it's a mix of speculation and caching, right?

1:05:57Because you're speculating what would happen if they accepted it. And then you have this value that is cash, this suggestion. And then when they press tab, the next one would be waiting for them immediately. It's a kind of clever, heuristics -lash trick that uses a higher level caching and can give the... It feels fast despite there not actually being any changes in the model. And if you can make the KV cache smaller, one of the advantages you get is like maybe you can speculate even more. Maybe you can get here's the 10 things that could be useful. But I like, like, predict the next 10. And then it's possible the user hits the one of the 10.

1:06:36It's like much higher chance than the user hits like the exact one that you showed them. Maybe the typing other character hit. And we sort of hit something else in the cache. So there's all these tricks where the general phenomena here is, I think it's also super useful for RL is, maybe a single sample from the model isn't very good. But if you predict like 10 different things Turns out that one of the 10 that's right is the probabilities much higher There's these pass at key curves and you know part of our L shot like what RL does is You can you can exploit this pass at K phenomena to make many different predictions and and One way to think about this the model sort of knows internally has like has some uncertainty over like which of the K things is correct or like which of the case things does the human wants?

1:07:31When we are LR, you know, Kurshut Hab model, one of the things we're doing is we're predicting which of the hundred different suggestions the model produces is more amenable for humans, like which of them to humans more like than other things. Maybe like there's something where the model can predict very far ahead versus like a little bit and maybe somewhere in the middle and just, and then you can give a reward to the things that humans would like more and sort of punish the things that it would like and sort of then train the model to output the suggestions that humans would like more. You have these like RL loops that are very useful that exploit these passive kickerves.

1:08:12Oman maybe can go into even more detail. Yeah, it's a little, it is a little different than speed, but I mean, like technically you tie it back in because you can get away at the smaller model if you RL your smaller model and it gets the same performance as the bigger one. That's like, and while I was mentioning stuff about KB about reducing the size of your KB cache, there are other techniques there as well that are really helpful for speed. So kind of back in the day, like all the way two years ago, people mainly use multi -head attention. And I think there's been a migration towards more efficient attention schemes like group query or multi -query attention.

1:08:55And this is really helpful for then with larger batch sizes, being able to generate the tokens much faster. The interesting thing here is this now has no effect on that time to first token, pre -fill speed. The thing this matters for is now generating tokens. And why is that? Because when you're generating tokens instead of being bottlenecked by doing the super paralyzable matrix multiplies across all your tokens, your bottleneck by how quickly it's for long context with large batch sizes, by how quickly you can read those cache keys and values. And so then that's memory bandwidth and how can we make this faster?

1:09:38We can try to compress the size of these keys and values. So multi query attention is the most aggressive of these, where normally with multi -head attention you have some number of quote -unquote attention heads and some number of kind of query heads. Multi -query just preserves the query heads, gets rid of all the key value heads. So there's only one kind of key value head and there's all the remaining query heads. With group query, you instead, you know, preserve all the query heads and then your keys and values are kind of, there are fewer heads for the keys and values, but you're not reducing it to just one.

1:10:22But anyways, like the whole point here is you're just reducing the size for your KV cache. And then there is the MLA. Yeah, multi latent. That's a little more complicated. And the way that this works is, it kind of turns the entirety of your keys and values across all your heads into this kind of one latent vector. that is then kind of expanded in for its time. But MLA is from this company called DeepSeek. It's quite an interesting algorithm. Maybe the key idea is sort of in both MQA and in other places, what you're doing is sort of reducing the number of KV heads that Vanage you get from that is, there's less of them, but maybe the theory is that you actually want a lot of different, like you want each of the keys and values to actually be different.

1:11:19So one way to reduce the sizes, you keep one big shared vector for all the keys and values, and then you have smaller vectors for every single token, so that when you can store the only smaller thing, There's some sort of like lore rank reduction. And the lore rank reduction, at the end of the time, when you eventually want to compute the final thing, remember that your memory band, which means that you still have some compute left that you can use for these things. And so if you can expand the latent vector back out. And somehow this is far more efficient because you're reducing, for example, maybe you're using big 32 or something like the size of the vector that you're keeping.

1:12:04Yeah, there's perhaps some richness in having a separate set of keys and values and query that kind of pairwise match up versus compressing that all into one. And that interaction at least. Okay, and all of that is dealing with being memory bound. Yeah. And what, I mean, ultimately, how does that map to the user experience trying to get the same thing? Yeah, the two things that a map zoo is you can now make your cash a lot larger because you've less space allocated for the KV cash. You'd maybe cash a lot more aggressively and a lot more things. So you get more cash hits, which are helpful for reducing the time to first token.

1:12:43For the reasons that we're kind of described earlier. And then the second being when you start doing inference with more and more requests and larger and larger batch sizes, you don't see much of a slowdown in as it's generating the tokens the speed of that. It also allows you to make your prompt bigger. For certain. Yeah. Yeah. So the size of your KVCash is both the size of all your prompts, multiply by the number of prompts in your process in parallel. So you could increase either of those dimensions, right? The back size or the size of your prompts without degrading the latency of generating tokens.

1:13:18All right. You wrote a blog post shadow of workspace iterating on code in the background. Yeah. So what's going on? So to be clear, we want there to be a lot of stuff happening in the background, and we're experimenting with a lot of things. Right now, we don't have much of that happening, other than the cash warming or figuring out the right context to go into your Kamehke Proms, for example. But the idea is, if you can actually spend computation in the background, then you can help the user maybe like at a slightly longer time horizon then just predicting the next few lines that you're going to make, but actually like in the next 10 minutes what are you going to make and by doing it in the background you can spend more computation doing that.

1:14:04And so the idea of the shadow workspace that we implemented, and we use it internally for experiments, is that to actually get advantage of doing stuff in the background, you want some kind of feedback signal to give back to the model. Because otherwise, you can get higher performance by just letting the model think for longer. And so, like, a one is a good example of that. But another way you can improve performance is by letting the model iterate and get feedback. And so one very important piece of feedback when you're a programmer is the language server, which is this thing that exists for most different languages, and there's like a separate language server per language, and it can tell you you're using the wrong type here, and then it gives you an error, or it can allow you to go to definition, it sort of understands the structure of your code.

1:14:58So language servers are extensions developed by, like there's a TypeScript language survey developed by the TypeScript people, a Rust language survey developed by the Rust people, and then they all interface over the language survey protocol to VS Code. So that VS Code doesn't need to have all of the different languages built into VS Code, but rather you can use the existing compiler infrastructure. For Linting purposes? What is it? It's for Linting. It's for going to definition, and for seeing the right types that you're using. So it's doing Typechecking also? Yes. Typechecking and going to references.

1:15:31And that's like when you're working in a big project, you kind of need that. If you don't have that, it's like really hard to code in a big project. Can you say again how that's being used inside cursor, the language server protocol communication thing? So it's being used in cursor to show to the programmer just like in VS Code. But then the idea is you want to show that same information to the models, the IAM models. And you want to do that in a way that doesn't affect the user, because you want to do it in background. And so the idea behind the chat at workspace was, okay, like one way we can do this is, we spawn a separate window of cursor, that's hidden.

1:16:12And so you can set this flag and electron is hidden. There is a window, but you don't actually see it. And inside of this window, the AI agents can modify code however they want, as long as they don't save it because it's still the same folder, and then can get feedback from the lenders and go to definition. and iterate on their code. So like literally run everything in the background. Like as if, right. Yeah. Maybe even run the code. So that's the eventual version. OK. That's what you want. And a lot of the blog post is actually about how do you make that happen? Because it's a little bit tricky.

1:16:47You want it to be on the user's machine so that it exactly mirrors the user's environment. And then on Linux, you can do this cool thing where you can actually mirror the file system and have the AI make changes to the files. And it thinks that it's operating on the file level, but actually that's stored in memory. And you can create this kernel extension to make it work. Whereas on Mac and Windows, it's a little bit more difficult. And but it's a fun technical problem. So that's why. One maybe hacky, but interesting idea idea that I like is holding a lock on saving. And so basically you can then have the language model kind of hold the lock on on saving to disk.

1:17:33And then instead of you operating in the ground truth version of the files that are saved to disk, you actually are operating what was the shadow works face before in these unsafe things that only exist in memory that you still get Lintra errors for and you can code in. And then when you try to maybe run code, it's just like there's a small warning that there's a lock and then you kind of will take back the lock from the language server if you're trying to do things concurrently or from the shadow workspace if you're trying to do things concurrently. That's such an exciting feature, by the way.

1:18:00It's a bit of a tangent, but to allow a model to change files, it's scary for people, but it's really cool. To be able to just let the agent do a set of tasks and you come back the next day and kind of observe like it's a colleague or something like that. Yeah. And I think there may be different versions of runability where for the simple things where you're doing things in the span of a few minutes on behalf of the user as they're programming, it makes sense to make something work locally in their machine. I think for the more aggressive things, we're making larger changes that take longer periods of time.

1:18:36You'll probably want to do this in some sandbox remote environment, and that's another incredibly tricky problem of how do you exactly reproduce, or mostly reproduce to the point of it being effectively equivalent for running code, the user's environment, which is remote sandbox. box. I'm curious what kind of agency you want for coding. Do you want them to find bugs? Do you want them to implement new features? What agency you want? So by the way, when I think about agents, I don't think just about coding. I think so for the practice, this particular podcast, this video editing, and a lot of, if you look in Adobe, a lot of this code behind, it's very poorly documented code, but you can interact with premiere, for example, using code and basically all the uploading, everything I do on YouTube, everything as you could probably imagine, I do all of that through code and including translation and overdubbing all of this.

1:19:33So I envision all of those kinds of tasks, so automating many of the tasks that don't have to do directly with the editing. So that, okay, that's what I was thinking about. In terms of coding, I would be fundamentally thinking about bug finding. Many levels of bug finding and also bug finding like logical bugs, not logical like spiritual bugs or something. One's like sort of big directions of implementation, that kind of stuff. Let's do a fine on bug finding. I mean, it's really interesting that these models are so bad at bug finding. when just naively prompted to find a bug. They're incredibly poorly calibrated.

1:20:17Even the smartest model. Exactly. Even know one. How do you explain that? Is there a good intuition? I think these models are really strong reflection of the pre -training distribution. And I do think they generalize as the loss gets lower and lower. But I don't think the loss in the scale is quite, or the loss is low enough, such that they're like really fully generalizing on code. Like the things that we use these things for, the frontier models that they're quite good at, are really code generation and question answering. And these things exist in massive quantities and pre -training with all of the code and GitHub on the scale of many, many trillions of tokens and questions and answers on things like Stack Overflow and maybe GitHub issues.

1:21:05And so when you try to push into these things that really don't exist very much online, like for example, the cursor tab objective of predicting the next edit, given the edit's done so far. The brittleness kind of shows, and then bug detection is another great example, where there aren't really that many examples of actually detecting real bugs and then proposing fixes. And the model's just kind of like really struggling. But I think it's a question of transferring the model, Like in the same way that you get this fantastic transfer from pre -trained models just on code in general to the cursor tab objective, you'll see a very, very similar thing with generalized models that are really good at code to bug detection.

1:21:48It just takes like a little bit of kind of nudging in that direction. Like to be clear, I think they sort of understand code really well. Like while they're being pre -trained, like the representation that's being built up almost certainly, somewhere in the stream, the model knows that maybe there's something sketchy going on, right? It sort of has some sketchiness, but actually eliciting the sketchiness to actually, part of it is that humans are really calibrated on which bugs are really important. It's not just actually saying there's something sketchy. It's like, it's just a sketchy trivial.

1:22:26It's the sketchy, like you're going to take the server down. It's like a part of it is maybe the cultural knowledge of like why is the staff engineer staff engineer a staff engineer is good because they know that three years ago someone drove a really you know sketchy piece of code that took took the server down and as opposed to like as opposed to maybe there's like, you know, you just This thing is like an experiment. So like a few bugs are fine and you're just trying to experiment and get the feel of the thing. And so the model gets really annoying when you're writing an experiment, that's really bad.

1:23:00But if you're writing something for super production, you're writing a database, right? You're writing code in Postgres or Linux or whatever, like your Linus Torval, it's sort of unacceptable to have even in that case. And just having the calibration of like how paranoid is the user? But even then, if you're putting in a maximum paranoia, it still just doesn't quite get it. Yeah, yeah, yeah. I mean, but this is hard for humans to to understand what which line of code is important, which is not like you, I think one of your principles on a website says if a code can do a lot of damage, one should add a comment that say this, this, this line of code is, is dangerous.

1:23:43And all caps, 10 times. No, you say like for every single line of code inside the function, you have to, and that's quite profound. That says something about human beings, because the engineers move on, even the same person might just forget how it can sync the Titanic, a single function. You might not intuit that quite clearly by looking at the single piece of code. Yeah, and I think that one is also partially also for today's AI models, where if you actually write dangerous, dangerous, dangerous in every single line, like the models will pay more attention to that and will be more likely to find bucks in that region.

1:24:26That's actually just straight up a really good practice of labeling code of how much damage this can do. Yeah, I mean, it's controversial. I guess some people think it's ugly. Well, actually, it's like, Like, in fact, I actually think this is one of the things I learned from Arvid is, you know, like I sort of aesthetically, I don't like it. But I think there's certainly something where like it's useful for the models and in humans just forget a lot and it's really easy to make a small mistake and cause like, bring down, you know, like just bring down the server and like, like, or go first, we like test a lot and whatever but there's always these things that you have to be very careful.

1:25:08Yeah, like with just normal doc strings, I think people will often just skim it when making a change and think, oh, I know how to do this. And you kind of really need to point it out to them so that that doesn't slip through. Yeah, you have to be reminded that you can do a lot of damage. That's like, we don't really think about that. Yeah. You think about, okay, how do I figure out how this works so I can improve it? You don't think about the other direction. Yeah. Until we have formal verification for everything. Then you can do whatever you want and you know for certain that you have not introduced a bug if the proof pass.

1:25:45We can really what do you think that future would look like? I think people will just not write tests anymore and The model will suggest like you write a function the model will suggest a spec and you review this spec and in the meantime Smart reasoning model computes a proof that the implementation follows the spec And I think that happens for most functions. I think this gets at a little bit, some of the stuff you were talking about earlier with, the difficulty of specifying intent for what you want to software, where sometimes it might be, because the intent is really hard to specify, it's also then going to be really hard to prove that it's actually matching whatever your intent is.

1:26:24Like you think that spec is hard to generate? Yeah, or just like, for a given spec, maybe you can... I think there is a question of like, can you actually do the formal verification? Like, is that possible? I think that there's more to dig into there. But then also, Even if you have the spec, if you have the spec, how do you have the spec? Is the spec written in natural language? Yeah, how is it more for the spec? No, the spec wouldn't be formal. But how easy would that be? So then I think that you care about things that are not going to be easily well specified in the spec language. I see, I see.

1:26:57Yeah. Yeah. Maybe an argument against formal verification is all you need. Yeah. But there's this massive document. replacing something like unit tests. Sure. Yeah. I think you can probably also evolve the spec languages to capture some of the things that they don't really capture right now. Wait, I don't know. I think it's very exciting. And you're speaking not just about like single functions. You're speaking about entire code bases. I think entire code bases is harder, but that is what I would love to have. And I think it should be possible. And because you can even, there is a lot of work recently where you can prove formally verified down to the hardware.

1:27:40So like, you formally verify the C code and then you formally verify through the GCC compiler and then through the very log down to the hardware. And that's like an incredibly big system, but it actually works. And I think big code business are sort of similar in that they're like multi -layered system. and if you can decompose it and formally verify each part, then I think it should be possible. I think the specification problem is a real problem, but how do you handle side effects? Or how do you handle, I guess, external dependencies, like calling Stripe API? Maybe Stripe would write a spec for their API.

1:28:14But you can't do this for everything. Like, can you do this for everything you use? Like, how do you do it for, if there's a language, like maybe people will use language models as primitives in the program. and there's a dependence on it and how do you now include that? I think it might be able to prove that still. Prove what about language models? I think it feels possible that you could actually prove that a language model is aligned, for example, or like you can prove that it actually gives the right answer. Nice to dream. Yeah, that is, I mean, if it's possible, I have a dream speech, if it's possible, that will certainly help with making sure you call those out bugs and making sure AI doesn't destroy all human civilization.

1:29:01So the full spectrum of AI safety to just bug finding. So you said the models struggle with bug finding. What's the hope? You know, my hope initially is and I can let Michael, Michael chime into it, but it was like this. It should first help with the stupid bugs. It should very quickly catch the stupid bugs. Off by one error, sometimes you write something in a common and do the other way. It's very common. I do this. It's less than in a common and maybe right to greater than something like that. The model is like, yeah, it looks catchy. Do you sure you won't want to do that? But eventually, it should be able to catch harder bugs to you.

1:29:42Yeah, and I think that it's also important to note that this is having good bug finding models feels necessary to get to the highest reaches of having AI do more and more programming for you, where you're going to, you know, if the AI is building more and more of the system for you, you need to not just generate but also verify. And without that, some of the problems that we've talked about before with programming with these models will just become untenable. So it's not just for humans. Like you write a bug, I write a bug, find the bug for me, but it's also being able to verify the AI's code and track it.

1:30:17It's really important. Yeah, and then how do you actually do this? Like we have had a lot of contentious dinner discussions of how do you actually screen a bug model. But one very popular idea is, you know, it's kind of potentially easy to introduce a bug than actually finding the bug. And so you can train a model to introduce bugs in existing code. And then you can train a reverse bug model that can find bugs using this synthetic data. So that's like one example, but yeah, there are lots of ideas for how to do it. You can also do a bunch of work not even at the model level of taking the biggest models and then maybe giving them access to a lot of information that's not just the code.

1:30:58Like it's kind of a hard problem to like stare at a file and be like, where's the bug? And you know, that's hard for humans often, right? And so often you have to run the code and being able to see things like traces and step through to bugger There's another whole another direction where it like kind of tends toward that And it could also be that there are kind of two different product form factors here It could be that you have a really specialty model that's quite fast That's kind of running in the background and trying to spot bugs and it might be that sometimes Sort of sort of Arvid's earlier example about you know some nefarious input box bug It might be that sometimes you want to like there's you know there's a bug You're not just like checking hypothesis free.

1:31:30You're like this is a problem I really want to solve it and you zap that with tons and tons and tons of compute and you're willing to put in like $50 to solve that bug or something even more Have you thought about integrating money into this whole thing like I would pay probably a large amount of money for if you found a bug or even Generated code that I really appreciated like I had a moment a few days ago when I started using cursor or generated Perfect like perfect three functions for interacting with the YouTube API to update captions and full localization in different languages. The API documentation is not very good.

1:32:11And the code across, like if I googled it for a while, I couldn't find exactly, there's a lot of confusing information and cursor generated perfectly. And I was like, I just set back, I read the code, I was like, this is correct, I tested it is correct. I was like, I want a tip on a button that goes, there's $5. One, that's really good just to support the company and support what the interface is. And the other is that probably sends a strong signal, like, good job. Right? There's much stronger signal than just accepting the code, right? You just actually send like a strong good job. That, and for bug finding, obviously, like, there's a lot of people, you know, that would pay a huge amount of money for a bug, like a bug bounty thing, right?

1:32:58Is that, do you guys think about that? Yeah, it's a controversial idea inside the company. I think it sort of depends on how much you believe in humanity, almost. I think it would be really cool if you spend nothing to try to find a bug, and if it doesn't find a bug, you spend zero dollars. And then if it does find a bug, and you click accept, then it also shows like in parentheses like $1. As you spend $1 to accept the bug. And then of course there's a worry like, okay, we spent a lot of computation. Like maybe people will just copy paste. I think that's a worry. And then there's also the worry that like introducing money into the product makes it like kind of, you know, like it doesn't feel as fun anymore.

1:33:42Like you have to like think about money and you all you want to think about is like the code. And so maybe it actually makes more sense to separate it out. and you pay some fee every month and then you get all of these things for free. But there could be a tipping component which is not like it costs it. It still has that dollar symbol. I think it's fine, but I also see the point where maybe you don't want to introduce it. Yeah, I was going to say the moment that feels like people do this is when they share it, when they have fantastic example, they just kind of share it with their friend. There is also a potential world where there's a technical solution to this, like honor system some problem too, where if we can get to a place where we understand the output of the system more, I mean to this stuff we were talking about with like, you know, error checking with the LSP and then also running the code.

1:34:27But if you could get to a place where you could actually somehow verify, oh, I have fixed the bug. Maybe then the Bouchy system doesn't need to rely on the honor system too. How much interaction is there between the terminal and the code? My, how much information is gained from if you run the code in the terminal. I can use, can you do like a loop where it runs the code and suggest how to change the code if the code in runtime gives an error. It's right now they're separate worlds completely. Like I know you can do control K inside the terminal to help you write the code. You can use terminal contacts as well inside of checkman .k kind of everything.

1:35:07We don't have the looping part yet. So we suspect something like this could make a lot of sense. There's a question of whether it happens in the foreground too, or if it happens in the background, like what we've been discussing. Sure. The background is pretty cool. Like we do running the code in different ways. Plus there's a database side to this, which how do you protect it from not modifying the database, but OK. I mean, there's certainly cool solutions there. There's this new API that is being developed for. It's not an AWS, but you know, it's certainly, I think it's in Planet scale. I don't know if Planet scale was the first one to add it.

1:35:45It's the stability sort of add branches to a database, which is like if you're working on a feature and you want to test against the broad database, but you don't actually want to test against the broad database, you could sort of add a branch to the database. And the way they do that is they add a branch to the right analog. And there's obviously a lot of technical complexity in doing it correctly. I guess database companies need new things to do. They have the databases now. And I think like TurboPuffer, which is one of the databases we use as is going to add, maybe branching to the right -end log.

1:36:24And so maybe the AI agents will use branching, they'll like test against them branch, and it's sort of going to be a requirement for the database to like support branching or something. We're really interesting to you branch a file system, right? Yeah. If you like everything needs branching, it's like that. Yeah. Yeah, it's like the problem with the multiverse, right? If you branch an everything, that's like a lot. I mean, there's obviously these like super clever algorithms to make sure that you don't actually use a lot of space or CPU or whatever. Okay, this is a good place to ask about infrastructure.

1:37:01So you guys mostly use AWS. What are some interesting details? What are some interesting challenges? Why do you choose AWS? Why is AWS still winning? Hashtag. AWS is just really, really good. It's really good.

1:37:18Whenever you use an AWS product, you just know that it's going to work. Like it might be absolute hell to go through the steps to set it up. Why is the interface so horrible? Because it's just so good. It doesn't need to be like... The nature of winning. I think it's exactly. It's just nature they're winning. Yeah, yeah. But, Aydavila, as you can always trust, it will always work. And if there is a problem, it's probably your problem. Okay. Is there some interesting challenges to... You guys have pretty new startup to get scaling to like to so many people and yeah, I think that they're It has been an interesting journey Adding you know each extra zero to the request per second But you run into all of these with like the you know the general components you're using for for caching and databases Running to issues as you make things bigger and bigger and now we're at the scale where we get like you know into overflows on our tables and things like that And then also there have been some custom systems that we've built, like for instance, our retrieval system for computing a semantic index of your codebase and answering questions about a codebase that have continually, I feel like, been one of the trickier things to scale.

1:38:30I have a few friends who are super senior engineers and one of their lines is like, it's very hard to predict where systems will break when you scale them. you can sort of try to predict an advance, but like there's always something weird that's gonna happen when you add this extra to your end. You thought you thought through everything, but you didn't actually think through everything. But I think for that particular system,

1:38:59for concrete details, the thing we do is obviously we upload when we chunk up all of your code and then we send out sort of the code for embedding and we embed the code and then we store the embeddings in a database but we don't actually store any of the code. And then there's reasons around making sure that we don't introduce client bugs because we're very, very paranoid about client bugs. We store much of the details on the server like everything is sort of encrypted. So one of the technical challenges is always making sure that the local index, the local code base state, is the same as the state that is on the server.

1:39:44And the way, sort of technically we ended up doing that is, so for every single file, you can sort of keep this hash. And then for every folder, you can sort of keep a hash, which is the hash of all of its children, and you can sort of recursively do that until the top. And why do something complicated? One thing you could do is you could keep a hash for every file. And every minute you could try to download the hashes that are on the server, figure out what are the files that don't exist on the server, maybe just created a new file, maybe just deleted a file, maybe you checked out a new branch, and try to reconcile the state between the client and the server.

1:40:23But that introduces like absolutely ginormous network overhead. both on the client side, I mean, nobody really wants us to hammer their Wi -Fi all the time if you're using cursor. But also, I mean, it would introduce like, ginormous overhead on the database. It would sort of be reading this, a tens of terabyte database, sort of approaching like 20 terabytes or something database like every second. That's just, just, kind of crazy. You definitely don't want to do that. So what did you do? You just try to reconcile the single hash, which is at the root of the project. And then if something mismatches, then you go, you find where all the things disagree.

1:41:05Maybe you look at the children and see if the hash is matched and if the hash is still matched, go, look at their children and so on. But you only do that in the scenario where things don't match. And for most people, most of the time, the hash is matched. So it's a kind of like hierarchical reconciliation. Yeah, something like that. Yeah, it's called the Merkel's free. Yeah, yeah, I mean, so yeah, this is cool to see that you kind of have to think through all these problems And I mean the the point of like the reason it's gone hard is just because like the number of people using it and you know Some of your customers have really really large code bases to the point where We you know, we originally wrote a dark code base, which is which is big But I mean just just not the size of some Company that's been there for 20 years and sort of has a enormous number of files and You sort of want to scale that across programmers.

1:41:54There's all these details where building a simple thing is easy, but scaling it to a lot of people, like a lot of companies is obviously a difficult problem, which is sort of independent of actually. So there's part of the scaling our current solution is also coming up with new ideas that obviously we're working on. But then scaling all of that in the last few weeks, months. Yeah. And there are a lot of clever things, like additional things that go into this indexing system. For example, the bottleneck in terms of costs is not soaring things in the vector database. The database is actually embedding the code.

1:42:27You don't want to re -embed the code base for every single person in a company that is using the same exact code, except for maybe there are different branches with a few different files or they've made a few local changes. Because embeddings are the bottleneck you can do is one clever trick and not have to worry about the complexity of dealing with branches in the other databases where you just have some cash on the actual vectors computed from the hash of a given chunk. So this means that when the end person that a company goes and events their code base, it's really, really fast. You do all this without actually storing any code on our servers at all.

1:43:06No code data stored. We just store the vectors in the vector database and the vector cash. What's the biggest gains at this time you get from indexing the code base? I could just out of curiosity like what benefit users have. It seems like longer term, there'll be more and more benefit, but in the short term, just asking questions of the code base, what's the usefulness of that? I think the most obvious one is just you want to find out where something is happening in your large codebase. And you sort of have a fuzzy memory of, okay, I want to find the place where we do X. But you don't exactly know what to search for in a normal text search.

1:43:49And so you ask a chat, you hit command enter to ask with the codebase chat, and then very often it finds the right place that you were thinking of. I think like you mentioned, in the future, I think there's only going to get more and more powerful, where we're working a lot on improving the quality or retrieval. And I think the ceiling for that is really, really much higher than people give a credit for. One question that's good to ask here, have you considered and why having you much done sort of local stuff to where you can do the, and it seems like everything we've just discussed is exceptionally difficult to do, to go to the cloud yet to think about all these things with the caching and the large code base with a large number of programmers that are using the same code base.

1:44:33You have to figure out the puzzle of that. A lot of but most software just does stuff, this heavy computational stuff locally. Have you considered doing sort of embeddings locally? Yeah, we thought about it. And I think it would be cool to do it locally. I think it's just really hard. And one thing to keep in mind is that, you know, some of our users use the latest MacBook Pro. And but most of our users, like more than 80 % of our users are in Windows machines, which, and many of them are not very powerful. and so local models really only works on the latest computers. And it's also a big overhead to build that in.

1:45:14And so even if we would like to do that, it's currently not something that we are able to focus on. And I think there are some people that do that, and I think that's great. But especially as models get bigger and bigger, and you want to do fancier things with bigger models, it becomes even harder to do locally. Yeah, and it's not a problem with like weaker computers. It's just that for example if you're some big company Have big company code base. It's just really hard to process big company code base even on the beefiest macro pros Yeah, so even if it's not even a matter of like if you're if you're just like a student or something I think If you're like the best programmer at a big company, you're still gonna have a horrible experience If you do everything locally, you could do Edge and scrape by, but again, it wouldn't be fun anymore.

1:46:07Yeah, like at approximate nearest neighbors on this massive code base, it's going to just eat up your memory and your CPU. And that's just that. Let's talk about also the modeling side where, as Arvets said, there are these massive headwinds against local models where one things that seems to move towards MOEs, which, like one benefit is maybe there are more memory bandwidth bound, which plays in favor of local versus using GPUs or using Nvidia GPUs. But the downside is these models are just bigger and total. And, you know, they're going to need to fit, often not even on a single node, but multiple nodes.

1:46:48There's no way that's going to fit inside of even really good MacBooks. And I think especially for coding, it's not a question as much of like does it clear some bar of like the models good enough to do these things and then like we're satisfied, which may be the case for other other problems and maybe we're local models shine, but people are always going to want the best, the most intelligent, the most capable things. And that's going to be really, really hard to run for almost all people locally. Don't you want the most capable model? You want Sonnet, you? And also with O - I like how you're pitching me.

1:47:26Would you be satisfied with an inferior model? I'm one of those, but there's some people that like to do stuff locally, especially like. Yeah. There's a whole obviously open source movement that kind of resists, and it's good that they exist actually, because you want to resist the power centers that are growing are. There's actually an alternative to local models that I am particularly fond of. I think it's still very much in the research stage, but you could imagine to do homomorphic encryption for language model inference. So you encrypt your input on your local machine, then you send that up, and then the server can use loss of computation.

1:48:09They can run models that you cannot run locally on on this encrypted data, but they cannot see what the data is. And then they send back the answer, and you decrypt the answer, and only you can see the answer. So I think that's still very much research, and all of it is about trying to make the overhead lower, because right now the overhead is really big. But if you can make that happen, I think that would be really, really cool, and I think it would be really, really impactful. Because I think one thing that's actually kind of worrisome is that as these models get better and better, they're going to become more and more economically useful.

1:48:44And so more and more of the world's information and data will flow through one or two centralized actors. And then there are worries about, there can be traditional hacker attempts, but it also creates this kind of scary part where if all of the world's information is flowing through one node in plaintext, you can have surveillance in very bad ways. and sometimes that will happen for, you know, initially will be like good reasons, like people will want to try to protect against like bad actors using AI models in bad ways. And then you will add in some surveillance code and then someone else will come in and you know you're in a slippery slope and then you start doing bad things with a lot of the world's data.

1:49:33And so I'm very hopeful that we can solve homeomorphic encryption for like, doing privacy preserving machine learning. But I would say that's the challenge we have with all software these days. It's like there's so many features that can be provided from the cloud and all this increasingly reliant it and make our life awesome. But there's downsides and that's why you rely on really good security to protect from basic attacks. But there's also only a small set of companies that are controlling that data. They obviously have leverage and they could be infiltrated in all kinds of ways. That's the world we live in.

1:50:10That's so. Yeah, I mean, the thing I'm just actually quite worried about is sort of the world where, I mean, the entropic has this responsible scaling policy and where we're on like the low ESLs, which is the entropic security level or whatever, of like of the models, but as we get you like, couldn't go to ESL, three ESL, four, whatever models, which are sort of very powerful. But for mostly reasonable security reasons, you would want to monitor all the prompts. But I think that's sort of reasonable and understandable where everyone is coming from, but Matt, it'd be really horrible if sort of like all the world's information is sort of monitored that heavily.

1:50:51It's way too centralized. It's like sort of this really fine line you're walking where on the one side, like you don't want the models to go rogue on the other side, like humans, like, I don't know if I trust like all the world's information to pass through like three model providers. Yeah. What do you think it's different than cloud providers? Because I think this is a lot of this data would never have gone to the cloud providers in the first place, where this is often like you want to give more data to the EI models. you want to give personal data that you would never have put online in the first place to these companies or to these models.

1:51:37And it also centralizes control where right now for cloud, you can often use your own encryption keys and it like AWS can't really do much. But here is just centralized actors that see the exact plaintext of everything. On the topic of context, that's actually been a friction for me. When I'm writing code in Python, there's a bunch of stuff imported. You could probably intuit the kind of stuff I would like to include in the context. Is there, how hard is it to auto figure out the context? It's tricky. I think we can do a lot better at computing the context automatically in the future. One thing that's important to note is there are trade -offs with including automatic context.

1:52:29So the more context you include for these models, first of all the slower they are. And the more expensive those requests are, which means you can then do less model calls and do less fancy stuff in the background. Also for a lot of these models they get confused if you have a lot of information in the prompt. So the bar for accuracy and for relevance of the context you include should be quite high. But this is already we do some automatic context in some places within the product. It's definitely something we want to get a lot better at. And I think that there are a lot of cool ideas to try there.

1:53:07Both on the learning better retrieval systems like better and betting models, better re -rankers. I think that there are also cool academic ideas. You know, stuff we've tried out internally, but also the field is grappling with writ large about, can you get language models to a place where you can actually just have the model itself understand a new corpus of information. And the most popular talks about version of this is can you make the context when it's infinite? Then if you make the context when it's infinite, can you make the model actually pay attention to the infinite context? And then after you can make it pay attention to the infinite context, to make it somewhat feasible to actually do it, can you then do caching for that infinite context?

1:53:45You don't have to recompute that all the time. But there are other cool ideas that are being tried that are a little bit more analogous to fine tuning, of actually learning this information and the weights of the model. And it might be that you actually get sort of a qualitative the different type of understanding if you do it more at the weight level, then if you do it at the in -contetics learning level, I think the jury's still a little bit out on how this is all gonna work in the end. But in the interim us as a company, we are really excited about better retrieval systems and picking the parts of the code base that are most relevant to what you're doing.

1:54:17We could do that a lot better. Like one interesting proof of concept for the learning this knowledge directly in the weights is with VS code. So we're in a VS code fork and VS code. The code is all public. So these models in pre -training have seen all the code. They've probably also seen questions and answers about it. And then they've been fine tuned and early shift. You've been able to be able to answer questions about code in general. So when you ask a question about VS code, sometimes it'll hallucinate, but sometimes it actually does a pretty good job at answering the question. And I think this is just by, it happens to be okay to it, but what if you could actually like specifically train or post train a model such that it really was built to understand this code base?

1:55:04It's an open research question, one that we're quite interested in, and then there's also uncertainty of like do you want the model to be the thing that end to end is doing everything. IE, it's doing the retrieval and its internals and then kind of answering a question creating the code or do you want to separate the retrieval from the front -ear model where maybe you'll get some really capable models that are much better than like the best open source ones in a handful of months. And then you'll want to separately train a really good open source model to be the retriever, to be the thing that feeds in the context to these larger models.

1:55:40Can you speak a little more to the post training model to understand the code base? Like, what do you mean by that? Is this a synthetic data direction? Is this? Yeah, I mean, there are many possible ways you could try doing it. There's certainly no shortage of ideas. It's just a question of going in and trying all of them and being empirical about which one works best. You know, one very naive thing is to try to replicate what's done with the S code and these frontier models. So let's like continue pre -training. So I'm kind of continued pre -training that includes general code data, but also throws in a lot of the data of some particular repository that you care about.

1:56:19And then in post -training, meaning in let's just start with instruction fine tuning, you have like a normal instruction fine tuning data set about code, but you throw in a lot of questions about code in that repository. So you could either get ground truth ones, which might be difficult, or you could do what you kind of hinted at or suggested using synthetic data, i .e. kind of having the model ask questions about various pieces of the code. So you kind of take the pieces of the code, then prompt the model or have a model propose a question for that piece of code, and then add those as instruction files to new data points.

1:56:58And then in theory, this might unlock the model's ability to answer questions about that code base. Let me ask you about OpenAI 01. What do you think is the role of that kind of test time compute system in programming? I think test time compute is really, really interesting. There's been the pre -training regime, which will as you scale up the amount of data and the size of your model get you better and better performance, both on loss and then on downstream benchmarks and just general performance. So we use it for coding or other tasks. We're starting to hit a bit of a data wall, meaning it's going to be hard to continue scaling up this regime.

1:57:42And so scaling up 10 test time compute is an interesting way of now increasing the number of inference time flops that we use, but still getting, like yeah, as you increase the number of flops use inference time, getting corresponding improvements in the performance of these models. Traditionally, we just had to literally train a bigger model that always uses that always used that many morph lots, but now we could perhaps use the same size model and run it for longer to be able to get an answer at the quality of a much larger model. And so the really interesting thing I like about this is there are some problems that perhaps require 100 trillion parameter model intelligence trained on 100 trillion tokens.

1:58:23But that's like maybe 1%, maybe like 0 .1 % of all queries. So, are you going to spend all of this effort, all this compute train a model that costs that much and then run it so infrequently? It feels completely wasteful. When it's said you get the model that can, that is, that you train the model that's capable of doing the 99 .9 % of queries, then you have a way of inference time running it longer for those few people that really, really want max intelligence. How do you figure out which problem requires what level of intelligence? Is that possible to dynamic a figure out when to use GPT -4, when to use a small model and when you need the O1?

1:59:09I mean, yeah, that's an open research problem, certainly. I don't think anyone's actually cracked this model routing problem quite well. We'd like to, we have like kind of initial implementations of this for things, for something like cursor tab. But at the level of like, going between 4 -0, Sonnet to 01, it's a bit trickier. Perhaps, there's also questions like what level of intelligence do you need to determine if the thing is too hard for the 4 -level model? Maybe you need the 01 -level model. It's really unclear. But you mentioned that there's a pre -training process, then there's post -training, and there's like test -time compute, and that's fair to sort of separate.

1:59:55Where's the biggest gains? Well, it's weird, because like test -time compute, there's like a whole training strategy needed to get test -time to compute to work. And the other really weird thing about this is, like outside of the big labs, and maybe even just open AI, no one really knows how it works. Like there have been some really interesting papers that show hints of what they might be doing. And so perhaps they're doing something with tree search using process reward models. But yeah, I think the issue is we don't quite know exactly what it looks like. So it would be hard to kind of comment on where it fits in.

2:00:34I would put it in post training, but maybe the compute spent for this kind of, for getting test time compute to work for a model is going to dwarf pre -training eventually. We don't even know if O1 is using just like chain of thought, RL, we don't know how they're using any of these. I don't know anything. It's fun to speculate. If you were to build a competing model, what would you do? One thing to do would be, I think you probably need to train a process reward model, which which is, so maybe we can get into reward models and outcome reward models versus process reward models. Outcome reward models are kind of traditional reward models that people are trained for these four language models, language modeling.

2:01:20And it's just looking at the final thing. So if you're doing some math problem, let's look at that final thing you've done, everything, and let's assign a great to it, how likely we think, like what's the reward for this outcome. Process reward models instead try to grade the chain of thought. And so OpenAI had some preliminary paper on this, I think, last summer, where they use human lablers to get this pretty large, several hundred thousand data set of creating chains of thought. Ultimately, it feels like I haven't seen anything interesting in the ways that people use process reward models outside of just using it as a means of affecting how we choose between a bunch of samples.

2:02:02So like what people do in all these papers is they sample a bunch of outputs from the language model and then use the process reward models to grade all those generations alongside maybe some other heuristics and then use that to choose the best answer. The really interesting thing that people think might work and people want to work is tree search with these process reward models because if you really can grade every single step of the chain of thought, then you can kind of branch out. and explore multiple paths of this chain of thought, and then use these process word models to evaluate how good is this branch that you're taking?

2:02:40Yeah, when the quality of the branch is somehow strongly correlated with the quality of the outcome at the very end, so like you have a good model of knowing which branch you take. So not just in the short term, like in the long term. Yeah. And like the interesting work that I think has been done is figuring out how to properly train the process, or the interesting work that has been open source and people I think talk about is how to train the process reward models maybe in a more automated way. I could be wrong here, could not be mentioning some people, but I haven't seen anything super that seems to work really well for using the process reward models creatively to do tree search and code.

2:03:18This is kind of an AI safety maybe a bit of a philosophy question. So open AI says that they're hiding the chain of thought from the user and they've said that that was a difficult decision to make, they instead of showing the chain of thought, they're asking the model to summarize the chain of thought. They're also in the background saying they're going to monitor the chain of thought to make sure the model is not trying to manipulate the user, which is a fascinating possibility. But anyway, what do you think about hiding the chain of thought? One consideration for OpenAI, and this is completely speculative, could be that they want to make it hard for people to distill these capabilities out of their model.

2:03:55it might actually be easier if you had access to that hidden chain of thought to replicate the technology. Because that's pretty important data, like seeing the steps that the model took to get to the final result. So you could probably train on that also. And there was sort of a mirror situation with this, with some of the large language model providers, and also this speculation, but some of these APIs used to offer easy access to log probabilities for the tokens that they're generating. and also log probabilities of the prompt tokens. And then some of these APIs took those away. And again, complete speculation, but one of the thoughts is that the reason those were taken away is if you have access log probabilities similar to this, hidden training thought, that can give you even more information to try and distill these capabilities out of the APIs, out of these biggest models, into models you control.

2:04:46As an asterisk on also the previous discussion about us integrating a one, I think that we're still learning how to use this model. So we made O1 available in cursor because like we were when we got the model we were really interested in trying it out. I think a lot of programmers are going to be interested in trying it out. But O1 is not part of the default cursor experience in any way up. And we still haven't found a way to get integrated into an editor into the editor in a way that we reach for sort of every hour, maybe even every day. And so I think that the jury's still out on how to use the model.

2:05:28And we haven't seen examples yet of people releasing things where it seems really clear. Like, oh, that's like now the use case. The obvious one to turn to is maybe this can make it easier for you to have these background things running, right? To have these models and loops, to have these models be a gen tech. But we're still discovering. Should we clear we have ideas? Yes, we just need to try and get something incredibly useful before we put it out there. But it has these significant limitations. Even like barring capabilities, it does not stream. And that means it's really, really painful to use for things where you want to supervise the output and instead you're just waiting for the wall text to show up.

2:06:13Also it does feel like the early innings of test time computing search where it's just like a very, very much of VZero. And there's so many things that like don't feel quite right. And I suspect in parallel to people increasing the amount of pre -training data and besides the models in pre -training and finding for Xter, you'll now have this other thread of getting searched to work better and better. So let me ask you about strawberry tomorrow eyes. So it looks like GitHub, Copilot might be integrating a one in some kind of way. And I think some of the comments are saying, does this mean cursor is done?

2:06:59I think I saw one comment saying that. I saw it time to shut down cursor. Time to shut down cursor. Thank you. So it's the time to shut down cursor. I think this space is a little bit different from past software spaces over the 20 times where I think that the ceiling here is really really really incredibly high. And so I think that the best product in three to four years will just be soon much more useful than the best product today and you can like wax poetic about Motes this and brand that and you know, this is our advantage But I think in the end just if you don't have like if you stop innovating on the product you will you will lose And that's also great for startups that's great for people trying to enter this market because it means you have an opportunity to win against people who have you know, lots of users already by just building something better.

2:07:50And so I think yeah, over the next few years it's just about building the best product, building the best system and that both comes down to the modeling engine side of things and it also comes down to the editing experience. Yeah, I think most of the additional value from cursor versus everything else out there is not just integrating the new model fast like a one. It comes from all of the kind of depth that goes into these custom models that you don't realize are working for you in kind of every facet of the product, as well as like the really thoughtful UX with every single feature. All right, from that profound answer, let's descend back down to the technical.

2:08:31You mentioned and geotoxonomy of synthetic data. Oh yeah. Can you please explain? Yeah, I think there are three main kinds of synthetic data. The first, so what is synthetic data first? So there's normal data, like non -synthetic data, which is just data that's naturally created. I .e. usually it'll be from humans having done things. So from some human process, you get this data. Synthetic data, the first one would be distillation. So having a language model, kind of output tokens or probability distributions over tokens. And then you can train some less capable model on this. This approach is not going to get you a net like more capable model than the original one that has produced the tokens.

2:09:18But it's really useful for if there's some capability you want to elicit from some really expensive high latency model. You can then distill that down into some smaller task -specific model. The second kind is when one direction of the problem is easier than the reverse. A great example of this is bug detection, like we mentioned earlier, where it's a lot easier to introduce reasonable looking bugs than it is to actually detect them. This is probably a case for humans, too. So what you can do is you can get a model that's not training that much data, that's not that smart to introduce a bunch of code and then you can use that to then train, use a synthetic data to train a model that can be really good at detecting bugs.

2:10:08The last category I think is, I guess, the main one that feels like the big labs are doing for synthetic data, which is producing text with language models that can then be verified easily. So like, you know, an extreme example of this is if you have a verification system that can detect if language is Shakespeare level and any of a bunch of monkeys typing in typewriters, like you can eventually get enough training data to train a Shakespeare level language model. And I mean, this is the case, like very much the case for math where verification is is is actually really, really easy for formal formal languages.

2:10:48And then And what you can do is you can have an OK model generated a ton of rollouts and then choose the ones that you know have actually proved the grand truth here. There are some trained that further. There's similar things you can do for code with lead code like problems or where if you have some set of test that you know correspond to. If something passes these tests, it is actually solved a problem. You can do the same thing where you verify that it's passed the test and then train the model and the outputs that have passed the test. I think it's going to be a little tricky getting this to work in all domains or just in general.

2:11:24Having the perfect verifier feels really, really hard to do with just open -ended, miscellaneous tasks, you get the model or more long horizon tasks, even encoding. That's because you're not as optimistic as Arvid, but yeah. So that third category of a choir is having a verifier. Yeah, verification is it feels like it's best when you know for a fact that it's correct and like then it like it wouldn't be like using a language model to verify it'd be using tests or formal systems or running the thing to doing like the human form of verification where you just do manual quality control. Yeah, yeah, but like the language model version of that where it's like running the thing and it's actually understand that.

2:12:05Yeah, no, that's for somewhere between. Yeah, yeah. I think that's the category that is most likely to result in like massive gains. What about RL with feedback side RLHF versus RLAIF? What's the role of that in getting better performance on the models? Yeah, so RLHF is when the reward model you use is trained from some labels you've collected from humans getting feedback. I think this works if you have the ability to get a ton of human feedback for this kind of task that you care about. RL -A -I -F is interesting because you're kind of depending on, like this is actually kind of going to, it's depending on the constraint that verification is actually a decent bit easier than generation.

2:13:03Because it feels like, okay, like what are you doing? You're using this language model to look at the language model outputs and then prove the language model. But no, it actually may work if the language model has a much easier time verifying some solution then it does generating it. Then you actually could perhaps get this kind of recursive, but I don't think it's gonna look exactly like that. The other thing you could do is, that we kind of do is like a little bit of a mix of RLEIF and RLEChef, where usually the model is actually quite correct, and this is in the case of cursor tap, picking between like two possible generations of what is the better one.

2:13:41And then it just needs like a hand, a little bit of human nudging with only like on the order of 50, 100 examples to like kind of align that prior the model has with exactly what you want. It looks different than I think normal RHR -LHF or usually training these reward models and tons of examples. What's your intuition when you compare generation and verification and generation and ranking? Is ranking way easier in generation? my intuition would just say yeah it should be like this is kind of going back to like if you use if you believe p does not equal np then there's this massive class of problems that are much much easier to verify given a proof than actually proving it.

2:14:29I wonder if the same thing will prove p not equal to np or p equal to np. That would be that would be really cool. That be of whatever feels metal by AI who gets the credit. No, they're open philosophical question. I'm actually surprised. I'm actually surprisingly curious what what what like a good bat for one one AI will get the fields metal will be actually don't have a lot of special. I don't know what a monster here is. Sorry, Nobel Prize or feels metal first. Feels metal. Well, feels metal level. Geals metal needs to solve. I think it feels metal comes first while you would say that of course It's also this like isolated system.

2:15:12You know if I and yeah sure like I don't even know if I don't need to do I feel like I have much more to do there I felt like the path to get to IMO was a little bit more clear because it already could get a few IMO problems And there are a bunch of like there's a bunch of low -hing fruit given the literature at the time of like what what tactics people could take I think I'm one much less first in the space of deer improving now and to Yeah, less intuition about how close we are to solving these really, really hard open problems. So you think it'll be feels my first it won't be like in physics or in...

2:15:46Oh, 100%. I think that's probably what we're likely to be. Like it's probably much more likely that it'll get then... Yeah, well I think it puts the like I don't know like BSD, which is a birth, marriage and diet conductor, like we want iPods or any one of these like hard, hard math problem I'm sure it's actually really hard. It's sort of unclear what to pass you to get even a solution looks like. Like we don't even know what a path looks like, let alone. And you don't buy the idea. This is like an isolated system and you can actually have a good reward system and it feels like it's easier to train for that.

2:16:22I think we might get fields metal before AGI. I mean, I'm be very happy. I'm very happy. But I don't know if I think 20, 20, 20, 30, 20, feels metal. Feels metal. All right. It feels like forever from now, given how fast things have been going. Speaking of how fast things have been going, let's talk about scaling laws. So for people who don't know, maybe it's good to talk about this whole idea of scaling laws. What are they? Where do you think stand? And where do you think things are going? It was interesting, the original scaling loss paper by OpenAI was slightly wrong because I think of some issues they did with learning rate schedules.

2:17:09And then Chinchilla showed a more correct version. And then from then, people have again deviated from doing the compute optimal thing because people start now optimizing more so for making the thing work really well, given an inference budget. it. And I think there are a lot more dimensions to these curves than what we originally used of just compute number of parameters and data. Like inference compute is the obvious one. I think context length is another obvious one. So if you care, like let's say you care about the two things of inference compute and then context window, maybe the thing you want to train is some kind of SSM because they're much, much cheaper and faster at super, super long contacts.

2:17:55And even if maybe it is 10x worth scaling properties during training, many have spent 10x more compute to train the thing to get the same level of capabilities. It's worth it because you care most about that inference budget for really long context windows. So it'll be interesting to see how people kind of play with all these dimensions. So yeah, let me speak to the multiple dimensions. Obviously, the original conception was still can't the variables of the size of the model as measured by parameters and the size of the data is measured by the number of tokens and looking at the ratio of the two.

2:18:26And it's kind of a compelling notion that there is a number or at least a minimum. And it seems like one was emerging. Do you still believe that there is a kind of bigger is better? I mean, I think bigger is certainly better for just raw performance and raw intelligence. I think the path that people might take is I'm particularly bullish on distillation. And like, yeah, how many knobs can you turn to if we spend like a ton, ton of money on training, like get the most capable cheap model, right? Like really, really caring as much as you can. Because the naive version of carrying as much as you can about inference time computers, what people have already done with the Lama models are just overtraining the shit out of 7B models on way, way, way more tokens than is central obstacle.

2:19:20But if you really care about it, maybe you think to do is what Gema did, which is, let's not just train on tokens. Let's literally train on minimizing the KL divergence with the distribution of Gema 27B. So, knowledge distillation there. And you're spending the compute of literally training this 27 billion model, a billion parameter model on all these tokens, just to get out this, I don't know, smaller model. And the distillation gives you just a faster model, smaller means faster. Yeah, distillation and theory is, I think, getting out more signal from the data that you're training on. And it's like another, it's perhaps another way of getting over, not completely over, but partially helping with the data wall, where you only have so much data to train on, let's like train this really, really big model on all these tokens and we'll distill it into this smaller one.

2:20:10And maybe we can get more signal per token for this much smaller model than we would have personally trained it. So if I gave you $10 trillion, how would you spend it? I mean, you can't buy an island or whatever. How would you allocate it in terms of improving the big model versus maybe pay for HF in the RLHF or... Yeah, I think there's a lot of these secrets and details about training these large models that I just don't know and only privy to the large labs and the issue is I would waste a lot of that money if I even attempted this because I wouldn't know those things. Suspending a lot of disbelief and assuming like you had the know -how and offer or if you're saying you have to operate with the limited information you have now.

2:21:04No, no, no, actually, I would say you swoop in and you get all the information, all the little hewist, all the little parameters, all the parameters that define how the thing is trained. If we look in how to invest money for the next five years in terms of maximizing what you call raw intelligence, I mean, isn't the answer really simple? You just try to get as much compute as possible. Like at the end of the day, all you need to buy is the GPUs and then the researchers can find find all the like they can sort of you can tune whether you want to be trained a big model or a small model like. Well, this gets me to the question of like, are you really limited by compute and money or are you limited by these other things?

2:21:47And I'm more privy to Arvid's beliefs that were sort of idea limited but There's always it. But if you have a lot of compute, you can run a lot of experiments. So you would run a lot of experiments versus use that computer -trained gigantic model. I would, but I do believe that we are limited in terms of ideas that we have. I think, yeah, because even with all this compute and all the data you could collect in the world, I think you really are ultimately limited by not even ideas, but just like really good engineering. Like even with all the capital in the world, would you really be able to assemble?

2:22:29Like there aren't that many people in the world who really make the difference here. And there's so much work that goes into research that is just like pure, really, really hard engineering work. That is like a very kind of hand -wavy example if you look at the original Transformer paper. You know how much work was kind of joining together are a lot of these really interesting concepts embedded in the literature versus then going in and writing all the code, like maybe the Cuda kernels, maybe whatever else. I don't know if it ran in GPUs or TPs originally, so I said it actually saturated a GPU performance, right?

2:23:04Getting no more to go in and do all this code, right? And no, it's like probably one of the best engineers in the world, or maybe going a step further like the next generation of models, having these things, like getting model parallelism to work and scaling it on like thousands of, of maybe tens of thousands of V100s, which I think GBD E3 may have been. There's just so much engineering effort that has to go into all these things to make it work. If you really brought that cost down to, maybe not zero, but just made it 10x easier, made it super easy for someone with really fantastic ideas to immediately get to the version of the new architecture they dreamed up that is like getting 50, 40 % utilization on the GPUs.

2:23:49I think that would just speed up research by a ton. I mean, I think if you see a clear pastime improvement, you should always sort of take the low hanging fruit first, right? And I think probably open the eye and and I'll do other labs that did the right thing to pick off the low hanging fruit. where the low hanging fruit is like sort of, you could scale up to a GPT 4 .25 scale. And you just keep scaling and things keep getting better. And as long as, there's no point of experimenting with new ideas when everything is working. And you should sort of bang on it and try to get as much juice out of the possible.

2:24:30And then maybe when you really need new ideas for, I think if you're spending $10 trillion, probably want to spend some, then actually re -evaluate your ideas. Like, probably your idea limited at that point. I think all of us believe new ideas are probably needed to get all the way there to AGI. And all of us also probably believe there exist ways of testing out those ideas at smaller skills and being fairly confident that they'll play out. It's just quite difficult for the labs in their current position to dedicate their very limited research and engineering talent to exploring all these other ideas when there's this core thing that will probably improve performance for some like these amount of time.

2:25:22Yeah, but also these big labs like winning. They're just going wild. Okay. So how big question looking out into the future? You're now at the center of the programming world. How do you think programming, the nature programming changes in the next few months, in the next year, in the next two years, next five years, ten years. I think we're really excited about a future where the programmers in the driver's seat for a long time. And you've heard us talk about this a little bit, but one that emphasizes speed and agency for the programmer and control the ability to modify anything you want to modify the ability to iterate really fast on what you're building.

2:26:09And this is a little different, I think, than where some people are jumping to in the space. Where I think one idea that's captivated people is, can you talk to your computer? Can you have it built off for you? As if you're talking to an engineering department or an engineer over Slack. And can I just be this sort of isolated text box? And part of the reason we're not excited about that is some of the stuff we talked about with latency. But then a big piece of reason we're not excited about that is because that comes with giving up a lot of control. It's much harder to be really specific when you're talking in the text box.

2:26:49And if you're necessarily just going to communicate with a thing like you would be communicating with an engineering department, you're actually advocating tons of tons of really important decisions to this bot. And this kind of gets at fundamentally what engineering is. I think that some people who are a little bit more removed from engineering might think of it as, you know, the spec is completely written out. And then the engineers just come and they just implement. And it's just about making the thing happen in code and making the thing exist. But I think a lot of the best engineering, the engineering we enjoy, involves tons of tiny micro decisions about what exactly you're building and about really hard trade -offs between speed and cost and just all the other things involved in a system.

2:27:36And as long as humans are actually the ones designing the software and the ones specifying what they want to be built, and it's not just like company run by all AI's, we think you'll really want the human in a driver's seat dictating these decisions. And so there's the jury still out on kind of what that looks like. I think that one weird idea for what that could look like is it could look like you can control the level of abstraction you view a code base at. And you can point at specific parts of a code base that maybe you digest a code base by looking at it in the form of pseudocode. And you can actually edit that Cedar Code 2, and then have changes get me down at the sort of formal programming level.

2:28:23And you keep the like, you know, you can gesture at any piece of logic in your software component of programming. You keep the inflow text editing component of programming. You keep the control of, you can even go down into the code, you can go at higher levels of abstraction. Well, also giving you these big productivity gains. It'll be nice if you can go up and down on the abstraction stack. Yeah. And there are a lot of details to figure out there. That's sort of like a fuzzy idea. Time will tell if it actually works. But these principles of control and speed in the human and the driver's seat, we think are really important.

2:28:54We think for some things, like Arbit mentioned before, for some styles of programming, you can kind of hand it off chatbot style. If you have a bug that's really well specified, but that's not most of programming, and that's also not most of the programming. We think a lot of people value. What about the fundamental skill of programming? There's a lot of people like young people right now kind of scared like thinking because they like love programming but they're scared about like will I be able to have a future if I pursue this career path. Do you think the very skill of programming will change fundamentally?

2:29:30I actually think this is a really really exciting time to be building software. Like we remember what programming was like in you know 2013 2012 whatever it was. And there was just so much more craft and boilerplate and, you know, looking up something really gnarly. And you know, that stuff's still like this. It's definitely not at Ed's hero. But programming is a way more fun than back then. It's like we're really getting down to the the delight concentration. And all of the things that really draw people to programming, like for instance, this element of being able to build things really fast and speed and also their individual control, like all those are just being turned up a ton.

2:30:15And so I think it's just going to be, I think it's going to be a really, really fun time for people who build software. I think that the skills will probably change too. I think that people's taste and creative ideas will be magnified and it will be less about, maybe less a little bit about boilerplate text editing, maybe even a little bit less about carefulness, which I think is really important today. Sure a programmer, I think it'll be a lot more fun. What do you guys think? I agree. I'm very excited to be able to change. Like just, one thing that happened recently was like we wanted to do a relatively big migration to our code base.

2:30:53We were using async local storage in Node .js, which is known to be not very performant and we wanted to migrate to our context object. And this is a big migration in effect, the entire code base. And so all of a sudden I spent, I don't know, five days working through this, even with today's AI tools. And I am really excited for a future where I can just show a couple of examples and then the AI applies that to all of the locations and then it highlights, oh, this is a new example, like what should I do and then I show exactly what to do there. And then that can be done in like 10 minutes. And then you can iterate much, much faster than you can, then you don't have to think as much up front and stay it stand at the blackboard and think exactly, how are we gonna do this because it costs us so high, but you can just try something first, and you realize, oh, this is not actually exactly what I want, and then you can change it instantly again after.

2:31:47And so, yeah, I think being a programmer in the future is going to be a lot of fun. Yeah. I really like that point about, it feels like a lot of the time with programming, there are two ways you can go about it. One is, you think really hard, carefully, upfront about the best possible way to do it, and then you spend your limited time of engineering to actually implement it. But I must refer just getting in the code and taking a crack at it, seeing how it lays out, and then iterating really quickly on that. That feels more fun. Yeah, just speaking to generating the boilerplate is great. So you just focus on the difficult design, nuanced, difficult design decisions.

2:32:30migration. I feel like this is this is a cool one. Like it seems like large language models able to basically translate for one program language or another or like translate like migrate in the general sense of what migrate is. But that's in the current moment. So I mean that the fear has to do with like, okay, as these models get better and better, then you're doing less and less creative decisions and is it going to kind of move to a place where it's you're operating in the design space of natural language or natural language is the main programming language. And I guess I get asked that by way of advice.

2:33:05Like if somebody's interested in programming now, what do you think they should learn? Like to say, you guys started some Java and I forget the, oh, some PHP. Objective C. Objective C, there you go. I mean, in the end, we all know JavaScript script is going to win. And not TypeScript is going to be like vanilla JavaScript. It's going to eat the world and maybe a little bit of PHP. And I mean, it also brings up the question of like, I think Don Knuth has a, this idea that some percent of the population is geeks. And like, there's a particular kind of psychology in mind required for programming.

2:33:47And it feels like more and more that expands the kind of person that should be able to can do great programming might expand. I think different people do programming for different reasons, but I think the true maybe like the best programmers are the ones that really love just like absolutely love programming, for example, they're folks in our team who literally when they get back from work, they go and then they boot up cursor and then they start coding on their side projects for the entire night and they say, oh, so 3am doing that. And when they're sad, they said, I just really need to code. I think there's that level of programmer where the succession and love of programming I think makes really the best programmers and I think these types of people will really get into the details of how things work.

2:34:55I guess the question I'm asking, that exact program, I'll think about that person. When the super tab, the super awesome praise be the tab is succeeds. And you keep pressing tab. That person in the team loves to curse the tab more than anybody else. And it's also not just like pressing tab is like the just pressing tab. That's like the easy way to say it and the catch phrase. But what you're actually doing when you're pressing tab is that you're injecting intent all the time while you're doing it, sometimes you're rejecting it, sometimes you're typing a few more characters. And that's the way that you're sort of shaping the things that's being created.

2:35:38And I think programming will change a lot to just what is it that you want to make? It's sort of higher bandwidth. The communication to the computer just becomes higher and higher bandwidth as the person you like, just typing is much lower bandwidth then communicating in Todd. I mean, this goes to your manifesto titled Engineering Genius. We are an applied research lab building extraordinary productive human AI systems. So let's speak into this like hybrid element. To start, we're building the engineer of the future, a human AI programmer that's an order of magnitude more effective than any one engineer.

2:36:16This hybrid engineer will have effortless control over their code base and no low entropy keystrokes. They will iterate at the speed of their judgment, even in the most complex systems. Using a combination of AI and human ingenuity, they will outsmart and outengineer the best pure AI systems. We are a group of researchers and engineers, we build software and models to invent at the edge of what's useful and what's possible, our work has already improved the lives of hundreds of thousands of programmers. And on the way to that, that will at least make programming more fun. So thank you for talking today.

2:36:53Thank you. Thanks for having us. Thank you. Thank you. Thanks for listening to this conversation with Michael, Swale, Arvid, and I'm on to support this podcast. Please check out our sponsors in the description. And now let me leave you with random, funny, and perhaps profound programming code I saw and read it. Nothing is as permanent as a temporary solution that works. Thanks for listening and hope to see you next time.