VS Code and Agentic Development with Kai Maetzel

30:03You see every single detail of this. So you can actually verify my words here. So in your own day-to-day experience. So the current state is we actually have specific prompt for, I would say, roughly every model family, sometimes more detailed. Sometimes we go down and say, oh, it's not just a GPT model. We really separate between GPT-5, GPT-5 codecs, GPT-5-1 codecs. We move so fast that I say 5 rather than 5-1 as an industry. So different entry points pretty much. in our main profile generation, right? And then we'll pretty much pick what tools are available for that particular model, what are additional instructions that need to be given and so on.

30:53We customize the instructions that are outside of the actual tool descriptions. But we don't have a model, we don't have it yet in code where we actually say, no, we know exactly that this is the tool description that works better in this particular kind of model. That's actually something that we discussed several times, never quite made it to the point of, yeah, now it's coming. The last iteration plan, we again had the same conversation. And I was saying, oh, we do this right when we have shipped at the beginning of December. We'll have model-specific tool descriptions, right, where pretty much the prompt file can overwrite and saying, oh, if this tool shows up, right, And here's actually the tool description that you should use.

31:43So another kind of detailed topic in here is how you assemble context for the agent in terms of you're operating in this type of repo. You have these things beyond tools. There's people who do different amounts of like pre-injection of context. Maybe it's not just system prompt, but it's got a whole bunch of additional things. how do you think about the right ways to sort of present things to the agent so it kind of starts out going the right direction versus everything's in on-demand tool calling so there are a couple of things here that and let me actually start with the tools first right so most models these days have been trained with a particular tool set right so out of the box they already know a certain set of tools, like for example, apply patch, right, for GPT models, string replace for sonnet models, those kinds of things that they are must have, right, in those individual tools.

32:46And then the next question is beyond this, how well do models actually generalize, right? And so you will, you want the tools and then again, right, the context, as I said, in what kind of environment is that particular prompt now executing, right? Is that foreground agents, background agents, and so on? So that's the first kind of question, right? Those tools, how are they actually represented in your prompt? Then the next one is, let's say you have a couple of MCP servers installed. So most of, some MCP servers have only one or two tools, right? But others come in with dozens and dozens of tools.

33:31Most models have a limit how many tools you can actually put in a prompt, so 128. But then on top of this, there's a lot of tokens that you actually put in there. So how many tokens do you actually want to spend on tools that are rarely used or only in particular specific situations are being used? So a technique we're using there is, like for example we go and take all of the tools that an mcp server gives us and we actually now create pretty much virtual categories of tools right and in these kinds of virtual categories they are represented as tools in their own right we give this to the model at the very moment the model decides to call one of those virtual tools then we pretty much expand it so but now you have immediately this kind of trade-off discussion, which is the very moment you do this, you actually have...

34:30You've blocked a KV cache. Exactly. Exactly. Right. So in some cases, right, some models actually support that you put this at the end of the prompt. Others actually don't. Right. So now you immediately have to make this trade-off. And that's pretty much where a lot then also of evals come in. Right. You run in all of those different configurations. You compare, this is optimizations functions that you have to hit here, which is like, oh, if I blow my cache once or twice over such a long time, I'm still good, right? Or you're saying, no, actually, I can run with a slightly larger prompt, right?

35:10That is fine because I have a cache hit rate of 87 % or whatever, right? And I was like, this is okay, right? There's no big advantage here. So it's a constant kind of trade-off. And that is also true for all of the other context part, right? And there is not a real stable. I mean, there are some stables. Like you say who the user is. You say what the repository is that the user runs in, right? But the very moment already, like how much information do you give about the project itself? Like, for example, we put this kind of prefix in where we're saying, This is what pretty much the top level of the project looks like to the user.

35:50And we still believe that this is actually reasonable token spent. But then on top of this comes what we dynamically include. And dynamic inclusions is clearly like if you have an agents MD file, we have custom instructions that we actually do support. And custom instructions actually can be tailored in different ways. They can be just in a certain location, and that's fine. They can, in the front metal of those custom instructions files, you can say apply to, and then you can actually give glob patterns and saying, oh, you know, in this particular test folder, in a TypeScript file, this is a file that actually applies.

36:31And then there's yet another mechanism where you can actually give a natural language description under which circumstances that's custom instruction implies. So, and then we actually start collecting these and actually putting them also in the prompt, right? And that is actually a process that I think is the one that is the most valuable, right? Because you make sure that the user is in control of how much they pretty much AI prepared their code base, right? But when they did a really good job, then they can actually make sure that the agent extremely quickly gets to the right place, knows where to start, knows where to look.

37:14Let's maybe talk a little bit about interactions between agentic pieces and the IDE itself. You mentioned a couple examples of this, of if it's running in the foreground, use tools that connect to parts of the IDE so that they're running in the right place rather than using the terminal. But what is the surface area that you expose to the agent, to the IDE? And how do you think about changes coming from the agent versus coming from a human? So, I mean, it's all about how you interact with it, right? So again, the most straightforward form is you actually have a foreground agent running in VS Code.

37:59And that foreground agent, we give it actually quite an interesting set of things that it can do. It can look at terminals, it can read selections in the terminal, all of these concerns. It can run tools, watch tasks, all of these specialized edit tools that we then actually, where we actually are able to run pretty much snapshots at a given point in time, so that we can show you, oh, here's all the appropriate diffs and so on. There's a good chunk of tools that we actually give a foreground agent. In a background agent, that's quite different. In a background agent, we give significantly less.

38:40Why? The first thing is if you run the agent in the foreground, you have this kind of expectation that the agent actually is reasonably quick. If you think about this more, that is the interactive part. You don't want to sit there and wait two minutes and twiddle your thumbs. You want to get the answers relatively quickly. And then, again, you want to make sure that this all kind of is like the extension of what you would do anyways. But when you go and move something into the background, then you clearly don't want that it touches your UI state at any given point in time. You don't want it to, like we mentioned the example of the test runner a couple of times, right?

39:25You don't want it to mess around with your test runner. You don't want it to open up a terminal on you, right, so that your mouse all of a sudden clicks a different place and so on, right? So there are different tool sets that you're actually giving. A cloud agent is yet different. So while a background agent is still running on my local box and is still exposed to me closing the lid, neither a cloud agent is not. And there it's about in what containerized environment is that agent actually running? What is the project that you actually have? Can it build successfully in that container? or can it execute and test run in this container?

40:06Yes or no, and so on, right? And the agent. But again, the cloud agents have significantly less tools in order to do so. Remind me of your question again. Well, so my question was kind of how you think about those interactions. And you've given me a fair amount. This actually leads to something that, or a curiosity I had as I was listening to you, is how much does this differential exposure of different types of tools end up influencing how well the agent does. I'm imagining the same prompt in a IDE context versus a background agent versus a cloud environment might result in quite different coding behaviors.

40:49The model choice, I think, has a much bigger impact on the actual outcome. So what we're trying to do is really straddle the line between the user experience and the success of the agent, right? Because as we said, right, if you have a terminal tool, so execute terminal commands tool, you can get really far. You don't need an edit push, you know, and cat command with input redirection, right? And you see this is your edit and then it writes it to the file system and so on, right? So you don't really need a whole bunch of tools in order to make an agent successful going from A to B. There are some differences, like, for example, the industry introduced the to-do tools in order to have longer running agents, self-organizing, and so on.

41:48But in big parts, when you think through this, you don't need a huge amount of tools in order to make that successful. So that's one. So when we actually bring it in the foreground and give it more tools, then that's very specific to the environment, right? It's like, for example, when we come and say, no other agent needs to say, hey, maybe you should install this extension to have a better user experience, right? But inside VS Code, that clearly is a tool that is available and is particularly interesting. And you scaffold, for example, a new project, right? You go, you say, oh, I want to do this, right?

42:22And now create this workspace for me. are like, oh, and you go because you told it to go, but you don't have the Go extension installed. So it makes sense that the agent actually goes and saying, by the way, go install. Should I install the Go extension for you? So it's really more about the user experience that we try to give to folks in the appropriate environments they are in. That's really the biggest difference. And then it's also coming back to how you interact actually with an agent, right? I think for a background agent, I don't want to have a lot of interaction. There I want to have context isolation.

42:59I want to make sure that it's even running in a sandbox environment so that I'm not bothered by tool calls, right? So that I have to approve tool calls and these kinds of things, right? But in a foreground agent, right, that is more like, well, I'm not quite sure yet exactly what I'm doing, right? At least that is the use case I see primarily. people are talking about code, right? They kind of go and make a selection and say, hey, change this, right? Short prompts, right? They are not particularly long, right? Sometimes people go and actually use NES in order to start a change, but then they don't finish it and just go and forget everything, you know, hey, finish this up, right?

43:45And it should then be very quickly just, you know, in this particular file, just do the rest. or when people create new test cases, right? So test case generation usually is not something that takes particularly long, right? I mean, it depends, but in most cases, right, it's pretty straightforward. Pushing this in the background, right, and then coming back after a while to review it and all of this, it's usually more, it's like, no, no, I do it right now, right? And then immediately run it and then let me review it, right? So that there is no cheating going on, right? That the tests are not already playing to how the actual behavior is.

44:26So very different styles of interacting, right? One is that the foreground agent is really like short interactive behavior, talking about code, pointing at code, collecting pretty much the context that you want. That's all very code specific, right? Two background agents you don't really do, right? It's like you try to be precise the moment you started, right? And then at the end, yeah, you can follow up a little bit if you want, right? But it's different expectations, different levels of preparation and so on, right? And what I'm actually saying here is, right, there's more to this, which is when I say you talk about code, then I'm more like you actually are a person who cares about code and you are actually really rocking on something where you need to guide in regards to software architecture, certain patterns that you want to enforce, et cetera.

45:23There's this whole other world where you don't care, right? So you don't care what the code looks like. It's really just outcome-oriented, et cetera. And those lines, they also shift back and forth, right? They're good within the same project, by the way, right? 100%. I care about my core architecture. This tool, just vibe it. I don't care. Yes, exactly. Right. So it's exactly this, right? Where you pretty much, and this is a really interesting point, right? And I think as an industry, we might not, or we maybe don't talk about this enough, right? Which is that, how do you actually AI ready your code bases?

46:01That is exactly right. If you have a project, like, I mean, our code base, right? The initial commits and all of this are more than 10 years old, right? And since then, we built on top of this, right? In order to make our code base AI ready, we really have to think about what are the core abstractions that we really, right? Each and never go and change those things, right? If you should change anything here, we tell you, right? But then there are other parts that are a little bit more peripheral, as you said, right? some tool or so that you just want to have on the site now tool in a more generic way right that's like yeah just just go do it right and you might even just check in pretty much the prompt file that you use in order to generate this right so it's a very very different right so you think about what is untouchable what kind of lives at the periphery right where you care where you don't care most people really love using test-driven development for a bunch of the same tests are pretty much my prompts that I use for the implementation side.

47:08So there's really this great flexibility in people operating in quite different ways. I'm curious, when we talk about these different modes of operating, and the fact that we kind of flow between them, how do we connect the dots? So an example that I'm going to bring forward, I'm very interested to how you would think about this. I'm often working on something kind of interactively in that interactive mode, I'm thinking about it, and an idea comes, oh, it would be great if we do this. And I have a set of sort of predefined research style prompts that I can just kick off. So I'll like kick off a background agent and say, okay, go and research in my code base what it would look like if I were to do something like this, write me an analysis doc and go.

47:48It'll go off and do it as I continue on my main line. And at some point, I want to come back and pull almost suck that into interactive mode. Now, I can do this right now with like branches or doing things like that. But I'm curious if there's something in the IDE that lets me kind of, it's almost like I'm pushing ideas into the stack, and then I want to pop them down into my interactive world. There are different ways of thinking through this, right? So when you actually, it's interesting, because we really just discussed that. And we have a mockup, we have not implemented this yet, where a similar discussion came up, but more about at which point in time do I go back to a background agent or to a cloud agent, right?

48:36And yours is similar because it depends on the output of what the cloud agent generated for you. In your case, you want to see an analysis, right? You want to look at this and so on. So the way you think through this is whether you kick them off And at some point, you've got to go back. So you need an indication that it's telling you that it's ready for review. But then the interesting thing is that this is not necessarily just looking at something does not necessarily mean that you did all of your due diligence. So in a way, you need interaction saying, yeah, it's ready. You can go there. But then at some point saying, I took action on this is actually good.

49:19right so you want to have this awareness of those right and when you think about this i mean there's really really how should i say prior art right i mean when you think about email management tools and so on they're quite similar kind of characteristics so one thing that we had was pretty much at any given point in time right when you interact with this chat and so on clearly you can make these things disappear but you pretty much have awareness about where your background agents are and which ones are ready to review, right? You don't see the, if you don't want to, the running ones, you don't see the ones you took action on, but really just those that you haven't acted on yet, right?

50:00But they are done, right? They have produced what you asked them for. And then one of the mock-ups that I just talked about, right, is pretty much where we bring this right into pretty much the very top of the title bar, right? Where you have pretty much something that just can come down, right, as an overlay. And you just, like, super quick, you just see it, right? And it needs to be a first-class citizen. If this UI, what I describe is what it will look in the end, is a different question, right? But it is absolutely clear that you need this kind of peripheral awareness. No matter what you do, you need this kind of peripheral awareness that something else is right, right?

50:42I mean, there are workarounds for this, right? And that's actually something that sometimes we're maybe too focused on the one tool that we own and that they operate in, right? But again, right, I mean, if you want something from me, you select me, right? And I get a notification that you want something from me, right? So integrations and these kinds of other work environments that actually tell you, right, that you have a Slack channel with your agent, right? And just comes up and actually says, hey, I'm done, right? It's good, and you have the notification, and it fits in your other workflows and so on.

51:16So there is a lot to explore here. That's where I'm going. We can do some things in the IDE. We can do some things with GitHub and github.com, but I think it's not necessarily where the line is. If an agent is an actor, there's a lot of other kind of tools that already are custom-built for actors. right and so i think we we've got a broad analogy of thinking through these problems these problems i love that and it's a good segue to another topic here which is i mean vs code has always been very extensible very plug-in centric very open uh how are you thinking about that within this new world are there things that need to change i i know you mentioned mcp servers that's definitely one way of interacting, but like, what does, are there changes going on in that landscape as well?

52:12There's an interesting duality here, right? One is that in order to get new functionality into VS code, you had to write an extension. If you actually have direct access to LLMs who can actually operate some of that, right? You said, oh, I have a custom prompt file that does X for me, right? So to kick off and research background aging. Wow. But you can also have custom prompt files and do actually things in interactive mode in the IDE. And now if you give you a capability to keyboard shortcut this, all of a sudden there's extensibility right there without writing any extension and so on, right?

52:54So interesting kind of duality here, because some things you still want this extension, but some others, you have a lot of flexibility already without even required to write an extension. So that already changes extensibility, right? Then MCP services in your newish, new in these days, right? And a newish concept, right? And that is interesting, right? Because it has MCP spec covers a lot, right? But the most interesting one, the most used one is the one that you actually can make tool calls, right? You get pretty much a tool host, right? So, and that is clearly, I mean, there we're still very early, right?

53:40When you think about this, kind of obvious some of the aspects, right? But now you actually have people who kind of say, like Anthropic just posted this a couple of days ago, right? the whole part of a couple of weeks ago about Procramatic MCP tool calling. And then you're like, oh, I can clearly see where the cups are, but now we're really just making different APIs. It's like, why are we not calling them the real APIs? Why is there a differentiation between the normal APIs and the MCP servers, right? You can see that they then start fusing together, right? And that's just, or MCPs is just the API that you publish, right, and nothing else.

54:20And so that would make a lot of sense, right? But now you put this together with autonomy of an agent, and you end up in a potentially scary world, because now you need to think through the security implications. You need to think about how do you actually control this, right? What do you allow? What do you not allow? identity management permissions for agents, all of these kinds of things. So what we are doing today is we create a sandbox for some of this. But it already starts like people go and say, oh, context seven. It should be quite careful how I phrase this so that your takeaway is not, oh, this is context seven is an issue.

55:08But I give this as an example. right so you register your website there right it's crawling in markdown files i'm sure there's some sanitization going on but you know wherever there's sanitization you can actually play it now you have an mcp server that actually finds those pieces of documentation puts it in your prompt and now what now it starts building and executing code and so on right so it's this poison the well kind of problem right so you you need in a way control all entry points into this but this creates the most awful user experience yeah no this is this is a fascinating domain because in essence all of these large language models they're another form of running computation it's word programmed rather than formally programmed and so any mcp server is injecting code that's running on your box.

56:06Do you trust it? Do you trust everyone who's able to get anything into that? Yes, it's exactly the question, right? And what we just, in VS Code, for example, we had a, so the way you actually do tool approvals, right? First of all, this is just running the tool, right? So again, input outputs, right? So what you just said is the part where saying, oh, are you okay with this command being executed? And it can be local. If the MCP server runs local, it might install packages, right? UVX or NPX or SI if it didn't run yet, right? So interesting, right? But there's a number one, right? Are you okay with being this one executed?

56:53Saying, okay. So how do you want to do this for this session, for this particular call, right? Is there certain patterns in a command that you actually want to allow, right? There's a lot of room in order to get this kind of configuration, right? But then you make the call, right? And let's assume this is a remote MCP server. Now, what you pretty much said is, I'm okay that this server is being called, maybe with my authorization token, right? But now there is a response being computed and that response also goes into your either in a summarized form or in its actual form into the history of your chat, right?

57:35And I was like, oh, what now? Now you need to pretty much review everything that comes back. But again, right? That is awful. So you want to use then specialized security models to actually do this kind of monitoring, right? But now you're pretty much in this kind of, you know, who wins, right? Head-to-head wins, right? So that's what I haven't touched on. It's like this is chatted in one way, right? But now you go to terminal in terminal commands, and do you want every terminal command to ask for permission? Now you've got to go and say, no, no, let us actually read and understand what that terminal command is.

58:18And if that terminal command actually feels safe, let's do it. But then you also need to give the user control. I mean, a user in an enterprise setup might think about what tool should be called without permission, explicit, every single time permission, differently than if I'm running on a VM in the cloud that I just leased for this particular kind of use case, for example. I do wonder if it leads towards a world where essentially all development is actually happening inside of a container or VM. I think if you think this all logically to the end, that is the, I think, the part you're getting to, right?

59:03But then still, you need to control the inputs and outputs to this container, right? So fetching a web page, right? When you go and say, hey, I need the latest version of Node, right? You get an install command that runs in the terminal and this moment something comes into your box, right? So yes, you want this to be safe, right? You want to be able to close the doors and say you cannot get out of it, right? My point here is the problem doesn't go away, even if you put this into containers, right? But you can control the environment in a better way. But we still need to think about how to make this a good user experience, how to make that understandable for you and so on.

59:49And when I say understandable, is like, we're now talking about, we didn't say this explicitly, but I think our conceptual model here when we talked was, oh, there's this one or two background agents that I have, right? Now multiply this with 100, right? And all of a sudden, that is a very, very different problem, right? And we need to rock through all of this, right? I mean, that's already starting, right? When you think about this, right? So with cloud agents, for example, let's say you're on GitHub, right? You groom, you assign a bunch of issues to co-pilot, or you actually have auto-triaging enabled, right?

1:00:28So you go and say, oh, we auto-triaged certain ones. Now you get those. You get those PRs for this. You review them. Reviewing five or 10, depending on size might be good. Reviewing 200 a day, you're out of luck. I've reviewed more code in the last six months than I can remember, right? It's ridiculous. It's wild. So I think this gets to kind of where I want to take us towards the end. And we're getting closer to the end of our time here, which is where do you see this going over the next year or two? I hesitate to go too much farther out because as you've highlighted, things are moving so fast.

1:01:07But like, how is VS Code and this whole world of how we're managing the writing of code? I say it that way because maybe we're not actually writing the code, but we're managing the generation or writing of code. where do you see it going and what's coming down the pipe? I'm not sure I look two years down the pipe, right? Because we might surprise ourselves how quickly we end up in a different place. But when I think about, so we're still learning about the interactivity models, right? And that's active research where we go in and say we implemented one way, we implement the other way, we look how people actually accept it, right?

1:01:48Where does the percentage go? In the beginning, it was a lot of tap, tap, tap. Now it's more like, oh, the percentage is lower, but depending on the experience level of people and what part of the coding, right? When we talked about there's something that is really sacred land and then there are other things, right? So all of these kinds of things are influencing this, how those interaction models are. And that will change and we'll figure this out, right? I mean, different ideas will come from different areas and so on. But I think then there's this whole point about how do we use agents effectively?

1:02:26And I think that is also a very hard problem because, again, people, we run many agents parallel. Yes. But now, in what circumstances? If I have a project like VS Code where we go through 3 ,000 issues a day, sorry, months, not a day. but that's projecting forward two years right probably right but uh 3 000 issues a month right and that was just based on human activity right you you put now ai into the mix that number needs to go up right then how much of you can do more parallel work right because there are different boxes that you can execute, right? You own, let's say, as a team member, you own a bunch of tags, right?

1:03:21Those are yours. And you can go and have a couple of agents running on each of those tags. And that's kind of fine. But if I go and think about creating something new, right, then actually running multiple things in the background, that's way more complicated, right? Because it's easier to think step one, two, three, where one, two, three built on top of each other rather than, oh, it's one, and then it's two A, B, C, D, right? It's like it's way, way more complicated. We hit cognitive limits. Yeah, absolutely, right? As a human, we are pretty much the weakest link in the chain, assuming that we really get to a place where that code that comes out is in a good shape.

1:04:02But we're not there yet, right? But you can see that this is happening, right? So how do you actually really work with this level of parallelism and so on? So I think there's work to be done. I think where we clearly will end up, right? And that is now, just think about real world large scale operations, right? Where you go and say, hey, I have to make a change here. I want this new vertical feature to go in. but it now actually touches dozens of repositories, different service deployments, all of these kinds of things, right? So now you end up in a world where you pretty much need to create a plan, almost like a project plan, right?

1:04:49So agents need to go in, right? One solution is you have a monorepo and you just have one agent running around, right? But what is more likely is that it stays a distributed world, at least for many people out there. And then you need different instances of agents that actually went to manage and delegating to other agents. They are running and they need to communicate to each other, right? They need to report back where they are. The reporting back cannot be a markdown document anymore. They need to potentially go back and say, no, here is really the change tracker, right? Here are the different issues.

1:05:26They're linked to each other and so on, right? Maybe you need to see this on the planning board in order to understand this. It has a lot to do with, again, the human is the weakest link in the chain. It's about transparency, what is actually happening and so on. So I think there is a lot that will happen in this particular area where we need to go. Then there is one other aspect, but that's the one I personally struggle a little bit the most with, which is what is the, it is very easy to say, oh, I can code wherever I want, right? And if I have an idea, I just type it in my phone and send it off.

1:06:06And so I'm certainly true, right? There are some use cases for this, but I'm not quite sure how, I mean, I want to work on my iPad. That's clear, but this is just a replacement for, you know, I just sit in a different place. I don't want to use my laptop. So these kinds of things. But really, what's the role of mobile? Of smaller, smaller form devices. Really, how much do I want to do on my phone? I can see maybe voice that plays a big role in this one. But other than that, still, I don't want to review code. I was going to say, kicking things off, great. Reviewing code, miserable. That's right.

1:06:51That's right. And then I think that last part, and again, it's actually not that surprising when you think about this, right? We like to be creative, and in what environments are we creative? And I really could see that we're still in very traditional kind of collaboration forms. And as we said, you could have a Slack channel with your AI agent, for example, right? So these kinds of interactions. But I think the other one is, what is it that we really like as humans, right? We like to stand on a dashboard or together, you know, sit harder together and do something together, right? There's materials on the table that we shuffle around in order to talk, right?

1:07:36So these kinds of things. How would you replicate these kinds of things, right? And I think sometimes Microsoft had this studio PC, right? It's a large screen that you could flat down, right? And you now take something like this, right, where you can draw on the screen, particularly when you do UI development, for example, right? You go, you draw on the screen, you say, this is what I want, right? And so on. And then you can actually talk to it at the same time while you're drawing and saying, no, this here, right, should be a little bit more over here. What do you think about this? Give me two alternatives, right, and so on.

1:08:15And then all of a sudden you have a very, very interactive rock star that makes us happy as human. There's a lot of dopamine in this, right? And it can be very well AI supported by voice, by how it's actually multimodal inputs and so on, right? And there might be or there might be not code involved in this. There's code involved in it behind the scenes, right? But again, right, this is a rock star I can clearly see, right? And I think we will see a lot of this coming forward. I think that's a great cut point.