⚡ Inside GitHub’s AI Revolution: Jared Palmer Reveals Agent HQ & The Future of Coding Agents

0:03All right, we are here for a very special edition of Lanespace with my buddy, Jared Palmer, SVP at GitHub and VP at CoreEI at Microsoft. Correct. Dual title. Yeah. Twice the fun. Is it weird to have two jobs? Enough. I'm only on, I'm only, to full disclaimer, I'm only on day 13, I think. Yeah. So, early days. So, so far, so good. So far, so good. We've been trying to get you on the podcast for two years, I think. I think so, yeah. You, yeah, you're a busy guy. We don't do it in person, so. Yeah, we have to do it in person. Yeah, exactly. Way there. I should also plug that you have, if, if, if, Palmer fans should dig into your previous podcast with Ken Miller.

0:42How about that? Okay, so shout out to Ken. Shout out to Ken. Before that, you were building, I guess like V0 and AISDK and you were just sort of VP of AI at? At Vercel, yes. All AI initiatives and vibes. Yeah, and I feel like basically you went from sort of building one coding agent to now being the home for all coding agents. Is that like the general vibe of AgentHQ? I think that's right. Yeah, so backing up, I spent the last sort of two years or so building vZero at Vercel and AISDK. And then the summer took time off and now joined GitHub and today we launched AgentHQ among other things here at Universe.

1:25And yeah, it's going to be the home, we hope, of not only agents but also developers. And it seems like the gravity well of this new collaboration space that we're trying to build. Yeah. What do you think, like, basically that GitHub can do that you couldn't do at v0. GitHub is an enormous platform, right? 180 million. Yeah, it's 180 million developers. It's just the scale is immense, right? And v0 was focused on not only one language, but one framework, right? And a specific problem space with a built-in renderer. For those who are not aware of v0, it's like a Bolt or lovable, but it's built by Vercel.

2:07And it's focused on building Next.js apps, specifically Next.js apps. That constraint was rather liberating for the team at the time, and it lets us really, like, laser focus. I will edit it. I hope, thank you. I hope so. And obviously, at GitHub, you know, we're the home of all languages and frameworks and developers. And so the scope is broadened. And yeah, it's just a different part of the map, if you will, right? Yeah. So you've been basically covering the entire journey of coding agents from the start. What do you think, what's your personal journey through coding agents? We started out with Copilot, obviously GitHub started the Copilot trend.

2:49Tell us about the origin story of V0 and then how that develops and maybe what you want to see next with history. It's funny you ask that. As I've told this story multiple times, I feel like I've unlocked different parts of it in my brain by going back. You know, like, so maybe we'll have to figure out how retrieval memory works. By the way, interesting how memory works for agents. Totally. That's why I brought it up. This is small? As you, sometimes you discover new paths, right? Yeah. Anyway, the story goes like this. So when ChatGPT first came out, honestly, it was incredible, right? Like, world-changing, immediately, faster growing product ever.

3:27I look back at, like, the timeline and dates, and we were very early, like, when I was at Purcell, jumping into AI stuff. But the journey kind of went like this. So at the time, actually, there was no AI division. There was no AI group. I was actually the director of engineering for all of Vercel frameworks. And I was helping Next.js, Svelte, Svelte, Svelte, Svelte, Turbo Repo, Turbo Pack, Webpack, and all internal dev tools at Vercel. And I was helping the Next.js team dog food and test the initial implementation of server actions. And instead of building a to-do app, Gijermo, the CEO of herself, was like why don't you build like a playground?

4:06And I was like, okay, cool. So that led to the AI playground which is now just part of AISDK. We'll get there in a second. Which by the way, iconic for like side-by-side but also the... Right, so G told me that I got a DM. I remember because I was at a bachelor party and Gijermo, internally online, sends me a note like nat.dev is launching on Monday. You have to ship. And I've been working on it previously. And so I was like... Oh, you have the same idea. Yeah, so he got wind of it, I guess. So I definitely had to, like, jump into motion. And I didn't think... We didn't even ship chat first. So he just sends me this DM over the weekend.

4:41I'm at a match for a party. And he's like, nat.dev, this side-by-side, he sends me the link, and I play with it. I'm like, ciao. Okay, so I spring into gear, ship the AI Playground. What was cool about the AI Playground was it forced me to go through every single model provider's API docs and figure out their quirks of their nuanced streaming. because at the time it wasn't like everybody used OpenAI. It was like all little quirks. Some of them kind of were compatible. So that was my first foray to it. And then launched AI Playground. That shot to the top of Hacker News. And I remember I didn't even implement chat because that chat wasn't actually like a fake, like wasn't as important.

5:13It was just like complete completions. So eventually we back to chat. And that project, out of that came AISDK because I had already looked at all the model providers and all of the combinations. and I was like, okay, here's that chunk of streaming code you need. And then AISDK found that sort of niche of like, how do we focus on the part that we're going to be good at, which is like that UI aspect of it, but then also knock it in your way. So that, we shipped to AI Playground, then AISDK launched. And then, you know, we're always about demos and having great starter templates at Vercel. And I remember writing Gijero, I was like, you know what would be cool?

5:51This guy, Shad Cien, oh man, he seems like amazing. And his UI library is doing great. why don't we team up and ship a ChatGPT clone open source? And we did. We shipped this awesome template, which is now called Chat SDK, but it's great. And what that did, though, at Vercel, was it set us up for, like, rapid experimentation because we had this really good, like, pretty full-featured ChatGPT ready to rock with all the latest features. Yeah. So when it came to, like, rapid prototyping that summer, an hour, summer 23, it was so great. It was like liberating. So I remember at that point, I had gained some momentum internally and pivoted almost entirely to AI.

6:35And I had Shu Ding, who you're friends with, and Max Leiter, and Shad Sien now were cooking. And I think at that point, code execution had just come out. I think that's my timeline. Yeah, they called Sandbox Interpreter. Intercode Interpreter, that's what they call it at the time. And I had a very, as soon as I saw this, I had a very ambitious idea and proposal to present to Guillermo, which was like, what if there was some, and mind you, tool calls don't exist. The context window is 4 ,000 tokens. So, like, there's not much here. What if we had this thing where, like, you could prompt, and sometimes it would decode interpretation.

7:13And then maybe we could sort of, sometimes it would decode up interpretation. But then other times it would choose to render, like, UI. or then it would render sometimes like a document or... Inline in the chat. Yeah, it would just have different sort of render moments. Generative UI. Yeah, and maybe you could pipe them together. So like the output of one could pipe into... So if we did code interpretation, we coerce it to always emit like tabular data. Maybe we could pass that to another prompt that would just like a UI. And just like some idea there, it's kind of crazy. But if it sounds like these are just tool calls, that's exactly what these aren't really was, yeah.

7:49so it became pretty obvious that like so I can't be just a reminder at the time we just had a we had a sort of security debate like should we code interpretation with the ability to fetch data like giving it internet access was like kind of now they're like fine whatever whatever you want wow wow west but at the time it was like a little scary so we kind of said okay no to the code interpretation but this UI thing it's pretty neat and so that this like prompt to UI That was like the aha moment of v0, but the models were not very good, right? Or relative to where they are now. And it was the 4, GPT-4 era?

8:27This is just into the GPT-4 era, and now we're probably at a 16 ,000 token context window. So you can't really do chat. So we had to kind of invent this kind of new paradigm of like fake it with completion, but that forced us to do sort of the click, the initial v0, which launched, I think, in September 23, it would look more mid-journey. In fact, if you go back to the original tweet, it was like mid-journey for React because it was all very visual. And you could click on different components and elements and reprompt, but it was, again, we were kind of hacking this because we didn't have chat and we didn't have tool calls.

9:04And then fast forward again, you know, that launches and then probably like nine months later, it took us like nine months to get to like a million ARR. this little team, but then the models progressed. And from, you know, GPT-4, GPT-432K, the big boy, we never really got GPT-4 Turbo working. I don't know why. It never happened. And then switched to other Frontier models and then started doing our own models and stuff like that. But fast forward another 10 months or nine months or so, and then we rebased towards chat. And now the models finally could do chat. And the artifact pattern had evolved, so it was time to rewrite.

9:47When we launched V0, the chat version, or the new V0, whatever you want to call it, it's like 14 days, another million MRR, 14 days, another million MRR. It was like a rocket ship after that. And that just proceeded. And we just kept cooking. And so that's been the journey. We just kept perfecting. And what was really liberating for us was actually the focus on just one stack or one framework. When everybody else was trying to do general purpose coding agent, we were like, no, we're just going to focus on Next.js front-end and ShadCN. And that allowed the team to focus. So that's the story arc.

10:21I mean, to be fair, like, because Next.js is so dominant, basically everyone has to be good at Next.js. Right. But being focused on, like, even right down to the UI library and component stuff, like, that actually helps a lot. We also started working with all the Frontier model labs to help because it was in our Vercel's best interest to have them be great at Next.js. And also because of the post-train models, and you can read about this on the Vercel blog, the post-train harness that we created, we started sharing and stuff with other model labs and stuff like that. And we had all our data in a very hygienic state to work with them.

10:56Did you ever debate internally, and because from my seat at Cognition, I can also see this, where you should pick the best qualities of every model and string them together in the V0, or you have the model selector and you let customers choose? We went back and forth, and I think we launched, we went back and forth on all this. I think at the end of the day, there's pros and cons. Yeah. One of the benefits of having your own branded model or synthetic or composite is that you can stitch these things together. Yeah, there's like higher levels of and now it's a little different with with this agentic flow.

11:34But even look at what you like what you guys launched recently with these are actually wrapped. Right. So search is going to be a different model than what generation, but, you know, the Genesis. but like, because it's a sort of, but like search and Genesis are two different, like entire subsystems, right? So you can have search evals that are gonna be totally different. And so how do you, so where we ended up now, where are we, where it probably ended up now is like, for a long time, we didn't have model selector and then we had our own models, which were composites, which we talked about. And then like, and that would allow us to, you know, mix match.

12:10And I think that's probably what, it's also nice because you get as a, this is like the product app, you get to brand it. Yes. Right. And you can decouple it from the launch of the Frontier Lab. Yes. How do you guys, how does Cognition even build for it? Like APUs. Right. It's a synthetic unit, right? Yeah. So it gets a little wonky. Yeah. We can go on for pricing this stuff. It gets challenging. But the nice thing about having the like brand name model is that like you get to co-launch with the provider and they'll hype you up. So, but your billing needs to then is capped at whatever retail is, right?

12:48Or some. Right, right, right. You can't really charge too much of a premium. You can, but. Right. And people are like, what are my only payments? Yeah, I key, right? And it's like, well, then how do we charge you for sweet rep or something? Yeah, yeah, yeah. And so I think some part of it is the cynical, like you want to create a sustainable business and independence from the model labs. But the other part is genuinely, you actually do get better performance. You string together all these things. Yes, and so it's tough. I think what we've switched gears to GitHub, like we are all about model choice now and making sure that.

13:19And what's cool is that we also have Copilot, which is our harness, and Copilot CLI, but we also have third-party harnesses like Cloud Code and codecs and cognition now in AgentHQ. So you kind of get the best of both worlds. And I think that's going to be awesome and ultimately what people want. Yeah, I think also the model layer is not the right abstraction to do the switcher anymore, which is weird because that's where you started with the ISDK. Yeah, exactly. But now it's like the model and the agent have to be strictly tied together, like very, very strongly balanced. You can't loosely bound it and just do a generic interface because then you're just going to have the lowest common denominator of all the models.

13:58If you're in agent world, which may just be better than chat world, like in general, like better. Agent world is a much better abstraction. I'm calling it agent world, but I mean by like a loop with maybe compute runtime and like files. That's your definition of agent. You're dropping your official definition here. No, don't put me on that. But maybe, no, so my initial definition of agent for AIS, because like I actually fucked, I was dying on this hill. Because AISDK, everyone else is an agent framework and I think maybe they actually went to like this, I don't know what it says on the front page now, but like, who knows?

14:30But like an agent is, you know, an agent is orchestrating, you know, an API request with a queue and a for loop. Okay, but a coding agent now has meant so much more. There's like these, you know, coding agent SDKs and you've got sandboxing and file systems and tool calls. And I do think that is a uniquely, I'll call that agent world and I'm trying to get coding agents here. And yeah, I think that that seems to be where things are going. And even, I believe, the Claude Excel agent is basically, I was talking to Mike Krieger backstage, like, I think it's related to Claude code. It could be. I actually don't know how I should implement it under a hood.

15:09It wouldn't surprise me if it was. Yeah, yeah. They seem very all-in on skills, which is kind of like an interesting... What do you think of skills? It's kind of DXT, which is like the sort of bundled version of MCPs. Okay. The reason you don't know about it is it wasn't very popular. So skills is kind of like the second shot that is very LLM-pilled. It's like, just read my markdown and just read this directory of files and go nuts. As long as I can understand that you have the capability to run code, to read files, you're good. And actually, that is the universal interface, which is a file system.

15:47Right. Back to agent bosses. Yeah, which is kind of cool. Yeah, so I mean, I think what you're hitting at is this philosophy of our understanding of what coding agents, the minimum bar is over the last two years. Yeah. Right? You've lived this journey, and now you're basically kind of like the kingmaker. I don't know about that. You run Agent HQ, and I imagine you have other projects too, but Agent HQ is the big one that we're talking about here. Like, what are you seeing from, like, the different agents? Like, what do you want this to become? Such a good question. I think that AgentHQ and GitHub itself need to co-evolve.

16:33And, you know, one of the things that Microsoft has done really well is by putting things that are alike closer together. And so you think about the new core AI organization. Yeah. I've got Visual Studio, Visual Studio Code, GitHub, and parts of Azure all in one. And obviously the GitHub team and the VS Code team have been working closely together for a long time, but now we're really close together. And I think for me, one of the cooler things that Agent HQ can sort of offer is this seamlessness, this fluidity with your workflow, right? So if you saw in the demo today, we saw a demonstration of you use AgentHQ, you fire off a task, and it creates a PR, but you can also open that PR up in VS Code in one click.

17:19And that's awesome. And I think the vision for GitHub as it evolves is to look at those touch points where AI can be sprinkled in, you know, salt-based style, into the native workflow, whether you're assigning an issue or maybe some new stuff that I think we should focus on. Maybe it could be like, how do we resolve a merge conflict? Oh my God. Right? Like, how do we maybe pop open an action or like get in, right? I think solving merge is my definition of AGI. Totally. But like you get that error on an action and you're like, we've all been in that sort of flow where like actions kind of don't work locally or is that tool act.

17:59If you're trying to, I don't know, I've got that set up on my machine. I haven't done this in a while. So you're pushing up and you're this like, okay, what if we could just put like, you know, comment or kick off a task to solve this for you or throw things there. I think what I'm trying to describe is this like this workflow where it's just like seamless and fluid and you can stay in a flow state across all devices, mobile, web on github.com, or in your local editor. And I think that's where my focus is going to be in the next six months or so. Yeah, yeah. Just a side tangent on this. So one of the things that Microsoft also owns, I don't know if it's Microsoft or GitHub, is dev containers.

18:39And I think a very important concept for sandboxing environments, whatever you call it, it is kind of a light version of what Docker containers are, kind of. Right. Do you see that as a standard that we should invest in as like a thing? Because it's supported in VS Code. I don't think it's just that popular outside of VS Code. Yeah, it's used internally at GitHub too for like development at GitHub. Oh, yeah, yeah. Which is cool. Yeah, I think they were so far ahead almost. But now there's like sandboxes, there's so many of these days, right? So I think Cloudflare just launched theirs. There's Daytona, there's here, Purcell, Modal, which I think Lovable uses.

19:21I don't know. I mean, you probably have your own. I don't know. What do you guys use? Just some Kubernetes pods. Okay, you guys are rolling it yourself. I think that's maybe the runtime. But there's work and discussion about what that runtime should be, even internally at Microsoft. We've got a couple different competing things, so we'll figure it out in the next cycle here. But there's a great point. like there's a lot of cool stuff that is in a dev container. You've got VS Code loaded, you've got a file system, you've got a sandbox, you've got the security protocol. Yeah. But also like wired in to get up enterprise.

19:52Yeah. And like ready to be packaged. So there's lots of goodness there. Yeah, I see like the number one pain points that Combination has, but also Codex, also presumably the other guys, is repo setup. Yeah. Which is effectively what dev containers and a Docker file does for you is like run this thing, then that thing, set this up, do that thing. Why is it so hard? Like, why haven't we resolved it? I don't know. I think it's hard because you can't predict what's in the repo, right? So it's like, and you don't know when they've bundled FFmpeg. You just don't know. It's nice when, like, if it's just Next.js, you just run PMPM install.

20:28Correct. Correct. You can, like, do special, like, there's obviously through constraints, you can make optimizations. And I think the general purpose container is just, like, challenging. That being said, though, I think there's probably some work to do on auto detection and preempting and stuff that can be done there. But it's just a bigger, it's a broader problem space, right? Yeah. So fun fact, when I was at Nellify, I actually wanted to reach out to Rizal to do like a standardized open source auto detection thing of frameworks. Oh yeah. And like we never, we never really got internal momentum on that.

21:04It was an idea. I was like, shouldn't this be open source, you know? Yeah. Like auto detection is a common utility that everyone needs. Yes. Yes. I remember that. I'm having a flashback. It's like, yeah. Everyone builds their data. Probably we shouldn't all build it. Right. No, it's, and then also, like, what are your defaults? They're not exactly the same, which would be better to, like, just having even the same, like, preference stack of defaults. Yeah. Is the right? Yeah. Would be great, because then we could move the whole ecosystem together from, like, to PMPM or Bunt, right? Okay, so are there other movements or protocols or standards that you're interested in?

21:44Like MCP was a big winner this year. There's other, like, I don't know, A2A, ACP, all this. That's been interesting. I'm not as familiar with. ACP, the payments or one or the Zed one? Oh, no, the Zed one. The Zed one, okay. And then the payment that was a Stripe or Coinbase? Stripe. Stripe, yeah. That's very cool. We've had him on the pod. Okay, yeah, that's very cool. It'd be interesting to see if that takes off. I mean, it's true. It's true. Yeah, but it's supposed to be adopted by the clients, right? And I think that's fascinating. The MCP is huge, it seems. It is the way that a lot of the, especially when it comes to digital transformation or some of our enterprise customers, it's where they are able to add context.

22:26In addition to that, we also have custom agents that we announced today, too. So you can work with prompts and stuff within your agent HQ and customize these agents for different tasks and those can have MCPs and such. And I think that's going to be really powerful from a platform perspective. It gets me excited. That's what I think is shipping now and the next, but we're always on the lookout for the next thing. I don't know, what's on your, what's top of mind for you? For standards? Yeah, standards. Standards? Oh, should we be talking about that? Dev container. Look, I think dev container just is a PR problem.

23:00It's a great idea. Right, right, right. Just no one makes it interesting. I think you can do it, basically. Okay. Add it to my list. But before that, probably, you have a bunch of other stuff that I do want to get to. But just staying on the AI stuff, I think we're actively exploring computer use as a thing because it kind of got going a little bit. People were very excited, and then they found out it was slow and bad and inaccurate. It is computationally intensive, my understanding. It's getting better. Yeah. especially with open vision models like DeepSeq OCR and Omole OCR. Like, just give it a few more turns of the scaling.

23:41It seems like it's like you need that edge case and primary, just see how it's like a modality worth pursuing. Yeah, I think a lot of people are, on the code gen side, the code agent side, a lot of people are trying to think about, all right, we had this evolution from co-pilot to like a more energetic sort of, like a cloud code situation is where I think that the status is. Like, what's next, right? What's the obvious next step? Making them good? Making them good, yeah. You don't like little cars? No, no, I don't. It's more just like, you know, the devils in the details, like going from 90%, going to like hill climbing, it gets steeper, in my opinion.

24:21It gets steeper. And so going from 90 % success to 95 % to 98 % to 99 % to 9 % of success, I mean, we're really hard. Paying Mercor a lot of money for expert programmers of open source maintainers. Then you realize along the way, maybe the users aren't that good at it. No, but I just think there's a lot of work to do to finish the swing. And there's a big difference between 98 % and 99 % correct. And that's noticeable. And this used to hit, if you're working on an AI product, you probably don't realize how, you've probably seen this, most people are blind, like living in la-la land about how poor quality their AI product lately is.

25:03Unless they're really measuring like the number of error-free sessions, like how many errors are coming from the infra providers, like, you know, how many requests are dropped, how fast, how fast these things are. And so that's something that we cared about at ForSell quite a bit and like we'll care about. Do you have like a daily review of your dashboard? I don't know. Daily? Daily? Oh, it's slow. Okay. I thought you were going to say daily is too much. No, Oh, and even thinking of us like every three hours, a roll-up of key metrics and stats. Yeah. And like one of them was like error-free sessions and other things like that.