Amp: The Emperor Has No Clothes

28:31Is the GPT-5 codex only being available in codex making you nervous about future availability of like cutting edge models and like does they put more emphasis and like figuring out maybe like an open source strategy they make it available to api customers it's delayed and if they were doing that i really think that for the most part i take these model houses at their word and they wanted to get it out to their first party product as quickly as possible because they honestly need to gather more data and they're iterating in public so yeah i would love it if all the model houses perfectly coordinated with us before they released anything.

29:09But I know that would slow them down. And I don't want to slow them down like that in the same way that we want our customers to give us grace and help us iterate in public. Yeah. I think there's an interesting just dynamic in the market. Like when Cursor switched from Sonnet to GPT-5, it's like the default model that was like, you know, 200 million of revenue for Entropic that kind of went away and like moved on to GPT-5. So there's kind of like, okay, we're all friends now, you know, but maybe later that's going to change. But yeah, it's an interesting. The other thing also is that, you know, if you're building an agent and you're not at one of the model houses, you can use multiple models from different providers.

29:44Right. So which is what we do. Like we, when you use AMP, you're using a model from Anthropic, you're using a model from OpenAI and you're using a model from Google. And we're also very close to shipping like a fast open source model that we can use as a different sub agent in there too. And, you know, when you put it like this, it seems silly to say we only use one model of like this family because they all have different strengths and weaknesses. I think we are one or two months away from a possible news cycle. That is the foundation model companies have spent billions of dollars in CapEx and hired like crazy.

30:22And now, you know, they're no longer the best in this realm. And there's a huge stampede away from them. That's very possible. And I'm not saying anything new. just imagine last May when people were counting Anthropic out before Sunnet came out. Things change so fast here. Yeah. Yeah. Yeah. And I think OpenAI, obviously with Johnny Ive and some of that, it's moving more in a consumer fashion as well. So it's been interesting to see the big push on Codex. I would have imagined them to go more towards education, kind of like big. And I know they have a lot of big enterprise contracts for like Chagipity Warrior Enterprise kind of thing.

30:59So, yeah, you guys, I think, are in a good spot because you have both like the source craft trust, like you said, but also like AMP. I see a lot of great stuff on Twitter. You know, people are like, I just put all my AMP agents running. I came back. It's great. It's like I think it's now on that wave of like, OK, this is like one of the best tools out there. And like if you're like a serious engineer, you should probably use AMP in some capacity and then make your own choice. how difficult is it to think about what goes in your harness versus like what people should build so you have custom commands you've done a great job on like the tooling where like people can put like executables as tools instead of having to define like an mcp server it's like yeah how much of it you're like hey we're just giving you the tools versus how much you want to be opinionated with things like i mean i think of like compacting conversation is like maybe one of the key commands that people have and like in clock code you can give a custom prompt to compact like a what's that discussion like yeah the main assumption again everything is changing we got to be able to move fast that means what you want is i don't use the picture of a harness often what i use is like a scaffolding like you want to build a scaffolding around the mall a wooden scaffolding that if the model gets better or you have to switch it out the scaffolding falls away you know like the bitter lesson, like embrace that a lot of stuff might fall into the model as soon as the model gets better, right?

32:27Because then it can remember more, whatever, or it doesn't, why invest three months in like a separate apply model when the next generation, you know, 0.7 version or 0.8 or whatever version of this model can now do all of the edits on its own. So that's again, the bigger thing. And with that in mind, we really try to restrict a lot of the features that we add around them all. And you can do a lot of stuff, like we could be busy all day adding stuff in our clients and whatnot, making the product more complicated, but we don't want to. So that's the first thing. The other thing is we're living in strange times.

33:03We're living in strange times from a product development perspective, where basically I think the old triangle of design, product, and engineering, it's kind of changing. It's not a triangle anymore. I don't know what shape it is, and it's a blob, whatever, but it's not a triangle anymore. And the reason for this is because you can't build a roadmap. You can't say, this is what we're going to build in the next six months. People don't know yet how these models can be used to their full extent. Everybody's figuring this out on the go. That's another thing. The other third thing there is, we just talked about this while having coffee before coming here is that the only UI basically is like a text UI and you can use this in the wrong way.

33:48And the example I used earlier was if, you know, you buy Gyra, for example, but you use it for your shopping list, Atlassian is happy about this, but that's not what they built the product for, right? But you can use it in the wrong way and still get results. The problem with LLMs and a lot of the models is that you can use it in the wrong way and it looks like you're getting results. You know, like you can use OpenAI, ChatGPT to look up serial numbers or, you know, technical specifications for a camera or something. And it will tell you this, you know, but it might be wrong or 99 % of the time or 98 % or 95 % of the time it might work, but in 5 % it might not work.

34:27So having non-deterministic LLMs as the heart of your product is something unprecedented that we have in software, I think. so with that in mind a lot of the features what we see you know where people build like elaborate workflows like i have my custom slash commands and they trigger custom subagents and they in turn trigger custom mcp tool calls on behind which again another model is doing inference again and taking the input and blah blah blah i think a lot of this will and has resulted in hangovers where people realize oh like this looks like it's a deterministic workflow it looks like it does the thing that I want it to do, but actually I can't use it if it only does it in 98 % of the time.

35:10So that's something we're really conscious of where I think everybody's experimenting, everybody's sharing their experiences, you know, the thread boy tweets about what to prompt, where and how, but you have to be super strict about not giving users a false sense of what the product can do and how reliable it is, because I think it's dishonest in some way and it doesn't lead to good results and you know just as an example i think of the last three months i would say we're we're ahead of the curve like using amp internally like we're ahead of like the mainstream agentic adoption like say a month or two where we've tried a lot of this stuff and then realized oh this wasn't the best use of our time or the tokens and now you see a lot of other people waking up to this famous on Twitter, Armin Roniker, the Python developer from Austria.

36:04He's done a lot of good stuff with Cloud Code and shared a lot of his learnings. And you could see that the way he tweeted was super excited, like a lot of things. I can now do this and this and this. And then a month later, it's like, oh, maybe, you know, having eight remote control agents that I control with my phone and let them run for 20 hours, maybe that's not as productive as I thought it would be. And, yeah, it's something that we're super conscious about. What are those things? What are like the failure modes that you heard from customers where it's like, hey, we tried AMP and it just didn't work at doing X, Y, Z.

36:39Is there a collection of those that you guys use as almost like a North Star as you keep building? I think like one of the things is the whole vibe coding stuff where people just use it and, you know, they're like, hey, I spent 10 bucks in tokens and it didn't build me the fall app or something. the failure mode of outsourcing the thinking but not the typing which i think it should be the opposite you still have to know engineering you still have to know how to program you still have to know your application and its architecture how it's deployed and then basically use the agent to do the work that you would have done but you have to know what the desired outcome is and whatnot like that's a common one where people just you know hands off the wheel agent you go and write this for me and then turns out a couple hours later oh actually this nobody understands it's spaghetti code amp it's different from the products it competes against so we've had one head-to-head loss with amp where we lost against the you know usual players and the reason why is one of them discounted their other product 100 for two years the other one discounted it 85 for two years, which is just crazy.

37:45And we wouldn't want to do that because are we really going to learn from that? And then how is it going to be used to be used in a different way? So usually the way that we might lose is there's some other product that would go to 80 % of the devs in a company that is like the base layer. Sometimes that's copilot or cursor and AMP is more expensive. It's more powerful. And they'll give it to that 20 % of devs that they trust more. And in a previous world, any software company would say, oh, no, we need to get 100%. We don't want our competitor getting in there. But actually, that means that we're able to even more focus on being bold and crazy because all those devs can always fall back to a cursor or a copilot.

38:26So we actually really like that kind of deal. The other thing there, I think a bunch of questions already touched on this, is that talking about segmentation or market or the ideal user, again, everything is changing. so what we try to do is we try to you know build a tool for people who are at the frontier or at least curious about it and want to figure out how to use these agents in the best possible way and that's based on the assumption that if you build for the mainstream user who not you know mainstream sounds like i don't know it sounds bad but what i mean is what i mean is if you build a product for somebody who does not know what a good prompt looks like you will fall behind right now because you will spend time and resources building stuff like the prompt enhancer and like blah blah blah blah blah but then you will end up building this and you miss the next step change that might happen so the way we think about it is we build for the people who already get that a lot of stuff is changing but we want to leave the door open if you're open to learning new things and you want to learn how to use AI and agents in your workflow, please come with us.

39:38We're happy to have you. But if you're skeptical and you think prompt engineering, that's a bullshit term, I don't care about this, we're not right now building a product for you because we would fall behind. Yeah. So prompt enhancer, that's a bullshit feature that doesn't actually work. The theory behind it is nuts because what helps LLMs is not tricks and phrasing your prompt in a certain way. It's fundamentally information that you have in your head that you can bring into the prompt. And if you don't have that in a prompt enhancer, LLM cannot magically conjure that up. It cannot narrow the search space for you.

40:12Custom subagents. The way that we disqualify that as something we wanted to build at this point is because you look at all of the tokens that you're sending to the model and it's so many more. It's so much more convoluted. We don't think that these models are trained in a way that would support this use case and the output What if this going in here? It's so much harder to debug. And MCP is another thing. MCP has done a great job in getting products to expose the verbs that agents might want to interact with. Although, in most cases, they do not actually get the right verbs exposed. But as a user-facing technology, it is such a common failure mode where a user will go and add in some MCP servers.

40:57Auth is a huge pain. But let's say they get over that hurdle. then they have, I don't know, 50 tools exposed that often are too low-level granularity. And it takes a ton of tokens in the model. It makes everything slower and more expensive. They're often misused. And it's just not a good experience. So, you know, there's all of these things that we've said no to, and other tools are bringing them in, and they're saying yes to all these things. I think it feels like they're making progress in the meantime, and people retweet, and people talk about how they're able to do these amazing things. But just the simplest example, that seems so obvious.

41:31And frankly, it confounds me that more people don't do this. You make it so that my Google Docs and Notion and Linear and GitHub issues are all accessible to my agent. The vast, vast majority of developers who use AMP or Cloud Code or anything else, they don't have all those context sources set up. That seems like such a slam dunk. So we built that, we ripped it out. Before we would move forward with that, we'd have to get an answer, even for our own usage, why are we not doing that? And it's frankly still puzzling to us, but we're not gonna touch that until we get confident about that. And to come back to the example, you mentioned compact.

42:03We have this in the product, but again, the hand is hovering over the rip it out button because I think compact is such a alluring thing where people think, oh, you know, I ran out of context. I hit that button. Now I'm back to the start. But you lose signal, you lose data. And it's something where are the models really good enough? Is compacting good enough to really glance over this that the user doesn't have to worry about it? or is it something where you would have to somehow make it clear to the user that, hey, look, your conversation has 50 messages back and forth. If you hit compact, this is all going to become blurry.

42:39You know, you're going to compress it and you lose signal, you use fidelity, and then you put it in a new context window. Are you sure this is the right trade-off? And some users are, but again, like it's strange times because now we have like this thing at the heart of our software, this, you know, orb from outer space that can do sometimes whatever it wants. And it's, it's strange to build on top of this. And it's strange to educate your users about this, that this is the thing, right? Like imagine, you know, the end of the nineties PC era, you had to build Microsoft Word. And then you say like, well, at the heart of this new personal computer, the Pentium 3, whatever, there's a weird orb from outer space.

43:19And sometimes if you bold text in Word, it actually makes it italic, you know? But that's the situation we're in. Like, that's the fact. Like, it doesn't always bold the text. I mean, it underlines it if you reach 150 tokens or 150 ,000 tokens or something. How do you teach this to the user? Yeah, and, you know, we're in the church of context engineering at the Chrome office. And when we had Jeff on the podcast, they talked about the context draw paper that they did. They mentioned specifically in coding, for example, showing previous failures was, like, not helpful at all to the agent. And so I think when you're compacting conversation, there's almost like, you know, if you have a long conversation, it usually means something went wrong along the way.

43:59And you had to like go back and forth and like a bunch of things that didn't work and you're keeping those in. But I've been trying to figure out what's like, what's that going to look like? In my mind, it's almost like if you take the idea of linear, which I use and I give to my agents just to get because then I have a canonic prompt for one issue. Because often you have to restart because it's like it just goes too much down the wrong path. A lot of people don't restart. A lot of people just try to keep going. Yes, that's bad. But how in that case, what can you take from that conversation as a learning and put it back in the upstream issue so that then the issue is like either more descriptive or as like more information that is not compacting.

44:40But it's almost like how you would do as an engineer. It's like you're doing it in your mind, right? You get an issue and then you start working and then you kind of update your mental model. It doesn't really work for agents, but people are not doing this like small increment in the initial issue. I would say in this case, it's still you cannot outsource your thinking, right? Like in this case, I don't think you can expect right now a model to say out of this conversation, this is the most important thing. Let me put this back in the linear thing. Maybe, you know, if you phrase it like this and automate it like this, it's always a perfect conversation.

45:11Maybe it works. but i think in this case you still have to be mindful of the context and what we encourage users to do for example in amp is to start a lot of small threads and be really you know do context engineering and be really strict about what goes into context and what doesn't and the other thing that i think touches this on is you know where a lot of cli tools for example have super verbose output and basil for it to call this out i'm not a big baseman but you could just call base out super verbose output so then the natural assumption is oh let's hide this from the user you know like let's let's abstract this away and summarize the output or whatever or whatever just the exit code or something and then you get into this dangerous territory where what you see is what you get is not true anymore and in the context what you see is like some other thing in the context and that can lead to issues but for me the meta thing here too is everything is changing.

46:08That means we're seeing this CLI tools right now are also adopting to being used by agents. So they're changing the output too. So if you focus on the fact that Bazel will always be robust and build something for this issue, you might be outdated in half a year where somebody is like, no, no, no, we have a Bazel agent wrapper and now this is not an issue anymore. One model that I have is if you are relatively on the cutting edge of using agents and there's some persistent problem like this, it feels kind of out of band, like how the model itself will update its memory or will update the linear issue.

46:41The model needs to be trained in order to do that better. If it's something like your own coding conventions, that's different. But if it's something fundamental that feels like about out of band from the agent, the model needs to be trained to deal with memory better or to accept the fact that it might have an incorrect view of its own history if you go back and edit it. And we're feeling these pains right now because people have only been using Agenda coding tools for a matter of months. Most people have been using them for like less than three months. And if we're only feeling them now, it takes a little bit of time for a team at a model house to go and do a fine tune of one of their really big models, or they've got other big models, the new revisions that are being trained, and they can only fit a certain number of experiments like this in.

47:24They're probably going to get half of their approaches wrong. So you can only do so much. And that's Thurston's idea of going with the grain of the model. And I mean, you've seen this, I'm sure, where a lot of users are going through this lesson where they, let me just add this MCP server that does everything I wanted to do. And then two days later, it doesn't use it. Like it never calls the tools. And it's like, yeah, it wasn't trained to do this. And you can sense, you know, like they have different philosophies in the model houses. I think Anthropic is, from what I can tell, working a lot or training a lot towards using memory, like storing information.

48:01One of JetTPD obviously has this OpenAI. So if you give it a memory thing, yeah, it might use this. But then you have the issue of, well, if I give it this other custom-made MCP that we build internally and our processes don't map to anything that OpenAI and Anthropic have seen or trained for, it won't be used and you won't get good results. And it's super strange, right? Yeah, I wrote this article for the GPT-5 release about models self-improving for coding. So I basically asked GPT-5, what are tools that would be useful to you to be a better software engineer? It's like, well, you know, give a list of like 10 tools.

48:37And I'm like, okay, implement them, wrote all the tools. And then I asked it to do the same task I'd done before, but with those tools. And then it goes through the whole task. And I'm like, which of the tools did you use? And it's like, oh, I didn't use any of them. And I'm like, what did you not? It's like, you know, to be honest, I don't really need the tools. I can just do this task, you know? And I think that's like a good metaphor just for like the trend of the models, which is like, hey, they're going to use less and less of this like custom made tools to fix today's issue. I think the things that we can bet on and I'm curious to hear your thoughts is like they're always going to have some sort of like test runtime.

49:13Like I don't think there's going to be a world in which the model is not going to run test and say, I'm sure this is going to work. The other one is there's always going to be some sort of like infrastructure as code to then handle the deployment side. So I think whenever there's going to be some runtime issue, they're going to need to understand where they're running. You know, so I think like you can put them in a box, having an actual Docker file and whatnot. It's helpful for them to explain what they have access to. What do you think are like other things that you don't expect the model to like in the model that you want to still expose to it?

49:47So we can assume it's going to test. We can assume it's going to have some definition of its environment. Are there other things that come to mind? I think test is a big one, and there's many different kinds of tests. So we had subagents in AMP, you know, among the first that come out with this conception of subagents, which is a separate context window, more curated set of tools. And I think there's a lot of potential to take a tool like test. And right now you invoke it by the bash tool and you have some complex invocation. too often it'll run all of your tests which is noisy and it takes a long time if you're in your editor and you've got something nice set up you can hit like a hotkey and then it'll only run the tests that you need you know at your cursor so giving the llm a tool like that seems to have a lot of potential and then that could even potentially be a smaller model a fine-tuned model for that task it could be multiple based on what projects or stack you're using and that could eliminate a lot of the confusion.

50:46Even with a good agents.md guidance about how to run tests, I still see with AMP, and I think we've tried to make this really good, it only gets it right maybe 90, 95 % of the time. Sometimes it'll run the wrong testing or it won't escape it correctly. And I think we can eliminate that with a sub-agent. So there's so much more potential to go deep in areas like that. And then for every language, it's a little bit different. So handle all those cases. Do you feel like that will just be built by each company on their own? Or do you think there's like a same default that you guys are going to build for that that is going to be effective for most code bases and test structures.

51:20This is where scale helps. And we have a lot of scale. So, you know, increasingly we're able to see in this framework for this standard Go unit test package, that's easy. VTest in JavaScript, that's easy. And once you start getting the more of the long tail, then, you know, it might have to just fall back to a really good model. But I I think that we could probably make something that's optimized for some of these more popular unit testing frameworks. And it's a combination of deterministic stuff and non-deterministic stuff. Because right now in my VS Code, I can hit Apple T if I'm positioned in a test file inside of one of those test blocks.

51:55And it's only going to run that one. So, you know, even that is a benefit. And now I'm mostly bottlenecked by your Playwright. It just takes a long time, man. But the crazy thing is the vast majority of devs who are building web applications with coding agents do not have Playwright. And if they have it, it is set up in such a shitty way where it cannot really log into their app. They don't have any pattern for that. So even something like that, that's another example of a subagent that's go and try this basic end-to-end testing flow described in natural language with the running application. And wouldn't it be great if it could also do it in parallel?

52:27So, you know, there's all these ways that you can improve. That's a great example. And I think touching on this, we have coding agents. They are productive. They add value. we cannot assume that everything around the agent in dev tooling or code bases will stay static so i think people are already adopting their code base to be better used by agents or they're adopting their tooling to be better used by agents their more descriptive help text or whatever it is so i think i don't know we should have a counter but everything is changing i don't know saying this again but we cannot build right now with oh this is the tool that's going to stick around given that all of the code bases and all of the processes and all of the dev tools will stay the same.

53:08We have to assume that this stuff will change too and we have to stay nimble. So we have to make like short bets or small bets and try and get us, you know, in small steps forward, but always be reactive to this stuff. That, you know, if people, again, let's not use Bazel again, but I think Playwright is a good thing where the feedback loop is incredibly important to working with these agents, like that the agent can see whether what it's doing is actually working. So what we've seen people now do is, well, instead of having the client log and having the browser log and having the database log, let's have one unified log because then it's easier for the agent to just look at this log and make sense of it.

53:47And then it turns out it doesn't have to be nicely formatted. It can be verbose. You can just have like JSON line outputs and whatnot because the agent can understand it much better than a human can. And I think that's just this little preview of more things that we will see where you're like, wait a second, this is not made for human consumption anymore. How can we optimize this for agentic consumption? And then maybe the game changes. And there's some things that now we get, for example, in my vitest suite, I have a knock to record HTTP calls. So whenever, especially for inference, like you can't really knock, we do a classification, things like that.

54:23You just need to see what happens. And then we just save the whole interaction and then the model can actually see what the API returned like in much detail and like it can reference it back in the future so when you add a new feature it can look at the test and it can see what the API usually returns and it's like oh okay it's gonna have that key and like the content and things like that I think there's more of that to be done I think there was maybe also a time in which having console logs was like really bad and I I think there's maybe not going to be a console log that is like only funneling to like not the actual console in the browser, but like some way for like the agent to see all of the details of like everything that is happening.

55:04What I haven't figured out is like, how do you instrument that? Because you cannot put a whole bunch of console logs that go somewhere else in the code because then you're also polluting the context window of the model. Right. So you need some other way to do it. But I think, yeah, the more you log in, the more the model can kind of like self iterate. And you just described like five approaches that seem absolutely worthwhile to go explore to improve how coding agents work. Somebody do it. We can do some of it at Kernel Labs, but we cannot do all of it. So somebody help. Again, like the world around us is also changing.

55:38Jose Valim, the creator of Elixir and, you know, contributor, co-contributor Rails. Can't remember the name, but basically they have a new framework tooling out that is... Phoenix. Yeah, it's for Phoenix, right? but it's the name of the, I can't remember, but it's about, well, what if you build a framework for an agent to, what if the agent is integrated into the framework so that you can, if the application fails to run, you can ask the agent that has access to all of the context. And that's going to be more and more, I think. Like a lot of, you know, developers will build stuff because they're fed up with like copy and pasting stuff around.

56:14So we're going to see this in developer tools. Well, I mean, Rails was like one of the first frameworks that I know that in the error page, they had a CLI that you could like use the logo context, you know? And I think like more of that, like in next you have the copy to markdown. Whenever you have an exception, you can copy the markdown, put it in there. That's the first sign. Yeah. Yeah. But I think there should, and in their docs too, you can like copy the markdown, but then it's like, you can only copy the markdown the whole page. And it's like, well, you know, maybe I only want to do this section or like I want to do one, two, three.

56:47I don't know. I think that's why the mentally fires of the world, Stainless, all these companies that do kind of like API docs and API generation from docs are like getting a lot of interest. I think you'll get more of that, but it's hard to get people to move over. You know, I'm sure you see it with like some of the Sourcegraph customers. It's like, how am I supposed to re-instrument this whole code base that is like 15 years old? It's true, but what we have said is we explicitly are building AMP for the people that do want to move. And that's been so liberating. And I think that that's the great thing about what you see in the market today, which is like you have all these companies that are like so AI first and like just use it and do great.

57:25And then you go on Hacker News and it's like, I've never got a single good result from AI. And I'm like, well, obviously that's not true. And like maybe the extreme is definitely true, though. I think to me, that's kind of like the thing is like the people that are spending$100 ,000 a year on AMP with two people, obviously they're getting value. It's not like they love burning money. But the people that are negative, to me, that's not always true because it's easy to be negative and like it doesn't cost anything, right, to put a comment that is bad. And so what's going to be the thing that forces the rest of the market to be whatever, man, let's just get on AMP and like make that work?

58:02They just have to see this work once or twice. You know, we've been in developer tooling for a long time with Sourcecraft, and it's always been hard for the last, say, 10 years to get a company to adopt a developer tool that does not immediately fit into their codebase. Because the codebase, that's the standard. Everything else has to adapt to our codebase and our processes and whatnot. What we're seeing now with agents is as soon as somebody has seen what it can do, they have such a multiplying effect or they bring so much value that people are willing to adopt the code base for this. Like the first time in how many decades where people are like, maybe our code base is wrong.

58:46Like maybe we should change the way we develop code to make more use of this. So I think people have to see this and then the agents will pull them along or like the, you know, the value that this brings will pull it along. Yeah, I'm curious. So I was on the board of a company called Launchable, which was founded by Kozuke and Kawaguchi, Bill Jenkins. And the idea behind Launchable was like, well, instead of running all of your tests, we'll use machine learning to figure out what tests are impacted by your PR and just run the small subset of them. and i think like what we found then the company got bought by clubbies but it was like in a lot of companies who go in there and they're like oh well how can we trust it though let's do a poc and then you do the poc and it's like it works great for the subset well you know work for the subset but like is it gonna work for like the whole test week and then you do a whole process and i think with coding it's like for some companies it's like they see a work on one task and they're like it's worth trying on every task and then there's another subset of companies that are like, well, you know, it works a little bit on the front end, but it doesn't work on like my Java service back there.

59:51So I'm not going to use it at all. I haven't quite figured out what's going to be the market pressure to make those people move along, you know, but it's like you said, it's like for some people it needs to work once, maybe for some other people, it's got to be one task that always feel my one task that I always use. We have built this kernel jam product, which is like an MCP playground and tester. And I have a task, which is like add YOLO mode, which is you know let a user toggle between auto running which sounds easy but it's actually quite hard without llm's work to like stop inference to approve a tool and then run it again and every model was failing until gpd5 codex and codex cli was like the first time i got it in like one shot it made the whole thing and i wonder if everybody should build some sort of like four or five tasks that are like okay if you can actually do this end to end then i'm like i'm in but i feel like people are still in denial of like, that's going to work.

1:00:43They don't want to have the conversation at all. If you look at that early adopter, the laggards, that chart of technology adoption, there's a reason why the early adopters are the tiny little start of the curve, you know, 3%. And it feels like so many of these arguments are people saying, well, what if we made a product that was for the early adopters, but somehow made the laggards also adopted early. Why aren't we going after that big market? It's the vast majority of the area under the curve. And it's like, because they fundamentally do not want what you are building. And maybe they should, maybe they're going to realize that, but you're not going to make them realize it.

1:01:23Or if you waste your time trying to make them realize it, you're going to be trounced by hopefully people like us that are only focused on the early adopters. It's a total mindset shift. And if you are just focused on building something for early adopters and you literally do not care and you set up your entire business and product to not care, not have to care about the people that are laggards, you can do a much better job. And that's what we're experiencing now. Let's talk about the outer loop, because I think that's kind of like the next step, at least for me. It's like, I think the coding agents themselves do great on a task by task basis, but then there's like, you know, PR review, which GitHub is like so slow and so clunky and it's so like order by file versus like, I think we should get to a world, which is like more semantic.

1:02:06It's like, hey, you know, these are really like the 50 lines of code that matter to look at and everything else is like, yeah, it's fine. You can like skim through it. How do you think about that when you want to, especially when you think about async agents, you know, there should be an easy way to spin them up, which I think it's fairly clear, but then I'm not sure if there's yet an easy way to catch up on what they're doing. You know, what I found when I use conductor, like VibeCamp, Ben, it's like a spin out five, six of them and I'm working on them and I kind of jump between them. And then my wife is like, let's have dinner.

1:02:38And then we have dinner and I go back and I'm like, what the fuck is going on here again? It's like, which one is doing what? And it's like, it's hard to like, just at a high level, see what each of them is working on, where it's getting blocked. Have you guys seen anything that works there? Have you been thinking about building any tools in that space? I agree, right? I feel this too. I think, you know, with our internal experiments, I think, you know, for example, this idea of well i just spawn 10 agents and they work and i control them i think stevie is doing this and he has like a whole workflow around this and it seems to work for him but for me i guess i'm a one tasker in my mind like i need to i can't do this like i cannot control five agents at the same time and then when i do it asynchronously i realize that i need to be really strict about how i review what they've done and that i also don't jump between them and then it's also you know making sure that you don't miss anything like i spun up so many agents and then haven't checked back on them because i forgot that they actually run so that's something you need to build in the product but yeah i don't think it's figured out you know like it's a there's so much to do still yeah it's wide open we think of it right now like if you're playing chess you can play one board at a time or the people in new york city central park who play against 10 different tables at once and they go and they sit down in front of the table they get oriented they make a move And then they go, and that's what we're trying to build.

1:04:03And it turns out, even if you've got a coding agent running in your editor in the CLI, and then it makes a big diff, you've still got to understand it. And it just becomes even more important when you have a lot running in the background. So we want to make it easier to orient yourself with what's the change. And there's a lot of stuff that is not in the realm of coding agents that would help, like having a deploy preview consistently available so you could just click and click through it. And then we want to make it fast for you to make a move and then, you know, get on with your next thing. Yeah.

1:04:30Or, you know, just UI. So at a glance, you can see, I don't know what it is yet, but at the first glance, so you can see what the agent actually did without having to go and read through like the emoji summary. Finally, we have it and blah, blah, blah, stuff like this. But to come back to your question of the outer loop, I think, and if Bjorn was here, he would talk for a long time about this because he's passionate about it, that the inner loop has changed a lot in that write, test, review, and whatnot. It's that you now review a lot more code. And what effect does this have? For me, for example, we don't do any formal code reviews on the AMP team.

1:05:09But it doesn't mean that code isn't reviewed because we use AMP to write 80 to 90 % of our code base. but that means everybody should review the code that the agent wrote so it's reviewed by at least one person right and that's not reflected at all in github yet like github is still based on this other mode where you tag somebody but then it's like well i actually went through two agents to produce this code and i reviewed it three times do i now tag five other people and right now we're stuck in this mode where people would say yes but i don't think it's going to hold that much longer Yeah. The other thing I noticed is like merge conflicts.

1:05:43Like I used to have very little because it's like, you know, I know what I'm working on and if I'm doing multiple tasks, I know how this is going to impact that. And I'm going to build towards it versus the agents, especially when you run them in parallel. It's like, they just start to change whatever is convenient to them. And then it's like across them, they're like changing the same thing. And so one thing we've been thinking about building is like, you know, how do you do better cross agent orchestration of like these changes? So I built for the GPT-5 post, it's like task manager, there's like CLI first, and basically any agent can like append what files they're touching, and then they can read what files other agents are touching and see what those diffs are to like implement them back.

1:06:23But then I think the question is like, well, maybe what they're doing now doesn't end up being the final thing. And now you're wasting all these tokens, like reviewing all these changes before review. I think at this point it's like, is Git well designed for this future role that we're going into? You know, there's like, I think everything is back on the table. I think maybe, you know, five years ago, it was like, you know, there was like a couple of YC companies doing, oh, we're like a new version control system. And I'm like, look, man, I'm not really interested in listening at this stage. And same way, programming languages.

1:06:51It's like, you know, when Chris Ladner even started working on Mojo, it's like, okay, because of AI, I understand why you need to build a superset of Python. And I think now with agents, it's like maybe clear why TypeScript should win because type checking is like very good for the model to like do self-improvement. What are like the other things? I think the interesting flex here is people assume that coding agents meet the bar of writing the exact same kinds of software to the exact same standard. And that is not necessarily an assumption that end users, consumers will apply. If they have software that's much faster, cheaper, much more personalized, if they can conjure it up on their own, then, yeah, you're going to tolerate if the loading state of this thing doesn't quite work correctly.

1:07:37So changing user demands and standards is an interesting thing that you can flex here. Yeah. What do you think about that? You know, we've been thinking about enterprise software moving more towards user-generated content, which is like, hey, you know, expenses are like a great example of like, you know, all these expense tools. where there's so many companies when like the core action that you're doing is like take one line of expense and tag it with different things but then you have to like set up all these categories and whatnot versus like just generate it for my company and for like each team separately because of different things and it's like to me that feels like more and more of that will become true and then the real value is like you know what's kind of like the underlying data store or like data stores that you're feeding into this.

1:08:22And I know some enterprises are building, already built kind of like internal like lavables, basically, where each employee can kind of like create a simple tool and then they connect the tool to like internal data stores. And they might be the only users of it. There's nobody else that does it. And I'm curious how you guys think about, I know that Bolt.new, for example, now has clock code integration. like where do you see the line move between like software engineers build software and like obviously amp is like a great tool for that versus going more upstream which is like any non-technical people can also plug into the code and like build things on top of it that feels in a way very different but also very similar in like the challenges that you need to solve for i think this idea of non-technical is the wrong way to look at it there are always going to be people that are good at unambiguously specifying what they want out of a computer.

1:09:15And we've had non-coders, including one of our board members who built something with AMP that replaced like 250K a year piece of software that he used for a lot of their internal fund tracking. He maybe took one computer science class. He hasn't really coded, but he's a really smart guy and he knows how to unambiguously specify what he wants to his CEOs, certainly, and now to a computer as well. So if you can get people like that, a tool that's really powerful, they don't think of themselves as a non-technical person. I think that's just such a bad mindset. So we want to build for the power user.

1:09:49And if that person has not been a coder, but they can pick it up really quickly, that's great. Again, we're completely focused on the people that know how to and want to get the very best out of this and that want the agent to win, that aren't trying to be like, oh, you know, nada, hey, it didn't do this thing. Tell me when it does. Yeah, we had this at the start a lot where people, whenever you have like an AI tool, I think there's a natural tendency by engineers to get it in a gotcha moment. You know, like, oh, I asked this and it didn't know this. And it's like, are you trying to get something out of it or are you trying to get it to fail?

1:10:26And, you know, it's not it's not worthwhile to build for somebody who doesn't want to fail. Yeah. Actually, if you fast forward how the world is going, you're seeing already over the last few years, companies have really slowed down their growth in engineering headcount. This is a global phenomenon. You're seeing engineers like here on the AMP team and other companies that are using agents really heavily. They're cutting out the middlemen. They're putting the people who are building the product closer to the customer because you can go and hear an idea from a customer. Literally in the meeting, you can kick off an agent to go and build it, and then you have a first draft of it.

1:11:00So overall, the person who's using the coding agent is getting so much closer to the problem. They're also going to share more in the rewards from solving the problem because without needing to share the profits with everyone else, there's naturally more to go to them. So I think if you fast forward this, it's not that the firm or big companies are going to completely go away, but you're going to have people that have an incredible vision in their head and that are so close to the problem and have an incredible incentive to go solve that problem. Equip them with a coding agent. If you're going to build a coding agent that those people want, that is way better and more valuable and you're creating more value.

1:11:34You're allowing more new things to be created in the world than if you were building a coding agent that is for the median developer that makes them 30 % better. So that's who we're targeting. And I don't think that that will necessarily look like vibe coding. Vibe coding is this really unproductive thing to discuss because everyone has a different definition of it. And too often it's having the agent write code with poor feedback loops and poor quality control. And I don't think that that's valuable, but it's giving that person the ability to build something truly great, really fast when they're so incentivized and they will have every desire for it to work well.

1:12:09And I know we're getting close to time, but a couple of things I want to touch on. So Thurston, I was reading through your blog. You left Sourcegraph a year and a half ago, then you joined back. Good job, Quinn. Bring him in home. Thank you, Thurston. But when you wrote a post about leaving, one thing you wrote is that when you first joined in 2019, one thing that Quinn told you is like, hey, Sourcegraph is your playground and you have skills and talents. And I want you to use those skills to move the company forward. How do you take this idea of like the power user getting close to the customer and like how people are going to build teams overall?

1:12:44Like there used to be engineering and product, like you were saying, the triangle that's kind of going away. What are like the type of people that you think are going to be most successful? Like how should people think about structuring teams? It's like obviously you're doing this with AMP in a way, right? You're like building a sub team and sub product within a larger company. Any tips that you have for other founders and executives? Thurston is incredible and AMP would not exist in any way without him. He has strong internal constitution of how he uses it and what's real and what's not. And it's so easy to get carried away with the hype, the possibilities, especially when you see other people, a lot of other smart people who are getting carried away by it.

1:13:25Thurston has this incredible ability to stay grounded. And that, with everything changing so fast, with it being such a hype cycle right now, That's really important. Also, just these first principles thinking, like how we've completely rethought how we build everything in AMP based on how should we actually do it rather than what has come before. Thorson is the rare person who's been at bigger companies, who's seen how Sourcegraph, how we build enterprise software in, you know, not the Google way, but in a different way, and has taken the parts of it that work and not the parts that don't. So all of that combined with someone who's an incredible engineer, incredible writer, communicator, that's a really powerful combination.

1:14:07So find those people. And then what I said when he rejoined is he is the dictator that made him feel really uncomfortable, as you can see. I hope you cut to his face. But that's exactly what you have to do. And had just put so much trust in people like that. And that also shows everyone else at the company that they can do crazy stuff, that they can go way beyond. They can take it to the extreme. They can make mistakes. And that's still okay because we're not trying to build something that's going to go really big in the current state. AMP is growing incredibly fast, but the most important thing is we're building the coding agent God, that thing in the future.

1:14:47And that's something that we're all in search of. So none of the mistakes, none of the successes in the month-to-month timeframe really matter. It's all about getting ourselves in the right trajectory and you got to do crazy stuff. So equipping Thorsten to do crazy stuff and to take the ideas that he has and make them scale up with all the reach that Sourcegraph has. That's been my goal. On the first principle thinking, how do you think about that? And so there's the world of evals and there's the world of vibes, right? Yeah. How do you approach it? Like, how do you look at the product and you're like, okay, this is good.

1:15:24This is bad. This is what we need to improve. Is there something formal that you guys use internally or is it mostly you as the dictator? Well, two part answer. I think the first part is to also answer the other question a little bit is what I've seen become more important or the shift I've seen is that, you know, I set the triangle of PM, designer, engineer. I think as an engineer or any of the three, you now need to know a lot more about the other parts. As an engineer, you cannot see yourself as the person anymore who types out a spec or turns a product PRD into code. I think you need to be aware of business.

1:16:06You need to be aware of a product. You need to know and have some taste for software. Otherwise, I think the value of your work will diminish over time because the pure typing out of code for most of the code, you know, exceptions being a John Carmack and, you know, whatever. For most of the code, I think the value will diminish. And we've already seen this, like, compare GitHub contribution chart today, its value to save two years ago, right? And to come back to the second part, like vibes and whatnot, I think we don't have any set evals. We don't. And this was controversial up until a week ago, I think, when I think Boris from, or two weeks ago from Anthropix said they don't have evals for the coding agent too.

1:16:50But we don't and we haven't had them. I've built evals before. I fine-tuned models before. I know that they're good. I love evals. I was addicted to LLM as a judge. I wrote about LLM as a judge. But for a coding agent who's supposed to work in many different code bases, who's supposed to work with many different types of prompt, who's supposed to work with many different types of tasks, it's a time investment that we cannot afford with everything changing and having to stay fast. And if you ship 20 times a day, you will get a lot of good feedback. I swear to you, I could tune my system prompt a little bit now.

1:17:25And then I would say by this evening, people on our team would go, why does it call this tool so often? Like, what's going on? What did we ship? And that's incredibly valuable feedback. And that's incredibly valuable, you know, when people dog food the product and use it all day. And how do I make these calls? I don't know. Like, I think it's experience of like, I think about software a lot. I love using software. I listen to a lot of business podcasts. I read a lot about business. I listen to a lot of software podcasts. I read a lot about software. And then I try to project like, what does the business need?

1:18:01How can we get growth to 10x? How can we get our users to 10x? How can I use my engineering capabilities to serve as a function of the business to reach those goals? How can I organize the team or get the team to help me reach those goals or together reach those goals? And, you know, it's hard to explain, but it's like, I feel like in this year, truly here at Softgraph, like everything I learned over the last, say, 15 years of my career is coming together in the sense that all of the hours spent listening to the Acquired podcast to help me as much as, you know, reading Hacker News for how many hundred hours and writing code for how many thousand hours, you know.

1:18:41And with code being now this tool that you can wield much easy or much fast or much more often, I think it's become much more important to how do you want to wield it and when and for what reason. I think the hard to explain is a great explanation why, you know, you just cannot one shot create these things because there's a lot of implicit preference. Awesome, guys. Anything to wrap call to actions? Are you hiring? Like who should reach out to you? requests for startups? What should people build that is going to be helpful to you guys? Yeah, I don't know. I don't know. We're always interested in talking to fellow engineers who are interested in agentic programming, figuring new stuff out.

1:19:21We want to hear from them, like what works and doesn't work. We're always willing to hire people with exceptional talents who are fully in this and realize that, you know, programming is changing a lot. And I don't know, What else? If you want to come on this journey with us and see where coding agents are going, then come along. Yeah. Use AMP. Send us your feedback. And we are just so excited. We feel like kids in a candy shop, just that we get to go build the future of coding. It feels like the final boss. Yeah. Nice. Thank you guys for coming on. This was fun. Thank you.

1:20:04Thank you.