Making the Case for the Terminal as AI's Workbench: Warp’s Zach Lloyd

17:09Zach Lloyd:Oh, my God. So we're still not fully out of this. Yeah, I mean, I could just explain the whole thing here. So our initial pricing was basically you do a subscription and you get a fixed amount of AI credits every month. and we priced it so that this is when we were at smaller scale. If you like fully utilized your plan, it would cost us money. But the hope was that the on the sort of on the average utilization that we would make money. Right. So it's like you have a plan that gives people 50 ,000 credits and most people only use 20 ,000. You can kind of price it around that. What happened was like the people just use more and more.

17:58Zach Lloyd:And so we got to a point where we were losing more and more money. And so from a company strategy standpoint, we had a choice. We talked to Andrew a bunch about this. Like we could either kind of play the like, and like we're growing really fast. Like the revenue is growing. You know, we're adding like a million in revenue every, it's since slowed down a little. It was like every five days or something. And it's like, we could play the game, go raise more money, but the margins were really bad. And so we decided that wasn't just like wasn't the smart long term strategic thing to do. And like also not like a race we can win to the earlier conversation here.

18:39Zach Lloyd:Like we we just can't beat people if the thing is cost. And so we wanted to know, like, are people paying for value? Will they pay if if we are margin positive? And so the way that we have like changed the pricing is so that it's much more consumption based. So you now pay for like a base plan of 20 bucks a month and then you buy credits on top of that. And we ensure that that like, you know, it's like in the old world, we didn't want people fully utilizing their AI because it would cost us money. Now it's like much better if people use more AI. It is more expensive for sure. And we've had a lot of user complaints around that, which sucks.

19:22Zach Lloyd:If any more customers are listening, like it's a bummer. It really does suck. But it's like we just could not afford to keep subsidizing the way that we were. And like all in all, I would say it's gone pretty well. Like we're still growing pretty well. And now it's like a growth that is sustainable and not like a, you know, unsustainable subsidized revenue growth. so tricky thing to do would you ever train your own models i think it we would definitely do like what um some of our competitors are doing where we would fine-tune models and do rl and that type of stuff i um it's hard for me to imagine us competing with training like a full frontier level model just the amount of capital that costs we do have a ton of interesting data like I think it's like actually a really interesting strategic asset for us in terms of like the workflows people are doing in the terminal how to improve them how people are interacting with our agent um so I think it's likely that we will we will do some sort of rl uh and I think it's also very likely that we are going to lean more into a mixture of models and more model routing to try to like give users the best experience when it comes to sort of latency see cost and quality which are the three yeah the three sort of like vectors here and do you see

20:49Sonya Huang:your role as kind of like optimizing that on behalf of users do you want to see yourselves as giving all options to users for them to pick yeah so our philosophy has been like make a great

21:01Zach Lloyd:default but then because these are developers they want control so we have like a we actually a couple variants of default. So we have like a default that's geared towards like efficiency and one that's geared towards performance. And then after that, we give people the raw choice. The raw choice is like a little weird because like increasingly we really want to like use different models for different things internally. And it doesn't map that cleanly under using say like, you know, GPT-5.2 for everything. So it's a little complicated, but I think it's actually something that developers like is the control.

21:40Zach Lloyd:And so I don't see us moving away from that right now.

21:43Sonya Huang:Yeah. One of the most interesting parts of where you sit is that you can actually see which models different developers are using. And so I'm curious in your user base, which models are most popular? Has it evened out a lot? And are there different flavors or personalities of what the different models are good at?

22:00Zach Lloyd:Like 70 to 80 % of our user base will use whatever we set our auto to and not touch it. And what we set the auto to currently is like it's a different one for efficient, a different one for performance. It's a mix of the codex. Sorry, GPT-5 too, but it's related to codex. And then Sonnet 4.5. When people are opting into choosing a model, lately Gemini 3 Pro has been very popular. It's a really good model. You know, what we will do is like we will test different variants in our auto model and see how people respond, how they engage with them. And I think we'll probably test Gemini 3. I've been impressed with it.

22:51Zach Lloyd:So, yeah, I would say if I had to like stack rank, I would probably say like the anthropic models are probably still most popular. And then between Gemini and OpenAI, there's a decent amount of people opting into each of those.

23:04Sonya Huang:What about Grok?

23:06Zach Lloyd:grok is not in warp um it could be in warp they've reached out a bunch of times to put it in warp we we i'm not i'm not at all opposed to putting in warp it's just like every time we put a model in warp we have to i like with like some concrete benefit to users and because it's a bunch of work to to tune our harness to work well with a model hmm i see interesting can you say a word more

23:33Sonya Huang:about that harness like what do you do uh to make your harness good so harness is like how you how

23:38Zach Lloyd:you prompt what tools you make available how it um how you manage context um and so like the big things that like are determining quality of harness are like it's literally like the language of the prompting it's the tool set definition um it's things like handling the context window so specifically when do you use something like a subagent where you where you go out and have something as a separate context window when do you summarize when do you truncate like we have things that have like you know you might run a terminal command that has like gigantic output and you don't want that all on your context window but you might want some part of in your context window so how do you sort of pick out the right stuff how do you do rag how do you integrate with mcp and so it's um there's just like some engineering and like alpha that goes into that the way that you make that good is by measuring uh like you can start just by like in a pretty like naive way and just give it a bunch of prompts but the way that you make it really good is by measuring and by measuring you can do it with um sort of like a fixed set of evals where we know what the results should be and like not all of them should work.

24:55Zach Lloyd:And so we have internal evals. You can do it on public benchmarks. And that's actually been really good for getting our harness to be awesome. It's just like going through the exercise of making our agent perform well on the public benchmarks. And then you can do it by looking at like, you know, user data. And so we use brain trust. So there's various platforms you can use. to like sort of look for patterns and failure modes in the agent interaction and then try to tune the harness and sort of replay them as evals. So, you know, that was a big mindset shift for us like to get to doing that, but that was 100 % necessary to do it all data-driven to get something that was good.

25:34Sonya Huang:Yeah, got it. And do you have your own tab autocomplete models? Is that just not even relevant for your product?

25:41Zach Lloyd:We don't have that. It's not super-duper relevant at the moment. Like, I think there would be incremental benefit if we were doing tab completion in the terminal for people. And I think for the hand-editing parts of Warp, by far the more typical use case is for code review of an agent's code than it is typing code. But it would be nice to have. It's just not an area of high priority for us.

26:10Sonya Huang:Yeah, got it. Awesome. I'd love to talk a little bit about how you see the future of coding interfaces and seeing the future Workbench evolving. So it seems like very much like you believe there's a convergence happening between kind of the traditional, call it Microsoft Word, IDE, GUI approach to writing code and then the chat with your computer agentic kind of style of terminal first approach. And those two are starting to merge. What other kind of UI innovations do you think are happening in terms of how people work with coding agents?

26:47Zach Lloyd:I think the biggest change that we're going to see in the next year is more and more like kind of like cloud agents or we call them agents where and this is already happening. Like we're investing in this at Warp where rather than a developer sitting at a keyboard giving a prompt, there's some system event that triggers an agent to do something. and that system event could be like you have a server that's crashing you have a cluster of user reports um someone has reported like a filed like a security incident against you and all those things are basically going to serve as context into an agent that gets launched and runs not on some individual's machine but like somewhere in the cloud and so what i think that implies is that you're going to want um the the sort of like workbench to become more of an orchestration platform more of a uh like kind of cockpit for managing not just your own agents but your team's agents i really think it implies you're going to need like a strong team concept um because you know the these things aren't it's not going to be the normal workflow of like i'm sitting at my desk like i i'm writing a coding change and i push a pr it's like agents are going to push prs And then agents are probably going to leave an initial round of reviews on PRs and they're going to file tasks in your task tracking system.

28:13Zach Lloyd:And so all this stuff needs like it needs tracking and it needs coordination and you need different ways of integrating it into existing systems. And so whoever, like, this is like what warps, probably like our biggest product focus for next year is on this type of evolution off of just like interactive agents into the cloud agents. Because I think it's going to be pretty transformative.

28:43Sonya Huang:And I imagine that's like a massive infrastructure push to be able to kind of, you know, run up and spin up. It is.

28:50Zach Lloyd:It's, it's, it's turning. Yeah. So like for us, it's turning us much more from a, like a product to a platform. And so the way that we think about building this out is building it on different layers of the stack where you have like an agent SDK, you have agent hosting if you want it. So like, you know, if you're a smaller company and you don't want to like set up spots in the cloud for your agents to do their work, Warp will host that for you. There's a whole category of startups that are going into this business of agent hosting, which I think is really interesting.

29:26Sonya Huang:It speaks to like this is like a real thing that's happening.

29:30Zach Lloyd:There's an API layer for once you have the agent running, how do you get its status and how do you how do you maybe take it over or see its progress? Where is it right? It's logs. And then there's like a management layer of like, what are all these things doing? What states are they in? What's the log? Like, you know, who started them? When did they produce PRs? And so I think it's cool because it's going to be the most impactful way to use these agents a lot. It's just not to have a person like driving them. I don't think that this means that the person driving the agent is going to go away either.

30:03Zach Lloyd:I think that there's like, it's the sort of types of tasks that are going to start with these ambient cloud agents are more like toil tasks or things that are one-shotable. And I think harder, more interesting software engineering is still going to be done by a developer like at their workbench. But yeah, this is how I see things evolving in the next year.

30:23Sonya Huang:That's awesome. And I think one of the more important scaling law charts is like the meter, like how long can the agent run for?

30:31Zach Lloyd:Yeah.

30:32Sonya Huang:what are you seeing in terms of how kind of long horizon these agents can be

30:36Zach Lloyd:i don't know at its max i would say like doing real coding tasks for us now is like 20 30 minutes something like that maybe like you can have it run longer just to be clear but the problem is it will start going in circles still there's still context limitations and like it's a it's a costly proposition to have agents running without people checking in and guiding them and just by far you get the best results when the agent is really steered so when you do like an upfront plan with the agent when you check in on the agent's work and so So I think, yeah, I think this will just keep on going up.

31:26But the, I don't know.

31:29Zach Lloyd:It's like hours and hours of work. It needs a really clearly defined task in order for that to even make sense to me. It needs to be doing like some like big code migration or some big, big task.

31:39Sonya Huang:Got it. What do you think the product might look like when it's good at kind of being this cockpit to manage, you know, swarms of these agents?

31:48Zach Lloyd:Yeah, we're, so we're building this out right now. And like we are having all sorts of internal debates on whether it's it's it should be one product or two products. The way that we're doing it right now and how I think other people will probably approach this is like a sort of like area of our app, which is about agent orchestration. um the reason i wonder if it should be a whole separate product for us is because like um you know it's it very much it feels more web-centric to me which we we can make warp work on the web but it's not the primary interface um it feels like potentially it has like a different user some of the time yeah um but the advantage of having it bundled into warp is that it makes the handoff from one of these cloud tasks to a developer extremely seamless.

32:45Zach Lloyd:And so very, very common workflow. Like we have this thing, like we have it running in our Slack and our linear. And so what will often happen is like you'll tag something in Slack and you'll be like, you know, can you make this fix for me, change this button position or whatever. And right now you need a developer to like tie the loop on that. So it'll do the work in the cloud and then you'll bring it onto your local machine. And it's very nice to have that be in one environment where you can just keep working on it seamlessly. So short answer is, I don't know, but it'll be a little bit more like a task management UI.

33:18Zach Lloyd:I don't think it's going to quite like, I know there's, there's like thoughts of like, is task management the primary primitive for developers to be working with? I don't buy that either. Like, I don't think like every developer is going to be doing all their work out of linear JIRA. But I do think there's some aspect of seeing what the agents are doing across various systems that developers are going to want.

33:38Sonya Huang:Yeah. Awesome. I'd love to close by maybe talking a bit about the state of agentic development and how the software engineering market will play out.

33:47Zach Lloyd:Sound good? Sure.

33:49Sonya Huang:I guess maybe for starters, where do you think we are in terms of like the frontier model, you know, capability frontier? Like where are the models good today where are they not um you know yes still producing all the errors like where are we

34:06Zach Lloyd:so i i'm constantly using this stuff uh i'm somewhat biased because i use it on warps code base which is like a very custom big rust code base but i think that's still an interesting perspective um the agents can do what i would think of as like medium complexity tasks pretty well if you give them a bunch of guidance uh they can't do like whole big projects um at least we haven't had success doing that they can't uh i don't trust them to make like very fundamental architecture uh decisions for us um so it's like you want like pretty constrained tasks but they're well beyond doing trivial tasks, like change the button, color, take the text.

34:56Zach Lloyd:Like they can make apps. They're very good at zero to one. They can solve like kind of hard bugs. We have a medium sized feature. Like, I don't know what a good example would be. Like I was adding a new slash command to warp the other day. And it's like, I just tagged the agent to do that, you know, in Slack and it made a 300 line PR and it was basically right. And so I think there's a bunch of headroom at the upper end.

35:25Zach Lloyd:If I had to put it on like a scale of zero to 10, I think we're at like a six, maybe. So I think it's like, it's real. It's game changing for how people work. But it's not at like the level of doing what a full time engineer on a hard product needs to do.

35:42Sonya Huang:And where do you think the bottlenecks are? Like, is it just people, you know, the models don't have enough context. We need to get better at giving them instructions. Is it just we need to keep scaling these things up? What are the biggest bottlenecks?

35:53Zach Lloyd:So I don't think context window is still a big issue. And even with the bigger context windows, it having like attention over the whole context window in a reasonable way is hard. I think like there's like an issue of it always having to like relearn everything. Like memory is not just seems like a slow, inefficient, repopulate the whole thing with a bunch of files. Take like there's no like continuous learning with it. So it's it's like this big stateless thing where you're kind of always starting from scratch and have to fill it up before you can set it loose. That's that sort of stinks. I would I would like to see that solved.

36:39Zach Lloyd:um there's still like how do you use it effectively as a developer we're very early like this stuff didn't exist a year ago and so how should you be doing context engineering how should you be setting up your projects so that agents can work well with them um that's like a problem we if you were to look across how people on our team use warp to build It's like high variance. And, you know, that's not great because it's like we have very, very great. We have very, very like rigorous standards around writing code and like almost no standards. I mean, we've tried around like how to use the agents.

37:20Zach Lloyd:No one has been taught how to use the agents. There aren't even agreed best practices on how to use the agents. And so I think that's pretty nascent.

37:27Sonya Huang:But yeah, got it. My experience whenever I try to vibe code a little bit is that the coding models still produce a lot of errors.

37:36Zach Lloyd:Yes.

37:36Sonya Huang:Is that going better over time? And that seems like to me in the category of stuff is like if you can verify it, like did it work or did it not, you should be able to RL it. And it's like, where are we today in terms of the state of, you know, how frequently are errors coming out? and like, can we actually RL that or am I misunderstanding something?

37:58Zach Lloyd:No, I think that there's still definitely producing errors.

38:04Zach Lloyd:It's interesting. So it's pretty infrequent that the agent at this point will produce something that doesn't compile for me, which I think is an interesting milestone. So like, I don't know, not that long ago, four or five months ago, that was a problem. Like getting to a compiling version of the thing. It compiles for me about 100 % of the time right now, which is amazing. It produces stuff with bugs and errors relatively frequently. I don't think it has a good way of closing the loop in terms of does the thing work. and so i think some version of browser use or computer use where the agent can not only make the change but verify the change from the user's perspective not the code perspective is pretty important are people doing that yet yeah i like we're working on stuff like that like the the the computer use all of the model providers are have like beta versions of like computer use APIs and, you know, browser use for sure, computer use we're looking at, like, I would be surprised if this wasn't a thing.

39:14Zach Lloyd:And I think it becomes even more important of a thing pretty soon, as more work is done remotely, because the real pain in the ass with the remote work is verifying that it works from a user perspective. So I think that's like a big part of it. And then I think if you have that loop, it's probably easier to do RL and get to things that are behaviorally correct, not just like static compile correct.

39:38Sonya Huang:Yeah, yeah, absolutely. Okay, well, looking forward to that. And then I guess, do you think that we're going to reach a super intelligence moment here? Like where the models are better at coding than the best human coders?

39:49Zach Lloyd:I have no idea. No idea? What I do think is going to happen is I think, I don't know if this is super intelligence, I do think like coding will be solved by models. And what I mean by that is like, I think that the limiting factor that we're going to come up against is just like expression of intent from from humans in terms of like, what do you want built? How do you how do you build it? Like, how do you express that clearly? Like, English is ambiguous.

40:20Sonya Huang:Isn't coding the truest expression of intent, though?

40:23Zach Lloyd:Yeah, but the problem is we're moving from a world where people speak in code to one where they just speak in English to try to build apps. And so we're like reintroducing ambiguity because developers, people building apps are no longer actually directly expressing what they want. They're going through this translation layer of telling it to a model what they want. And then the model produces the code. So it's an interesting, it's like an interesting step backwards there in a sense, but it's also way, way, way more efficient to do it this way. um yeah i think like we'll get to a point where you actually don't need to be on the frontier to have something that produces code that is as well matched to a person's intent as possible uh so i and i think that actually is an interesting thing from a competitive perspective i wouldn't want to be in the api business for coding tokens because i do think like Like at some point, you just won't need to be on the frontier and you're not able to charge a huge margin on top of it, which is why I think actually you see Anthropic and OpenAI and Google going so hard at the application layer because there's huge risk at the API layer.

41:39Zach Lloyd:It's just for this vertical in particular that I think things are basically solved within a few years. Yeah. I don't know that. That's just I'm prognosticating.

41:48Sonya Huang:Well, that's awesome. Do you think that people will ever, are people already thinking about the amount they spend on coding tools being, you know, the replacement of what they would be spending on, you know, hiring a few software engineers? Or are they thinking about in their heads as buying a tool still?

42:08Zach Lloyd:So when we talk to enterprises, it is still viewed as like, by and large, as like a productivity boost. Yeah. And that's like the way that it's being evaluated. In fact, it's really hard to measure even what like the effectiveness of this stuff. And so it tends to fall back to subjective measurements from engineers. Like, do you feel like you're getting a bunch of value out of this or not? Or maybe you look like Dora metrics or like, it's really hard to like to know. So I don't think that they're viewing it yet by and large, at least as as labor spend. And I think today, if you pitch like, here's a$200 ,000 agent to replace your$200 ,000 engineer or whatever, they would be like, what?

42:50Zach Lloyd:Like, no, like not even close. But I would expect that this starts to change.

42:58Sonya Huang:Why do you think will change that?

42:59Zach Lloyd:That's a great question. I think it's like increasing the automation use cases. Or maybe another way of thinking is like if companies start to launch products without engineers, I think that that will be like a major proof point. And to be clear, I don't want this to happen. I'm like an engineer at heart and I don't want people losing their jobs. But there will be projects, products that are launched where there's like very, very minimal engineering involved. And you're going to look at the spend for that and be like, OK, this was the cost of delivering the product. and you're going to be like, okay, with and without engineers, what's that like?

43:40Zach Lloyd:So I think you need more of that to happen. I don't think that's happening very much yet.

43:46Sonya Huang:Got it. And then maybe last question. I'd love to chat about how you see coding as an art form and therefore, you know, your role in the world evolving. You wrote this blog post I loved back in 2023 i think um everyone should go go give it a read it's called uh i think it's about the future of productivity interfaces being ask and adjust maybe say a word on that and and how you think you know three years in how you think that's evolved yeah so i wrote this like pretty shortly

44:18Zach Lloyd:after chat gpt came out and we started um like trying to deeply integrate it into warp and the the idea was this sound really obvious right now but the way that like productivity interfaces have always worked in the past was that that they were geared towards hand editing right and by hand editing um it could be like you go into figma and you're like drawing vectors or you go into google sheets and you're entering cells or you go into vs code and you're typing code and my thesis in that article was like that's going to change to a point where the primary interface is one is a i didn't have the word agentic at the time but i it was like ai based where you would ask uh ask the app to do to make the do the thing for you and then you as a human author would be responsible for adjusting and adjusting might mean like reprompting or it might mean free prompting failed.

45:21Zach Lloyd:It might mean like going in, like treating the prior hand editing interface as like using that to like complete your change. And I kind of think that's where we're at right now for a lot of like, especially for coding, it's really transitioning to you start by asking for something and then you adjust it. And another thing I said in that article, which I don't know if it's right or not, was that I was thinking about, are you going to be able to get rid of the adjustment piece? And my thesis was that the area where you're going to need the adjustment piece the least is in areas where there's a lot of acceptable solutions.

46:03Zach Lloyd:So that would be creative domains. If you ask for an image of something, there's probably a thousand outcome, a thousand images that might work for you. And so you can just reprompt, reprompt, reprompt until you get what you want whereas for something like code um or a spreadsheet where there's one thing that needs to be right that you would have to keep that ability to like get it perfect with a hand editing interface so that was the thesis i think it's it wasn't bad i think it's

46:33Sonya Huang:held up okay not bad um yeah i guess you didn't coin agentic editing back then no yeah the thesis This is spot on.

46:43Zach Lloyd:You want to know something we coined at Warp? What did you coin? Which we should have trademarked is agent mode. So we were the first product to launch a branded thing called agent mode. And if you look this up on ChatGPT and just ask where did this come from? It came from Warp. And now that's a very common way of describing the feature, which I wish we were getting some kickbacks for that or something.

47:11Sonya Huang:Totally. I love it. Well, thanks so much for coming on to share what you're doing and your observations on the coding market as a whole. It's such a white hot competitive market and the way that you think the terminal will be the workbench of the future and how it's going to evolve. It was awesome to have this chat today. Thanks, Zach.

47:30Zach Lloyd:Thanks, Sarah. It's awesome to be here.

47:41Thank you.