How Warp ships 2,000 PRs a month with AI factories | Zach Lloyd (CEO, Warp)

0:00Can I give you a hard time that humans really are the bottleneck? Because if you look at kickoff to PR time, it's 35 minutes. But if you look at PR to first human review, it's three and a half hours. And when you're doing, I think it was like over 2000 PRs in the last month. Like how do you keep things in the team from going, as I say, like chaos reigns?

0:20Zach Lloyd:What is a software factory? For us, at least, it's an actual noun. It's like a product concept where it consists of a bunch of repos, a bunch of MCP servers, a bunch of configuration, and then a bunch of agents, essentially. So a code review agent design, different agents, different automations, and it's all defined in code. Scoring happens across all runs, but then there's a second loop, popular thing on Twitter right now called self-improvement where you have like an observer agent and it can then create updates to your factory that will try to prevent the particular failure mode one of the things that's most helpful is like you could have these factory agents do computer use verification so in this case it made a video i'm just like talking to this thing that is doing this job that i've done for the last 20 years and it's now it's like doing it kind of better than me spoiler alert the humans are the problem.

1:17Welcome back to How I AI. I'm Claire Vo, product leader and AI obsessive here on a mission to help you build better with these new tools. Today I have Zach Lloyd, CEO of Warp, and he's going to show us exactly what he means by the software factory. He's going to show us how you can kick off tasks in Slack, what it means to go beyond the software development lifecycle and how a technical CEO uses non-technical tools with AI.

1:43Zach Lloyd:Let's get to it. This episode is brought to you by DX. In a recent study across more than 500 engineering organizations, DX found that spend on AI tools has grown 28x over the last year. The share of AI authored code is climbing, but overall innovation has remained flat. As teams generate code faster, new friction in code review and validation is offsetting those early velocity gains. DX tracks speed, quality, and cost together across the software development lifecycle, giving engineering leaders clear visibility into how AI impacts delivery and whether those investments are translating into real value.

2:23Download the full report at getdx.com slash howiai. That's g-e-t-d-x dot com slash howiai. Zach, thanks for joining How I AI.

2:38Zach Lloyd:Thanks for having me. Excited to be here. There are a lot of trends on the timeline right now, but one that I think is going to be big for 2026 or seems to be big for 2026 is the factory. The factory, this like fantastical idea that we have that we're going to, you know, input tokens and output enterprise value, something like that. At least we'll output code. So I, you know, I'm excited for you to show us as CEO, as engineer, as builder, you know, what to you is the factory and how it looks inside warp? The factory is definitely trending. I actually don't love the term factory, but it is the sort of thing everyone is standardized on.

3:19Zach Lloyd:It feels a little bit sort of dehumanizing to me. But yeah, the idea is exactly what you said. It's like, how do you, in a world where we have these sort of magical agents kind of harness their power in a more organized way to have them basically build software for you from your ideas. And so, yeah, excited to show you how I use factories and what it means to me. I was at Lenny's summit recently and I think Marty Kagan also said he hated the word factory. I wonder if it's more like Santa's workshop, right? Like we have magical elves that craft wonderful things that spark joy and are delivered gift wrapped for us.

4:05So I'm going to say the AI, the AI magical, magical workshop.

4:09Zach Lloyd:No, show us, you know, what does, what does this look like for you? Yeah, you did, you did say, you know, you put in your ideas, you output product. Is that real? how does that really work yeah so it kind of works like that um the the way that i think of of factories is like there's there's basically two types of personas who are using them so there's i'll show you like basically how we do this in warp there's like the builder persona so the builder persona is someone who is contributing ideas wants to build stuff and from that persona it doesn't look like at least the way that we have it said it doesn't look that different from like how one of these engineers or designers or product people on our team might build with a local coding agent the big difference you can see here so this is like this is our slack uh is that most work is starting in a public place and so the way we think of factories is like you um you basically can say in a public slack channel you can tag we named our factory wilson um you can tag Wilson and be like, okay, I want you to, to build something for us.

5:18Zach Lloyd:And when you do that, uh, what Wilson will do is like, essentially do not just like the direct build, which is what you might get if you're using like cloud code or codex, but we'll also like do a few other steps. So what, what it, what it will do is it'll kind of go through all the steps of like, first it will triage whatever this is. So it'll treat it as like an input to the factory, not just like a direct thing to build, meaning it will, you know, for this one, Harry's saying he wants to change the way this feature looks. He gives like pretty detailed instructions. He, you know, attaches an image.

5:59Zach Lloyd:He says use computer use. And this kicks off a flow where the first thing that the factory does is actually kind of open an issue. So you can see over here, it's like we use linear in addition to using Slack so that everything gets tracked. So the factory integrates not just with Slack, it opens an issue. Then it does the actual implementation. For this one, it's like, it was simple enough and ambiguous enough that we could actually implement it. Then it does, it will create the PR so it integrates also with like GitHub then it does the QA which I think is like really important these days like if you want to not spend all your time doing code review one of the things that's most helpful is like you could have these factory agents do like computer use verification so in this case it like made a video of the completed feature so it's like showing the thing, showing all the keystrokes.

7:01Zach Lloyd:And so what you get here is not just like, and then it gets merged, right? It's not just like the single step of build the thing. Like if you were doing this with like a coding agent in like the pre-factory world, what you would typically do is like pull up your local coding agent, do the change there, test it locally, and then like push it up to GitHub. But you get the whole thing of like the ticket, the the code review pr the video all all done together and then the other like really um kind of magical thing is that it's all done in public so if you are you know someone else wanted to come in and like look at this and see how the task was done or even contribute to it and so we all have people like multiple people on these slack threads you get to a world where as like Like a builder, you're no longer sort of working in this local silo.

7:54Zach Lloyd:Instead, you're working in a public space where everyone can see what you're doing. And so that's like from the builder's point of view, how they like use the factory. Does that make sense so far? Yeah. And, you know, I think what I'm reflecting for folks who are maybe still trying to grapple with, OK, like what's a coding agent? What's a factory? What's the difference? I think what I'm hearing from you is the factory, quote unquote, is comprehensively designed to reflect your version of what the software development lifecycle should be. And so it's not just like idea to code to PR to push. It is, OK, an idea needs to go through several steps.

8:34It needs to go down the conveyor belt of product and into like the funnel of issue tracking. And then we need to code it. And then we need to QA it. And then we need to have these very specific verification loops where we can take our sticker and say quality controlled by, you know, number one, two, three. And so to you and the way you're describing it, where it goes beyond a coding agent or a co-pilot is that it is actually designed to take the very specific end-to-end product steps, not just the engineering kind of like input-output.

9:09Zach Lloyd:That's definitely part of it for sure. It's like it does more of the software lifecycle. So it does more of those steps. There's a whole other bigger part of it to me, which is, so I showed you this from like the perspective of the individual builder. If you were to go into our Slack, you would see like this is happening over and over, like all day long. The people on our team are building in this way. And sometimes, by the way, it's not just like the engineers who were kicking off work like this. It could be work that's being kicked off automatically. So, for instance, we have, I don't know if you know Sentry, but it's like a crash reporting system.

9:53Zach Lloyd:And so we have signals when there are crash reports coming from like our terminal app that we try to automatically fix those. And those go into the factory, too. So it could either be human initiated. it can be initiated by like an external system and it can be initiated actually in any of these tools so it's like you can initiate it like we could have started this whole thing in linear we could have started this whole thing in github and so it's it's integrated into all the tools but the bigger piece of the factory approach in my opinion is not like it's not so much how it the individual builders work because it's not all that different from individual builders who might be using like like might be going into the terminal for instance and using a coding agent to do something it's different it's in public but what's really different in the factory approach is that everything is like centralized in the cloud and there's a whole other aspect to it which is for not the builders, but for the people who are like the managers who are like trying to scale software development on their team.

11:05Finally, something for the managers.

11:08Zach Lloyd:It's what everyone's been wanting, right? Too much attention on the builders. No, but in all seriousness, if you're like, like my concern, uh, as someone who's running a company, it's like, I want, I want soft, like it's a super duper competitive market. I want to see how quickly we're moving. I want confidence that the way that we're building software is actually improving over time. I want to make sure that we're not like wasting too much money. And in the world of like all of like every builder has their own individual local setup, that's very, very hard to get. And so this is like an engineering manager's dream.

11:49Zach Lloyd:It's like a, it's like a CTO's dream. Not that this is the coolest thing in the world, but it's like now I have the ability to sort of centralize and see what everyone's doing. So, for instance, on this screen here, it's like these are this actual data from our team in terms of like how we're using the factory. It's like there's a sort of sense of like how automated is the factory? What's our velocity? How long does it take us to like ship stuff? And so there's just all these measurements. Can we pause real quick on your productivity? Because I actually haven't seen this measure before. And we've seen a lot of measures, which is human interactions per PR.

12:30Just talk us through why that, because honestly, I haven't heard that one before and I like it.

12:36Zach Lloyd:You know, this sort of instinct underlying this is that we are going to be not to put this the wrong way, but it's like humans are a little bit of the bottleneck in terms of production itself. They're also the creative force. But in general, what companies want and we're trying to build for companies that work largely is like, how do you automate more? How do you make things that can truly just be done agentically be done in an almost fully automated way? And so, you know, the general instinct is like the more times you have to prompt, the more times you have to steer or like cajole your agent, that's going to be a limiter on throughput.

13:21Zach Lloyd:So this is where you start to get into like factory world. Like this is like imagine you're running like a Tesla plant or something. It's like how many times do you have to stop the line? And so, you know, we're trying to give engineering leaders a view of like, okay, how are you able to sort of make things more efficient over time? And quick question. Are those just because my mind, I mean, I, you know, as a once one time CTO, I'm like, yeah, this is exactly what I want. Are these interactions? Do you think of these interactions as like the prompts, the sort of like upfront steering? Are you thinking about like how many comments on PRs, how many loops, like how inclusive is this like interaction per PR score?

14:04it's it's all of those so because the and again i think this can definitely be refined over time

14:13Zach Lloyd:and bear in mind we're like trying to figure out so what is the right set of metrics but this actual metric because the factory is like integrated into all of your knowledge work tools it includes all those things like how many reprompts in slack how many comments on linear How many times did you have to correct the thing in code review? And so it's like a kind of like, yeah, it's a proxy for how much work did you have to do in order to, you know, get the thing to do the job. Okay. Wait, I want to give you a hard time. One more thing. Yes. Please do. Before we started recording, you said like how technical.

14:48And I was like, okay, well, we're going to put on the CTO, at least the CTO hat right now. Okay. Can I give you a hard time that humans really are the bottleneck? because if you look at kickoff to PR time, it's 35 minutes. But if you look at PR to first human review, it's three and a half hours, three and a half hours. And so it's like so funny that still cycle time, man, like it's it's the thing you really have to focus on. It's that it's that number. Yeah, there's still the bottleneck.

15:20Zach Lloyd:And we're, I mean, we're still doing human code review. Do you do all your PRs get human code review? Currently, all of our PRs get human code review. Wow. And so, now this is like, I think, a team choice, an organizational choice. Yeah. We have, the one thing that we have changed around this is not, like, we used to require, like the workflow used to be like person a on our team would build something with an agent and person b would review the agent's code we no longer require that like the person who prompts the agent can also review its code okay yeah you're making me feel better that's a better thing but it's still like we you know we we don't have yet complete trust i think the way this will evolve is like some percentage of some percentage of stuff will um will eventually will feel confident enough that we can skip that yeah i did an episode recently i built um an eve agent called merge mommy um and this is this this is the flow that i often see and even mature engineering organizations is what you do is you basically like risk score every pr automatically so go through your risk score and then anything that is low or extra low risk gets a stamped approval from merge mommy and then a human's allowed to just smash the button and merge anything that's like medium high or whatever or has some like outlier on risk requires human review and that just like lets you get that bottom tranche of the queue out um also just so you have good capacity for high quality human review on the things that really matter?

17:02Zach Lloyd:100%. I think code review becomes an exercise in risk management. I think that's right. I want to show you some other stuff in the factory just to show you how this is different than just standard interactive agents. So you get cost is on everyone's mind right now. And so you get like this is Asian cost. It doesn't factor in the sort of cost of the people at the moment. But it's like, you know, you can see like we were really expensive a few weeks ago. We made some changes to our model configuration and we've driven this cost down and we want to continue to drive it down. But just like having the centralized view for this across your whole team.

17:44Zach Lloyd:Can I ask you a question of cost? Quick question of cost. So, I mean, this is a pretty, you can see the drop here. I see this a lot when I'm talking to engineering organizations. they often see like a rise in cost of PR as their per PR as they're adopting AI, which you all already have. And then we see this drop as you're doing optimization. Do you feel like the biggest lever here is model right now? It's like that. Is that the lever? Yeah, it's it's model. I think model is the biggest. I think like. Context, like the way context management, like context management matters as a secondary thing um i would say model is the biggest yeah um and the the the way to really figure this out actually is to test which is something i want to show as well but um basically i think i think model is is the biggest one um any other questions on the just like the spend view No, that's great.

18:48Zach Lloyd:So the other thing, like this is where it really gets kind of factory oriented is like the way that we're changing development. And this is, I think this is most relevant for folks who are watching. We're trying to manage like these teams and scale coding agents is like we now, we're like measuring everything. And so this last thing in here, which we call scoring, basically gives you a view across all your agent runs of like how they're doing on different dimensions. And so we basically, because the factory is like a closed, kind of like closed loop system, like every time an agent does a task in the factory, that task gets recorded.

19:36Zach Lloyd:and what that opens is the possibility for you to go and like like go back retroactively and look at how well the task was done and so you could do this as as a person you could literally just go back and look at tasks but the other thing that you can do is you can have agents do this which is is what um is what we do and so for instance let's see if i can find an interesting one um like here's an interesting one like redundant tests so if you're if you're working with um you know what i'm talking about here it's like oh i know what you're talking about every pr i push it's like i have run and set up 135 tests exactly it's it's not it's like it's it's like and the reason it does this is because uh you know they're writing tests not to like prevent regressions necessarily but also just like test behavior along the way.

20:31Zach Lloyd:And so you end up with a bunch of tests that you don't need. And so if you take the factory approach, you kind of know this might be a failure mode. And what you can do is you can write essentially a score that uses LLM as a judge to be like, okay, I want to go and look at all of the agent runs. And I want to see how often an agent thinks, a different agent than And the one that did the task thinks that there's redundant tests. Does this make sense? Yeah, totally. And so you basically are like, okay, I'll pick a judge model. This is like a kind of medium smart judge model. You don't want it to be too expensive.

21:10Zach Lloyd:Otherwise, you end up spending a lot of money on scoring. You have it classified in terms of like for this task. Like how did it look? You pick like a sort of sampling rate in terms of how you want to do this. And then you get over time a set of runs where you can see that like sometimes this agent thinks that there were some like surplus tests. And so this is what I mean by like the real measurement. Then what you can do is like you can go and you can actually, you know, as a human, you can go and kind of look at what's happening here. But I'll show you there's also a better way to do this in the factory world where it's like you can actually use agents to sort of identify what's gone wrong in these runs and try to improve the factory.

21:58Zach Lloyd:Does this make sense? Yeah, I'm curious. Do you do this on a per run basis or on an aggregate basis across a set of PRs? Aggregate. Great question. So, yeah. So what we do is like the scoring and this is what I call it this thing. We call this like the score. The scoring happens across all runs. But then there's a second loop, which is another like popular thing on Twitter right now called self-improvement, where you take an agent and you basically say, OK, for all of the failed runs and you need like you need like a real sample, like maybe 20, 25 failed runs, you need some significant sample size.

22:39Zach Lloyd:Otherwise it starts to overcorrect based on single, based on single things. You have it look, you have like an observer agent look and it can then create updates to your factory that will try to prevent the particular failure mode. And so let's see if I can find one that's like, I don't know this, I'm just literally picking a random one here. but um this is finding some some issue with our skills that are driving the factory it's presenting evidence and it's saying okay we should change the definition of one of our factory agents in a particular way and if you go under the hood and look at this it's like it's changing this like step 10 of what our factory agents should do this makes sense yep and so this is um to me this is really exciting like the way the thing that's like enabling this whole thing to work is that um you define the factory in code yeah and so you know what i mean by that is like if if you were to look at this sort of definition you would say okay the factory what like what is a software factory it's like it's like for us at least it's an actual noun it's like a product concept where it consists of a bunch of repos a bunch of like mcp servers like a bunch of configuration and then a bunch of like agents essentially so like a code review agent design like different agents different automations um and it's all defined in code and the the value of doing it that way like in warp factories is that you get the ability to actually um you can like test different configurations and know like you're basically freezing the state of the factory at a given point so you can be like okay if we were to to change the factory definition then like run all these tasks again we could see if things were better uh it also makes it so that like um an agent can actually because these are all it's all code like coding agents can actually update the factory to make it better.

24:53Zach Lloyd:Does this make sense? Yeah. This episode is brought to you by Open Art Arena, the global leaderboard for creative intelligence. Every week, new AI models launch and everyone claims to be the best, but best at what? Open Art Arena is built to answer the question that actually matters. Which model is best for your specific job? Instead of one One overall winner, Open Art Arena ranks models across real creative tasks, from advertising and film to animation, product, graphic design, editing, and lip sync, covering both image and video. And these rankings aren't based on hype, they're judged by professionals, industry leaders, and working creators through blind evaluations.

25:40So judges never know which model produced which output. That means you can see how models actually perform when it comes to the creative work you're doing. So stop guessing which model to use. Explore rankings based on real creative work and find the right model for your project and save time and cost.

25:59Zach Lloyd:See the rankings at OpenArt Arena. Yeah, I just want to like, you know, kind of sum up where we are so far because I think there's so much rich stuff in here, especially for engineering leaders and builders. And then I know we're going to get to some workflows that are not engineering focused, which I'm excited about. But, you know, A couple trends, themes that I've seen as you talk through this. One, work happens in public. So I'm seeing this move towards instead of work being assigned in tickets and then tickets being worked on on laptops, work is happening in public, whether it's Slack or some other channel.

26:30And then executed in the cloud. So sort of anybody can interact with that. The factory does have this software development lifecycle kind of like definition to it. But on top of that, it has this aggregate meta analysis that you're doing across all the behaviors, which is giving, you know, as we said, giving the people, the managers, what they want, which is, are we getting more efficient over time? Is this factory actually effective? How many humans? How many agents? What's the inner balance between the two? Spoiler alert, the humans are the problem right now. And then what you're doing is you're also doing these evals against key behaviors in your factory that you want to correct.

27:14And this is something that when I talk to engineering organizations, I tell them all the time, which is such a challenge when you're using something locally, like, for example, Cloud Code, is I say you need session level telemetry because you need to be able to aggregate up the failures within sessions across your engineering team. And so some teams do this truly by like sucking up every local coding session to S3 and running their evals kind of like on their own in their own platform. But I do believe that if you do not have session, tool call, MCP call, test failure, computer use level observability into every single coding session, you're missing a lot of opportunity to optimize efficiency, cost, model, just like how your team uses the tools.

28:05And so I think it's super important that people do this. And then what you've added is this extra layer of great. Then take those insights and make the factory better. However, you define better.