Why the traditional pull request has a target on its back | CircleCI’s Rob Zuber

0:00Welcome back to Dev Interrupted, brought to you by Linear B. Quick thing before we start. My guest today is Rob Zuber, the CTO of CircleCI, and one of the most familiar voices on this show. And the idea that stuck with me from our conversation was how the pull request is really starting to feel like a relic. Because when machines write most of the code and expecting humans to read every line stops scaling. And Rob's version of the future is a system that tells you what the machines are doing with a high level of accuracy and being a part of the review process with you. And that's a big part of why Linear B sponsors this show in the first place.

0:38Because, you know, AI is writing the code. It's piling up. But the real question is what happens afterwards. Did it merge or hold up? Did it move the business and have impact on our users? These are the questions that Rob is really obsessed with and us too. And so measuring AI's impact is something that we talk about a lot in this episode. Here's my conversation with Rob. I'm really excited to dig in. I just want to start by introducing Rob Zubair, the CTO at CircleCI. Now, Rob is basically like one of the main protagonists of the Dev Interrupted Universe, if I were to say so. He's been on the show a number of times, but also worked with us on a regular cadence to talk about how engineering is getting impacted by AI, how it's really just getting garbled up, just chewed up and spit back out.

1:27And it looks totally different. And I personally always think that Rob is way ahead in terms of the thinking around this stuff. For example, when I first met Rob back in 2024 at an event we had in San Francisco, he posed a question to the audience. He said, why are we writing in a programming language designed for humans? And that really stuck with me. That was back in 2024. And now looking at where things are moving in our industry and how fast they're going, it feels like that future is just right around the corner. So Rob, it's really great to have you back to dive into the latest state of things and to see where your head's at on this kind of stuff.

2:05Yeah, super excited to be back. Thanks for having me. And it's funny that you bring that up. That was a big thing that was on my mind. back then and i've like totally changed my opinion but also still reflect on that a bit you know what i mean like so i measure most of my learning and development in days or weeks at this point so when you talk about a couple years ago i do remember that event it was super fun i didn't see it's gonna be hard to like try to map like two years right and i i've been at circle ci coming up on 12 years, I think. And so it, it all, it all feels very linear until the last couple of years were just like everything, you know, started changing so quickly.

2:49And again, as we've been saying, just like for me, that's super interesting and exciting. And I love learning this stuff, um, and exploring new opportunities, but at the same time, yeah, trying to remember what I was thinking about two years ago is super, well, it'll be fun. Let's dive in. I know when we pulled that out of the archives, it's like, wow, just how well this question has aged, but also to just like how many questions have now formed between now and then that are still unanswered, that we're still answering. You know, one of them being, there's this really funny phenomenon in engineering, especially in the last few years around declaring everything dead, you know, like MCP is dead or, you know, so-and-so is dead.

3:29This is like a thing we love to do. I get blamed about it all the time as, quote, a tech influencer in the media of dragging these protocols way out to pasture too early and then them ultimately finding their fit later on. So, you know, in your world right now, what feels dead to you? What's new? Are there things in your world that the internet's claiming is dead that you're like, that's so wrong, it's not? I'm just curious where your head's at. Yeah, it's a really fascinating exploration. Like, I think I've heard that CICD is dead. Our business numbers disagree, I would say. We're trying to help people deliver high-quality software quickly to their customers, and that goal is not dead.

4:14So the shape will change, but the idea that CICD is dead, I mean, we may come out of this with a very different look of how we do that work. But like I said, everybody's doing more of it is what I'm seeing. One thing that I like to, I wouldn't say proclaim dead, but I would say it has a target on its back is the pull request. Because that sort of as a form factor, right? Like the idea that I'm going to get my agent to use a CLI to create a webpage. Calling PR is basically a webpage that humans are meant to interact on, right? I mean, I know that you can interact with them so that someone else's agent can use an API to go get the state of that change and write some comments on it so that the original agent or yet another agent can come along and try to react to those comments.

5:12It's a very weird place as an interaction point. Like it doesn't make sense in the world that we're trying to transition into. And sort of, you know, if you go back to that question about human oriented programming languages, right, it makes sense for a minute in a transitional state where the only like we want human inspection to make sure the machines are behaving correctly. But humans were never great at reviewing code to begin with. Right. pick a PR and give it to the same person on two different days, give it to the same person at two different times of day based on whether they just ate or just had coffee.

5:55Are they in the post lunch kind of coma or are they super fired up with a ton of energy? Are they really interested or do they have five minutes before they have to run to the bus? And the quality of that feedback is going to vary so much. And so that model always felt a little bit broken. We just didn't feel particularly compelled to do something about it. But now, as we talk more and more about shifting bottlenecks in the SDLC and reimagining the whole SDLC, I really think that we will figure out how to do code review, code merging in a way that doesn't really involve a human. And when that happens, or some tiny fraction of it needs a human, whatever that might be, when that happens, I think that interface is going to feel really clunky to make it happen.

6:47I completely agree about the pull request definitely feeling like a relic of a time where it's served a purpose, but now almost the pull request is more like a stage than like an office where like the work is happening or people are interacting. A lot of times now the pull request is like, it's a statement of record where a bunch of systems come in and put their stamps on it, good or bad. Maybe a human's looking at it these days. Our numbers say they're not. You dove into those numbers too on our recent benchmarks report that we had. And so you're familiar with how very little human attention is able to get put on the amount of generated code and what goes through the gates that we have.

7:29And so the PR starts to feel a little performative at that kind of velocity, at that level of detail. Hell, it's also not useful because if I come in and maybe I don't like your PR or your PR is too big or there's ways that we need to change this. In the olden days, you'd go and change some lines in that PR. Now, like you're probably throwing that PR away. It's just like what you're doing is totally different with the code. Yeah. Yeah, I think that's exactly right. Like the. If you think about a diff even. Right. How like the presentation of a PR. So let's pretend for a second that human's going to read it.

8:07The presentation of alphabetical order by sort of tree structure of files and then the associated lines in numerical order that have changed within those files. if it's 10 lines of change and you could see it all on the screen you're like cool that makes sense right but when you're talking about what you can easily produce in a small increment of time with the with an agent today that which therefore feels like a useful unit right like taking a 2000 line pr and breaking it up to 10 lines at a time is nonsense right and then tack on your standard pr delay which you'll get out of your numbers right like why would we do that to ourselves.

8:51So if you look at that whole thing, then a question that I've been asking, and I don't know that I have, I'm starting to see ideas floating around. I don't have any great answers yet, is how we can present the diff when a human needs to review it in a way that is useful to the human, right? Like what is the structural intent oriented diff? And then within that, the machines have said, okay, if that's the intent, then this does what it should. Or if that's the intent this part looks a little bit sus and this is a kind of thing that looks like someone should review it from a security perspective so these 20 or 30 lines are worth looking at right but we open that up and we go oh my gosh you know github's like diff isn't rendered because it's too big that's basically what the page says right that's what says on all of our prs right now i'm like well that's how is that even useful to me But what I really want is, hey, Rob, machines covered 95 % of this.

9:48This little to have a look here and have a look here kind of thing. You know, I just thought of this. When someone hands me a legal document and I know they've done a good job and there's just like tape flags that say sign here, sign here, sign here kind of thing. Right. Like, great. I have people that are way better at reviewing legal documents than me. I just happen to sometimes be the person that needs to sign it. And then there's actually this very similar human context that's like, who was the person that reviewed it and how much do I trust them? And would I sign a thing based on what they tell me about the thing?

10:24Or I'm going to sit here and read every line of it. And so I think not just the surface when it can be, in a lot of cases, just automatically moved on. right so why are we doing all this interaction with agents talking to each other over some format but also then when i do need a human i think i'm going to surface that oh different use of surface but anyway i would i'm going to raise that up to them and flag it in a much more concise way right that's like here's the intent of the whole thing this part's a little bit suspect so let's just take a look at that part everything else is good to go yeah right it's like re-evaluating who the process is for obviously there's still a level of accountability has to happen in most cases a human still needs the answer for something someone needs to sign off on something you're right like maybe this moves towards just like the docusign experience of prs where it just like moves you through the lines that wants you to sign off on or like put your eyes on and you're trusting some other system underneath who knows but it's yet to we have to acknowledge at least that the new user experience and then also now the brand new agent experience that they need to be effective at it are very different and in that like diff driven world it made sense to show all of the files and like your folder hierarchy or in a semantic way that like we could glimpse and like know where other things lie but now that's noise for an agent who's like counting every single token going into it and trying to preciously contain it so it's like what you're optimizing for in presenting information is totally different.

11:59That's actually been really exciting for me just as a developer is finding this new class of like developer tooling, things in my CLI and otherwise part of my local coding process that aren't really for me. Like the outputs of them, however they're utilized, are really for an agent to audit something or to get a check on something that otherwise has a hard time getting a grip on. and just using those deterministically with an agent kind of lets them, it's like its own kind of guardrail, right? And now suddenly you're bringing things that used to maybe be, maybe this would be surfaced as one of those stamps that's going on the PR.

12:38Now it's happening even closer to the developer, even more locally, even more commonly before it hits the PR surface. So how do you think about the CICD getting pulled down almost into the developer machines because the development loop is now happening in my cloud platform or with my 30 sub-agents. Right, right, exactly. I think there's a lot shifting more left, even within the agent loop before it's asking for human intervention. So we're looking at a couple different things there. One, we launched something called Chunk Sidecars, which is basically like a direct attached CI quality validation environment that your agent is using as it works to say, okay, I made this change.

13:27Is that good? If not, I'll make some adjustments before I even tell the user, the developer, okay, this change is ready to go. So we think of that as like, again, I call it CI-like, meaning it looks and, well, I don't know affiliate of the looks and feels, the quality of the validation, right, is going to be much like what you're getting in your CI environments to break away from this, you know, works on my machine kind of problem, but happening in super short cycles, like we're thinking 30 to 60 seconds as a general target and giving that feedback directly to the agent as it's working so that by the time it goes up to like the more classic gate, right, of CI, meaning is this thing good to merge in and go to production, I have really high confidence.

14:15And the other thing that I think about a lot in there is then, you know, who's writing the tests, right? Like the agent is going to write the test too. Let's all be honest about that. No one's saying, oh yeah, let me just, let me just, I mean, I do kind of, in some types of development, I'll do sort of a TDD style where I say, what could possibly go wrong here, Claude or Codex or whatever, and get an answer and say, cool, can you write some tests that would expose that if it broke? Great. Now implement the thing. You know what I kind of go through that process. And that's often when I'm sort of working through an idea versus have like a really clear plan of what's, but I will still, in all of that, I'm using the agent to write the test, which sounds amazing, but the agent loves, I'll just generalize agents here.

15:05They love to like, well, yeah. Or to stub everything out, right. To mock everything. So you're not actually testing any of the business logic or any of the interaction with other systems or whatever. And it's like, cool. Everything passes. Cause its goal is to get to green, not to get to like very clear, you know, validation that all of your intent is correct and correctly implemented. Right. And so I think there's a role and we've been, we've been dabbling in this a little bit in, oh, I'm trying to find the right words, but ensuring that your validation techniques are of high quality themselves, right?

15:42That if something broke within your software, if you made a mistake, it would actually get caught. And there's actually, what's really interesting right now is we're still, I feel like all cool things in software happen in like the 60s and 70s. And we're still pulling out techniques that people thought of back then and saying, wow, now we have the data and the compute power and everything else to actually use this. Like somebody wrote it on a napkin in the seventies, uh, as particular, I'm thinking of something called mutation testing. And they said, this would be really cool that everyone was like, yeah, where would we find the computers to do that?

16:16Right. And now you could do that on a phone, you know, like the, the compute power that we have to, to drive this stuff is immense and particular to have an LLM analyze your code and say, oh, these are the trouble spots. This is where the risk is. Right. And I think risk analysis is a big thing when we talk about pulling out the kind of that DocuSign model, like pulling out, look at this and look at this and look at this. Right. So our ability to assess risk and then to assess the quality of our risk assessment, right. Is all getting back to like, what are we doing as engineers? Right. That's where we're putting our energy as opposed to like I sit here and I build a product, right?

16:55Because that's scalable, right? We've been doing continuous improvement as an industry for years, but now we're scaling that by saying, let's improve the machines that build the software, right? And if we actually work that the same way we've worked historically, like incidents or retros or whatever, then maybe the first time we just say, have at it machine, it's not going to be great. But if we dial the tooling on every time we have to step in, right? That's kind of my other thing about the DocuSign model. Every time I sign for something, I want to say, why was that exposed? Why was it possible to create this kind of risk?

17:28And how do we make the machines better so we do this correctly and I don't have to worry about it next time, right? And so, but if we don't start on that journey, we won't get to the end of it. I think what's really interesting is in all, like whoever you are, whatever company you work for, you have a competitor that's doing this right now, that's doing it probably better than you, right and so that's that's like where a lot we're talking about kind of the pressure to keep going and to do more and to build more right it's like somebody everybody's kind of checking over their shoulder because someone is has just been enabled to really like take on take on their business and there's so much like unexplored territory we all know it like the fog of wars the whole dang map and so everyone is very nervously exploring and trying not to get stuck in quarters trying not to go down the wrong way and waste a bunch of opportunities or time while someone else went and found gold immediately.

18:19Why couldn't that have been us? And so it's like, that's the world that I think a lot of engineering leaders are trying to navigate in. I love the shape of what you described, how those tools evolve, like this CICD-like or something. The word I've been using myself, it's almost like a new class of linters. It's not like smelling my code for formatting stuff as much as it is working with the agent on these really small, tight loops So on like an incremental, like it's editing code or reading code basis to kind of help keep it in guardrails. It's kind of like linting what it's able to do. And those kinds of things are allowing us to close a loop that up until now was totally open.

19:00We would do the best we could as engineers to create this like ornate pipeline that has like all of the stuff that goes through and checks all of this human effort that goes through this unbounded system. And since it wasn't really a fully closed loop, because AI didn't exist yet, it was kind of more like an instrument. Like you held it, you played it, it was like a single player kind of deal. But now it's exactly what's happened is what you've called out, is that the loop is closed. So now you can run it. Hey, Mr. Machine, do your thing. And even if it's not perfect the first time, now we can iterate.

19:32Because we spend all of that time building this instrument and playing this instrument. And while we're not maybe the ones playing the notes anymore, we know how it was all laid out and why we made the decisions we made to get there. And now we're in a world where it can make itself better, where it can find new purposes, new usage and new tools. And that's just going to require a lot of experimenting and a lot of measuring to understand what works for an org. Because all of that's exciting and cool to talk about on an individual practitioner level. you get lots of folks who've unlocked like 10x or 100x outputs as engineers and are super agentic and they're kind of leading the charge within their engineering orgs.

20:11But on an aggregate level, I'd say as an industry, most orgs, especially the larger they are, tend to be more on the less mature side of understanding AI's impact on their organization. You know, oh, we know we have cloud code. We know we have cursor. We know we love codex. And our engineers are using a lot of tokens. And you've even seen this narrative recently about people trying to track how many tokens they're burning. Are they maximizing it? Or now are they minimizing it? Everyone has a different opinion about this now. But I feel like people still miss the whole thesis of what did the code do after it was made?

20:50Did it even get merged? Did it cause an outage? Did it get reworked later on? Did it exist for a nice long time? Did it completely add completely brand new code instead of opting to maybe refactor something that would have made more sense? Like those are missed signals, I think, for most organizations. Curious how you think about leading your engineering work right now and how y 'all stay tuned on like the output signals that matter. It's super relevant that, I mean, everything you're saying, the cost is real. Right. Everyone, the, the, the time span from, Hey, we got to motivate everyone to use these tools.

21:33Cause there's real opportunity here to like, Oh, hold up. What is everyone doing? You know, was a week, I think in many organizations, you know, and, um, and, and there's definitely a spectrum, right. Of what you can achieve and what that's going to cost and the costs, You know, if you look at it in just raw cost, right? So ignore at your peril, but ignore for a second budgets, right? And actual how much cash you have on hand as a company. So if you ignore that for a second and you just did the math and said, you know, for what feels like an exorbitant spend, I can, you know, 3X the output even or the outcomes.

22:16Let's pretend outcomes for a second. I can 3X the outcomes of an engineer for a 50 % increase in costs. Like most people would do that all day, every day. Right. But it costs real money. Like someone has to, someone has to write the check. I mean, there's not actual check anymore. It's 2026, but you know, money is coming out of bank accounts and going to other bank accounts when this occurs. And, you know, Uber kind of famously, and there's been other examples since came out in March and said, we spent our whole token budget for the year. Right. And like people care, the investors care, business leaders care about total cash spend.

22:57That's a real thing. It matters, right? And so then the question becomes, what is the right amount and how do we navigate that as we grow? And if I tripled the outcomes of an individual engineer, let's try to make the math as simple as possible. if I triple the outcomes and I only spent 15 % more, like 50 % of their salary on tokens or whatever, that would be great if the minute they merged, I started realizing the revenue increase, right? Exactly. But that's not what happens in software. Like sadly, it's not what happens in software. And so there's still a time delay. And in that time delay, I've spent the money and I haven't earned back the money, right?

23:37And that would also be if we were perfect at building things that always returned in terms of revenue. It's more like a portfolio. We get more for some things, less for other things. So all that to say, there's a lot of scary math that is difficult to navigate right now. And predictability is often as valuable, if not more valuable in finance than the actual magnitude. If I can forecast it, I can manage around it. If it just comes out of the blue because one day some developer realized how to write a for loop, right? And spun up a hundred parallel agents that were just working nonstop, 24 seven.

24:16None of it was producing anything like that's a great, but the money's gone before you even notice it's happening. Right. And so there's a lot of, you know, a lot of people are building controls. A lot of people are trying to figure out how to put caps in place. They're doing things like routing, you know, okay. And how can I route this lower, you know, trying to educate people on how you don't need an opus for everything or, you know, those sorts of like, it's happening all over the industry. And I would say for us at CircleCI, just as our engineering team, we're exploring a lot of things, but we're trying not to throw too much at our people.

24:57Like one thing we've realized is some Some people just want to get some stuff done, right? Not for everyone, every day cannot be, try this new tool, try this new tool. Now we've changed the way that we're going to do this. We've changed the way that we're going to do this. It's like, great, but it's how I'm going to get updated on today's stack. It's time to go home and then tomorrow I'll learn tomorrow's stack, right? Like we need people to build stuff. So, you know, we, we kind of keep an eye on the outliers, right? Hey, we see that you've sort of picked up some pace here. That's awesome. Can you tell me a little bit about what you're doing?

25:26because it seems like there's much higher spend than outcome or output even. Let's worry about outputs. Probably actually easier. I'll revert my previous statement because I don't have to worry about whether we're building the right things. Let's just talk about building stuff, right? And sort of, hey, just show me a little bit of what you're doing. Could I tune a little bit about how you think about using the tools? Can we just click on that thing there that says Opus and set that to Sonnet and see if we get the same results? Just really basic tweaks because we want to encourage people to build and learn and drive.

25:58And as we, well, I'll get to that one second. So without a lot of like, either I need everyone to be super educated and detailed and understand, like, what's the point of a layer of abstraction if I need you to understand everything inside of it? And without forcing again, like new tools down everyone's throat every day, right? Now I need you to switch to this. Now I need you to switch to this. So we're trying to be like a little reasoned about it and pick the points that are going to be high leverage and sort of take advantage of them. And then the other thing that we're spending time on is as we discover tools and techniques or whatever, can we make those available to everyone in ways that they don't even really have to think about, right?

26:38So, I mean, generally, something as simple as a skill will get you to an outcome faster and burn less tokens or fewer tokens because the agent isn't figuring it out for itself over and over, right? So can we take those and distribute them well across the organization, make this part of how people work. Like, I don't need you to understand the skill as long as the skills are readily available in your agent. And the first thing, you know, the likely thing to happen is that when you ask an agent to do something, it's going to follow that skill. Right. And so there's a lot of that kind of work, but I guess to, you know, to summarize, it would be like, I don't want to put the onus on every individual because one humans are fallible, right?

27:18We know that we're going to forget to do stuff. We're going to mistype stuff. That's fine. That should be fine, right? And I also don't want to create a huge amount of churn. So I look for like, where are the kind of high leverage points as well as the outliers? What can I learn from the outliers? And then how can I take that and bake that into kind of the things that they just end up with for free? This episode is brought to you by Linear B, the engineering productivity platform. There's a widening gap between engineering teams that have turned AI adoption into delivered work and the teams that haven't.

27:48It's real, it's measurable, and it's growing every quarter. On July 30th, Linear B CTO Yashai Beery with Ben Lloyd Pearson and Andrew Ziegler walk through new engineering benchmarks built on 2.7 million pull requests from 250 plus engineering organizations. You'll see why high AI usage correlates with a 2x increase to PR merge rate and why more AI code doesn't automatically mean more shipped code. We'll also cover the lowest effort win that's available right now. turning on AI code reviews. Register to join live and get first access to the full report. Save your seat at LinearB.io. Yeah, it's really important that you create some sort of ecosystem or like substrate for your engineers and just your whole organization, really, because everyone's benefiting from this event, is to share what's working.