AI Engineering: Measure the PRs not the Code

0:00I'm thinking about cost per PR. It's really easy to look at that$600 bill that he had for that day of work and say like, wow, that's so much. But then when you divide it over how many PRs he worked on that day, it's like, oh, actually, it was only about$30 per PR. And then it changes how we think about the ROI, because the spend is not the problem. The spend with no return on that spend is the problem. Hi everyone, welcome back to Beyond the Pilot. Today's episode is a little bit different. We're pulling this one from a live panel that I moderated at VentureBeat Transform. It was all about agentic coding in the enterprise.

0:39And my guests were Emily Sherio from Kilo Code, which a fun fact got acquired the morning of the panel. They got acquired by Anaconda. We also had Jared Go from Symbiotic, who's doing a lot of work with physical AI. And Amal Jain talking about what they're actually doing at Replit. We get into what people are actually doing in agentic coding in production right now. Code review, not just code writing, has become the real bottleneck. and how teams are measuring success of their actual token spend. On top of that, we also look at where things are going in the next six months. And this sort of really compounds with VentureBeat's own research that we're seeing just how fast things are changing in agentic coding and how teams are actually building software.

1:28Make sure to listen to Emily's own unique strategy of how Tilo is measuring success for pull requests. That one I found really fascinating. Enjoy the pod. We'll talk to you soon.

1:43This is VentureBeats Beyond the Pilot podcast. It's about enterprise AI in action. I'm Matt Marshall. Today's episode is presented by Outshift by Cisco, Cisco's emerging tech incubation engine and driver of Agented AI, quantum, next-gen, infra, and beyond.

2:03All right, everyone. it's great to have you so we've got a really interesting panel now I think the original title from this was was sort of like vibe coding and I kind of like yeah that's like kind of last year or at least last month so I guess okay kick it off let me let me ask a little bit about you know I yesterday we had the CTO of Instacart here and the first question I asked him was does his teams still read code right and that turned into a very interesting conversation of where it turned out that he was saying basically 97 percent of them don't right and they actually sort of measured this so let's start with this emily uh their agents do i would say most people are uh reading each other's prompts and they have reviewing agents who are writing reviewing the code but But unless something's really broken or debugging, 99 % of the time engineers are not reading or writing code anymore.

3:05Jared? Yeah, I think for me, it's really about directing the focus of AI. That's one of the things I try to teach the folks at Symbotic, where it's like, if you ask the AI, you know, these are my criteria. Let's look at it from the lens of, you know, security, elegance, you know, being clean, concise code, water tightness. If you direct the AI in that way, I think that does most of the heavy lifting for you. then you can really not directly review lines of code. Okay. Yeah, we've at Replit gone very agentic, but this is one area we've kept it a little bit more conservative, where essentially we run an agent that would look at a PR and assess a risk score to it.

3:40And if the risk score is low, author just self-merges. But for others, we do have human reviewers sort of coming in and reading the code and giving feedback. So reading PRs is different than reading code, yeah? That's fair, yeah. And reading code, pretty much no at this point. At this point, it's like the PR is ready to go and the agent does all the work and you take on when it's already artifact. Jared, so you're doing physical AI. So is that different? Again, for you, are you reading the PRs at least? Different teams do. I think one thing that's really important about robots is safety is really important.

4:19So I think that's a big focus. There are teams where, you know, you need to do intensive simulation. You need to have reviewers for code. But there are a lot of things that are peripheral, like your logging system or something else that might not need that scrutiny. So I think it's really like you got to make sure you have your quality gates and you make a conscious decision about what's necessary in each case. But it's not one necessarily uniform rule or process across every single thing. So, OK, I'm going to ask you guys your stack, but your stack is going to be pretty obvious. Yes. All right.

4:46So, everyone, Emily's from KiloCode. They just got acquired this morning, which deserves - Thank you. Thank you. All right. So, do you want to... Let's just start with that. I'll come back to you guys about the stack in a second. Yeah. Tell us a little bit about KiloCode and then also about you got acquired by Anaconda, right? A very famous sort of Python ecosystem company. Yes. So KiloCode is an agentic engineering platform. We want to meet developers everywhere they are in VS Code, on the CLI, in JetBrains, on the cloud, on their phones, wherever they might be. We pair that with 500 plus models through the Kilo gateway or you can bring your own.

5:31We believe in a multi-model future where your software that you're using to do agentic engineering should be decoupled from the model that you're using to do it. And with Anaconda, we can now bring the power of Kilo code to the enterprise and secure environments and secure Python packages that Anaconda is known and trusted for. So really excited. We use Kilo to build Kilo every day. So all of our developers are using Kilo as their coding agent. We use Kilo code reviewer on our PRs, often with a different model. And then we use Kilo's security agent to help keep our platform secure, manage our CVEs and vulnerabilities as they're reported, as well as one of my favorite ways to use Kilo every day is what I call killing my to-do list.

6:24so I don't add something to my to-do list. I kick off a cloud agent to do it. And now with the Kilo iOS app, I can do that right from my phone. So really trying to meet people everywhere they are. So just quickly, the reasons for Anaconda acquiring you seem to me that... So one of the things that I found fascinating was that you're doing 10 trillion tokens a month, right? Really Kilo Gateway. Some serious traction there. I and not only that's a large percentage of that I'm guessing is open models right how many models do you support again over 500 okay so that sort of gives a sense and and Kilo itself basically can route between those and correct we have multiple auto models in Kilo that we can route for you auto efficient is one example where we we handle the routing for you based on the kind of tasks.

7:20So you select auto-efficient, we categorize the task and we'll route it to the correct model, giving you the best bang for your buck. And often what we see is that companies or individuals will use like a really expensive frontier model, think a opus, fable, sole category model to do their plan, kind of architect out whatever they're working on. And then they'll switch to auto-efficient or another open weight model to get more bang for their buck. And that's how they're really going to be able to leverage AI as much as they want, but stretch their dollar along the way. So I would just say quickly, this has been one of the themes in preparing for this conference of talking to a lot of the people that we've been talking to.

8:03They just don't want to talk about it publicly. This whole idea of having a router, which does all of this. And many of them will admit that, yes, they have something like that. But when it comes down to details, they are very reluctant to sort of, you know, actually say anything about which open models they're using, that kind of thing. Can you tell us a little bit about what are the most popular open models? Because you've got this amazing data. Yes, and we make it all public. So Kilo.ai slash leaderboard, you can see what models people are using in Kilo today. If you're using Kilo Auto Efficient, as you're using it, it's going to show you what model is being used.

8:41The other thing too is we also respect the model provider limitations that you provide. So if a customer is in the EU, they only want providers in Europe, they can set that configuration in the Kilo backend in the application. And then when their models are getting routed, they're only going to be routed to providers in Europe. So I can't say, oh, when you use auto-efficient for that task, you're always going to get this particular model because that's not how it works. It's factoring in what's important to you, what limitations you've what data retention policies you've established for your organization, what keys you brought in, what commits you might have, and it's factoring all of that into that routing decision.

9:27Again, so we can give you the best bang for your buck. Congratulations. It's a very big deal. Okay, guys, back to your stack. So I'm hoping that you're not just going to tell me key load code because otherwise it's going to look like an advertisement. Jared, what do you use? So at Symbotic, we use a combination of cursor, and we also have our homegrown magenta harness we call SimCloud. You know, we are sort of like there's pros and cons of each. I think one nice thing about moving a lot of our workflows to our own cloud native systems, we can just build everything in there and hooks to everything we want.

9:57We can control the tool surfaces and connectors and everything like that. And for a company like us doing physical AI, I mean, there's a lot of data at warehouses. There's data, you know, bot logs. There's everything. So trying to pull that all into one place really nicely is like, you know, doing in the right way, making it efficient, making it, you know, reusable knowledge from across the organization is important to us. Do you, so Cursor is an interesting one because a lot of people have left Cursor, right? You know, at least, you know, at some point. Is it sort of like dictated that everyone in the organization has to use the same thing or is it sort of people pick their own thing?

10:33The way we do it is we do have guidelines because I think there's too much fragmentation if we let everyone pick their own tool. So we kind of centralize around tools, but we keep our ears open to like if the tools not meeting the needs or something like that And so in our case we mostly fold the functionality that's missing into our tool But you know, we're trying to train about 900 engineers You know, I wrote a book called Beyond the Vibe and where I tried to try, you know Teach people the steps you need to do to do these kind of things and part of this was like, you know There's a lot of understanding about models that people don't have and how you train them to like, you know Learn the right skills and like theory of mind understand what is the model thinking and perceiving so I can work with it better So there's a whole bunch of stuff that I think we're passionate about, like transforming our organization here.

11:13Amal, Replit, what are you guys doing at Replit? Yeah, we've, you know, internal tool that we've built that lets, it's essentially sort of self-driving for software engineering at this point. where engineers are sort of giving it the task and how to verify whether the outcome's successful. And it sort of executes that end-to-end of planning, implementing, and testing, getting code reviews, merging, navigating CI, et cetera. And sorry, go for it. No, no, so I was going to say, that sounds definitely more like a full-function suite tool rather than just writing the code. It's more sort of the processes around it as well.

11:55Absolutely. Yeah. It's almost, you know, the idea was human on the loop, not human in the loop. We wanted to sort of solve that bottleneck. And the fact that, you know, it's a fleet of agents that run in their own cloud VMs and with access controls behind token proxies and such, they're secure. So we connect them to all the systems, you know, everything from GCPs to data dogs to Slack to everything else. and that means it has the right context. It had tools like browsers to self-test that it can really go that end-to-end mile. Do you guys make your own skills or MCPs or something to go with that?

12:33Absolutely, yeah. Exposing certain things via MCPs. We encourage folks to create skills and then share them. We have a repository of them. Yeah, it's been really fascinating to see how people have been using it. I think my favorite sort of anecdote recently has been a very gnarly bug deep in our systems that one engineer really couldn't repro or solve. So it goes to what we call a manager agent and told her to, hey, figure it out. The manager agent told her to go to sleep. It spun up a bunch of agents that repro, one of them found the repro, then spun up a bunch more agents to try a few fixes, found the fix.

13:12Six hours later, it had the PR ready for it, for this gnarly bug that a lot of people were very puzzled by. And that's been pretty amazing to see. So it sounds like you're very pro, if we're looking at sort of build versus buy for SWE tools and stuff like that, you're very pro to sort of build your own of what you guys actually need. Does that fit in with you? Yeah. Obviously, you are sort of building one of the tools, I'm guessing. You know, I still think a lot of the calculus comes down to where do I want my team to spend their energy. So I'm not going to build a CRM. I might build something that's kind of tech adjacent.

13:50A lot of our support workflows we've built in-house, our team, every member of the team has access to Kilo and is using Kilo, whether they've got the word engineer in their job title or not. So we're definitely building more than ever, but I wouldn't say that we're not also buying. We're buying where it's not core to what we're building for the business. Right. Sure. Yeah. You know, this is the most important decision. I think organizations are looking at making now. And I think just because you can build something doesn't mean you want to sit with the maintenance costs, being on call for that critical product or something like that.

14:25So you have to think about it really carefully. And I think it's on leadership to really send the right message that it's not like, hey, these all coders can just go wild and make everything. Because in reality, teams don't want to necessarily do that. So I think, yeah, just respecting that and thinking a lot about it. So would you say the cost has moved from the investment in building to really just the investment in maintenance. If you, sure, you can build something pretty quick nowadays, but maintaining it long-term is going to be the hassle. Yeah, like, you know, greenfield is so easy for agents.

14:54Brownfield, we all know, is where the actual challenge lies. So it's like, you know, you don't want to accumulate, oh, I built it in a week now. Oh, I'm going to spend the next month helping work through all these bugs or whatever, right? And also product decisions are key, right? Agent, I don't think, they don't make strong product decisions in a lot of cases. So those are things that you just take time and effort. So that takes away from your other focuses for your team. And it's the last 10 % that's hard, right? It's like getting the pretty UI or the pieces, you know, going for that is the easy portion.

15:22It's the like specific edge case. Those are the things that really drain the level of effort spend. At Roblox, we have a little bit of a different stance, obviously. You know, selling, letting people build internal tools or SaaS replacements is our better and better. So we actually encourage people to build things themselves. And it's almost like folks waking up and figuring out what generic expensive SaaS we could kill today. And our job then as the product builders of Replit becomes, how do we make that last 10 %? How do we make that maintenance easier where any non-technical person could do that?

16:01You don't need engineering for it. So, you know, we have agents that once you build an app on Replit and publish it that are monitoring it to make sure if it fails, they'll auto fix it or make sure things are secure or improve your app for you, make it really easy to do that. And I think for that, that shifted the calculus for us quite a bit internally. Not quite CRM yet, but, you know, we just churned a seven figure go to market software recently. We just built it in-house. this series is brought to you by outshift cisco's incubation engine by creating an open interoperable infrastructure outshift is enabling agents and humans to share intent context and reasoning the cognitive evolution for agents is here explore the internet of cognition at outshift.com and so what what are the lessons then for the people should take away for that if they do want to go that route and they are prepared to put in the effort and stuff like that, what are the tips that you would give them to be able to get that last 10%, to be able to make the tool that can maintain it?

17:09Yeah, I think it's important to be judicious. Like we've been discussing here, it's important to know how far to take this. You have to go in knowing the maintenance and such costs. And I think once that this is a solution, I mean, you should buy Ruplet to do it, make the job easy. It's quite hard, but you really have to think about security and maintenance and such. Interesting. All right. The next question I wanted to talk about was token maxing. So, you know, we've seen, I guess, and guys, tell me if I'm wrong, but like sort of last December, we saw an inflection in the models, right? where we saw some of the anthropic models suddenly get a lot better.

17:54The game then became, for the first three months of this year, who could token max the most to try and do things the most. Probably the last two months at least, everyone's been going, hang on a minute, this stuff can get expensive. So I'm curious to get your sort of take on this. Emily, let's start with you. Yeah. You actually had a really good anecdote. So please tell us about that one as well. So there's two sides to this. what am I hearing from my customers and what am I seeing as an engineering leader? And so much from the customers we're hearing all the time, hey, I accidentally spent my whole AI budget for the year and it's July 15th.

18:34So what do I do now? And we're really pointing them to the workflow that I mentioned earlier. You want the expensive models for planning and then you want to use open weight models, more affordable models for everything else. And the reality is, especially when it comes to agentic engineering, to coding itself, all the models know how to code. And the quality differences come down to the prompting. So realizing where you can really up-level your team to help them get the most out of the models they're using is going to make a big difference. Sharing skills, strong guidance, MCP servers with the right information, these are all things that are really going to empower models other than the state-of-the-art models.

19:17So that's one side of it. So internally, and this is what I was mentioning earlier, I've got one engineer who, if he were driving, I would say he has a heavy foot. He is, you know, consistently at the top of our usage board. And I regularly have to nudge like, hey, what are you doing there? And so one metric I've been paying attention to and being honest here, I still TBD on whether this is the best metric. But I'm thinking about cost per PR. um so you know he spent it's really easy to look at that 600 bill that he had for that day of work and say like wow that's so much but then when you divide it over how many prs he worked on that day it's like oh actually it was only about 30 per pr and then it changes how we think about the roi because the spend is not the problem the spend with no return on that spend is the problem and And so if we can be really intentional about like what we need isn't a metric of how much we're spending.

20:21What we need is a metric of how we capture the ROI for that spend. And I think as engineers, you know, you might ship something that makes it out today, but you might ship something that doesn't make it out till next week. The closest proxy we've probably got is a pull request merged. And so cost per pull request is the metric that I'm paying attention to right now. It feels like the closest proximity for how I can measure value. But I think, you know, three months from now, we might not be talking about token maxing at all. So we'll see. Jared, what about yourself with the physical AI stuff? Yeah, so when we rolled out these tools, I think the thing you need to think about most in your organization is the controls and giving like the distributed control.

21:05Because it's like, you know, the worst thing that happens is everyone's like, oh, I need more tokens. Go talk to this team. And then this team is like, well, let me talk to your manager. You don't want this triangular stuff. So you need a tool. Like what we built early on is a tool surface where managers can kind of see what were the recent PRs, what was the amount of usage, how does it trend over the course of the month. And then they're empowered to go right there and click, okay, I'll give you another, you know, to the next tier. And one important thing we did, too, is we have tiers. So we don't try to make it too fine because otherwise you're like, oh, should this person get, you know,$350,$375?

21:33We just have, you know, set tiers. And you can move people between them. And that's a financial tier? or yeah it's a financial like you know let's say you know 150 bucks at the start you know 450 you know above and then you can make exceptions for this per day per month and i think you know connecting that loop is really important because that ties managers into the value chain so exactly what emily was saying about like you know they're the people who can assess okay you're doing good work yes you know i recognize i put in a justification but you need to give your finance or predictability and so they're having a cap and saying you know we know how many people went up in cap this month makes a big difference when you're trying to like corral these costs and you know make things efficient.

22:11I'm curious do you find when you when you peg it to a financial amount like that that then it encourages more use of open models or cheaper models so that okay then the person suddenly now has to think about how they budget it themselves rather than just oh you've got you know 16 million tokens per day or whatever. I'll give you a very quick story that's kind of funny so yeah I think we I don't know if there are other cursor users but there was a change that were cursor made where we were grandfathered in under like the four cent per request plan even for frontier models and then they're like no now everyone's gonna pay full right and so that was a shock to everyone and having the limits in place everyone stopped to think about okay am I using this efficiently am I picking the right models so I actually thought that was a really great outcome where everyone was much more aware of how their usage ties to cost and could make good decisions and we had all sorts of good discussions people were saying hey you should try this model you should try this this works better for this c-sharp code this whatever so i think that yeah they people need to be aware i think aman what are you guys doing at riplet yeah um that when i was describing our stack i was talking about that internal tool um it started out with engineering and soon spread to the rest of the company um one of the folks looked at the bill eventually and they realized someone on the support side had blown through an insane amount of money.

23:29When we looked under the hood, it was because they were running an automation using GPT 5.5 Pro Max. And I think that was sort of like the first time we, a couple of months ago, where we sort of started to look, you know, take a look at like the return and the investment. And because so far, at least till that point, the ROI was rather clear. We could see engineering productivity 3x. So no one had questioned it yet. And so we finally started questioning it, finally started sort of you know very basic things like a visibility to everyone not in a way that it's anti-productive it's still just be aware of what kind of spend you're doing and then very basic things around model routing of course like that's that's a that's a very simple thing to do and even sensible defaults most tasks do not need the frontier so we're investing a lot more there because it also is something we do for our core product as well.

24:25So I think let's open it up. Do people have questions that they would like to ask the panel? We've got one here. We've got someone with a mic. Good afternoon. Thanks for this discussion. This is a question for Amal. So as you think about providing the service of Replit to VibeCoders, have you thought about actually providing this routing layer to the basically to your ISVs so that they can take advantage when they build products to distribute to their customers yeah great question so so far our stance has been very much we as the harness builders have a great sense of that cost versus capability sort of sort of spectrum where we are essentially making the the decisions on users' behalf of what model to use when and what capacity to minimize cost and maximize capability.

25:22And I think that's worked really, really well so far, especially for the types of users we serve that are, you know, most non-technical users wouldn't be able to, a fable to them is still a poem, not a model. So that's worked out pretty well so far. I think, But that could continue to change. Those are discussions we're trying to have. There's clear trade-offs of if you let users select the models, then it's a lot more responsibility on them to sort of navigate. I'll have just one short thing, which is like, you know, so for example, in our harness, we have like costs are upfront. And that was actually a big debate.

26:01Some people are like, well, if you put the cost, you might discourage people from using. And I'm like, well, they should think about it. Like if I did this task and I said like 85 cents, well that's pretty good depending on what it said you know so i think just for us like transparency and making sure costs are visible is i think a key a key thing do we have another question here in the middle

26:23um i think that was a really good metric on like measuring the usage on like pr creation but do you think like you're also thinking about pr merges because what i see is like the burden on the reviewers have gone like crazy up right where it's like you a developer would create five prs in a month now an ai agent can create 500 in a day right so it's like maybe the measurement might be like pr merges and then um um yeah i mean uh code deletion is also very underappreciated in the industry. And I feel like the more that people can delete some of the code that maybe can be optimized or maybe is unnecessary, I think that's a really good...

27:13Is that something that you're looking in your company as well? Yeah, thank you for the question. When it comes to PRs, if we think about cost per PR as spend over PRs, it has to be over PRs merged. Work done that isn't merged might as well have not happened. So we have to take all the work done sometimes in the process of figuring out the final solution. Like it's not a bad thing that all work doesn't get merged. But you have to factor in PRs merged as the denominator for sure. And then the other side of your question, agents do not like to delete lines of code for sure. I think that's where people judgment still comes in.

27:59Code review is currently the bottleneck in the process, whereas development was before. I think one of the big reasons code review is the bottleneck is because we've changed agentic engineering. We're all talking about developing software in a new way. But when you get a room this big talking about what code reviews looks like in this era, you're going to get 50 different answers. And so we haven't converged on what code review means quite yet. We've got our opinions at Kilo. I see what my best engineers are doing. And I have like a blog post in drafts about this somewhere. But I think every team needs to be rolling out a code reviewer agent or bot.

28:40We've got Kilo code reviewer ourselves that we're using. I think you mentioned them all that small PRs are getting like a bot review and merge, different teams are taking different approaches here. You don't want something that's touching your billing infrastructure or off auto merge, but we don't need review to look like the way review did in, say, 2022. And so it's figuring out what review looks like next. I think that's going to be the big theme of the next six to 12 months, for sure. I would point out, Adam, there's a conspiracy theory going around that I talked to the panel about this. So I do a lot of work in research.

29:21And one of the conspiracies is that a lot of the research labs are actually getting agents to file pull requests on open source things and then see what gets merged to know whether that was a good training signal for training your models to get better at that. So there are certain labs that have been accused of it more than others. All right, we don't have a lot of time left. Let's just quickly to wrap up. I'll introduce you. If we had spoken six months ago or perhaps eight months ago, it would have looked totally different, this panel. Right. So pull out your crystal ball. Where are we going to be in six months from now?

30:00Amal, you kick it off. Wow. Predictions are always hard. Just in the general direction, what do you think is going to change? Oh, my bad. What are you starting to see changes in? I think the model selection part, I think we're going to see a big sort of like emergence of open source and open weight models that are used for majority of tasks and frontier models sort of reserved for these key decisions, key architecture decisions and such. I think I think that's a shift we're gonna we're gonna see pretty aggressively Jared I think for me it's really like very long unattended work loops and I think you know I don't know if I believe the stats but you know the codex team was saying a lot of their tasks are like 8 to 12 hours I don't feel like you know you get the quality out of that little interaction today even with frontier models but I think it's certainly going up so we need to build the harnesses that have enough information and tooling and structure to support that kind of work which were most companies are not built for you know so you don't want to have to like wait for someone oh they have to upload this document to you for the agent to continue its work.

31:04So really seeing a 24 hour, 48 hour productive work, you know. Interesting. Emily, what are you going to be doing in Anaconda six months from now? Yeah, I think we're going to see a lot more model sovereignty as a push, whether that's geographic sovereignty of models or companies wanting their models on their infrastructure running in closed environments or isolated environments. I think we're going to see companies double down. We've been talking about moving a lot of workflows to AI, but the level of model we have now, if they haven't already, they're in the process of moving it. But part of that is going to be isolating from the external environment.

31:41So sovereignty of models is going to be really big. Thank you. Everyone, please thank Amal from Reflet, Jared from Symbiotic, and Emily from KiloCo. Thank you. This series is brought to you by OutShift, Cisco's incubation engine. By creating an open, interoperable infrastructure, OutShift is enabling agents and humans to share intent, context, and reasoning. The cognitive evolution for agents is here. Explore the internet of cognition at outshift.com. For more stories about the AI revolution, like and subscribe to the podcast and check out venturebeat.com to sign up for our newsletters.