Stop measuring AI adoption. Start measuring AI impact. | LinearB’s APEX framework

19:11Dan Lines:And for this, we define it as measuring how effectively AI is embedded into your production systems. And the North Star metric that we have for this is AI-assisted PRs. So what number of PRs over a specified time period, or what percent of PRs were assisted in some way by AI. And we kind of break this down by like coding assistance, code review, fully agentic systems. Like there's a variety of ways that this shows up. And you know, the, one of the things that the, that we note with this framework is that like usage dashboards alone aren't going to validate your impact. Like I imagine if you were to just go and look at raw usage today, you would see most of your developers are using AI at least on a weekly basis, probably daily at this point.

19:56Dan Lines:But we take it a step further than that and we tie usage to the actual pull requests that are entering into your code base and looking at how AI is contributing to the unit of work through the system. So let's start there with AI-assisted PRs, Dan. So what role do you think that this fits within Apex?

20:17Ben Lloyd Pearson:Yeah, yeah. Okay. So a few things there. I do think impact matters the most. Okay. So, and that is what I'm hearing from the customer base. Like, I still have to say, how does this affect things like cycle time, rework rate, PR size, change failure rate? Yeah. How does it affect my planning, accuracy, my delivery, all of that? It matters. And that's why it's not only about adoption. Yeah. And usually I'm hearing that from customers that feel, hey, I've already kind of rolled out AI. I'm ready for the impact side of it. But I will say there are also a lot of engineering organizations that are on the adoption phase.

21:04Ben Lloyd Pearson:And when you look at AI-assisted PRs, I think it's a really easy way. And I'll go back to the teams. You know, which teams within your engineering organization have like 70 % AI assisted PRs and greater, which teams have 50 % and greater and which teams are kind of just like starting out, let's say 10, 20%, that type of benchmark. And I do think that gives kind of like, okay, I can now get an understanding of like what maturity level my adoption is amongst my teams. And like I said earlier, usually what people are doing is trying to look at those teams that have a high adoption rate and high impact metrics.

21:48Ben Lloyd Pearson:Like cycle time has decreased, change failure rate has decreased. Now when I identify those teams, I can go in and inspect. What are these teams doing with AI that maybe other teams aren't? And then you go and try to replicate those behaviors. That's the pattern that I'm seeing. Yeah.

22:05Dan Lines:And it's like, it's really, again, it's like, it's like an anomaly detection, right? So like a team with really high AI usage, it could be one of two things. It could be a team that has learned something really good that is super beneficial that you should propagate to other teams where you can. the other side of the equation might be a team that's currently overwhelmed by ai slop right like maybe maybe they're just creating tons of ai generated code they're not reviewing it they're just pushing it to production and things are blowing up constantly and yeah so and that's where i think going from the a the ai leverage to these other metrics is really critical right because if you're just getting more ai assistance but it's it's messing up all the other stuff like predictability, the next thing that I want to talk about, then it's all for nothing.

22:51Dan Lines:So let's just move on to predictability then, which to us, that's ensuring that your commitments are reliable and that, again, the AI volume isn't adding instability into your systems. So we have planning and capacity accuracy as the North Star metrics to this. And of course, as AI increases this output variability that I'm describing, how do managers control planning and capacity accuracy, Dan, just to make sure the volume isn't upending their sprints.

23:23Ben Lloyd Pearson:Yeah. And this is the balance aspect of it. I'm happy you brought it up that way. I tried to talk to you and Ori yesterday about Mr.

23:30Dan Lines:Miyagi from Karate Kid, the balance.

23:34Ben Lloyd Pearson:And neither of you are watching Cobra Kai, which you gotta watch. But yeah, I always think

23:40Dan Lines:required watching for all engineering leaders.

23:42Ben Lloyd Pearson:Required watching, like Apex, Balance, Mr. Miyagi, all of this goes together. Yes, the balance side of it is if you're just generating a bunch of slop or junk, whatever, with AI, you would see, okay, my planning accuracy in my sprints are decreasing. I'm not actually doing what I say I'm going to do. It's not stories aren't getting completed. Bugs aren't getting fixed. I'm not actually providing value back to the business. I'm just messing around with AI and doing a bunch of volume stuff like we said in the beginning of the pod. And so I feel like that's one of the balancers. Hey, let's look for teams that, okay, you do have, let's say, nice AI adoption.

24:27Ben Lloyd Pearson:Yeah, maybe like 65, 70 % of your PRs are AI-assisted. But your planning accuracy is increasing. Your capacity accuracy is increasing. You're doing what you say you will. That's kind of like when I say like predictability, I think we wrote here, ensure delivery commitments remain reliable. You're still a reliable delivery team, even though you're utilizing AI. Like that's the, I think, balancing aspect of it. Yeah.

24:57Dan Lines:And this is where, you know, we haven't talked about quality indicators very much, but I do think this is where the quality starts to really come into the picture. because if you're creating software with high defect rates or if you're slow at responding to outages or failures or if you are constantly reworking existing code and refactoring it on just a constant basis because it wasn't architected the right way. All of these things are going to impact your predictability and the lens of AI, it's going to amplify it. It's going to make it even worse. You know, it's why this stuff is so important.

25:37Dan Lines:And speaking of important things, let's talk about the sort of tried and true, though probably not the most exciting letter within this, and that's efficiency. You know, no one really is excited about efficiency, but everyone wants to be efficient, right?

25:50Ben Lloyd Pearson:I'm excited about it.

25:52Dan Lines:Yeah, yeah, yeah. I mean, it certainly feels really good to be efficient. And, you know, and this is all about optimizing how your work flows from start to merge. Our North Star within this is cycle time, but we also, you know, getting to the quality side of things, we also throw in CFR, change failure rate, as a core metric to this. Because if you move faster, but you're creating more failures, you're losing the gains of being more efficient. So, you know, Dan, my question to you is, you know, if you're adopting AI, your coding time drops dramatically because AI is doing all the writing for you, but the review time spikes, your constraints just move.

26:31Dan Lines:right like isn't isn't that the case like how do we how do we use cycle time as the north star

26:36Ben Lloyd Pearson:within this yeah so i i said hey i i like efficiency uh let's pay homage to dora cycle time and cfr yeah these are dorometrics this matters the efficiency of how code flows through your sdlc and the end game actually making it out to production into the hands of a customer that matters a lot.

26:57Dan Lines:Yeah.

26:57Ben Lloyd Pearson:So, you know, to your point, Hey, let's say coding time decreases, uh, by a ton. And let's say even, uh, PRS open increased by a ton. If these PRS, uh, no one's picking them up for review. They're not actually making it out to production. They're getting thrown back. Maybe you do have standards in place that were kind of blocking these PRS. Uh, maybe the test coverage is there. Maybe they do make it out to production and your change failure rate is spiking. The customer that I was talking about earlier, quality was actually this person's number one thing on their mind in the AI era. The mindset was like, yeah, okay, we're deploying.

27:38Ben Lloyd Pearson:I think they're using Claude. Hey, everyone's using Claude. And I can see a lot more stuff is happening. But you know what they also told me? My number one problem is incidents in production. and this person told me they're caused by code. Like these are direct bugs in prod. So on the quality side, again, the change failure rate, paying homage to Dora. The rework rate, are we having to rework the code over and over again? That's where the efficiency is that balancer to the speed of AI. So like I told you, I'm in on efficiency. I still like the E.

28:14Dan Lines:Yeah, and I think it's really important too to break cycle time down because we've been touching on this multiple times, but cycle time is a fairly complex metric. There's a lot of stages that go into it and you really need to break down each individual component part. So what's your coding time? How long does it take PRs to get picked up for review? How long does that review take? And then once it's been approved, how long is it taking you to get it to deployed to production? Like any one of these stages could be your constraint within the system And you really need to go from that like high level efficiency view all the way down to like looking at the actual the actual segments that are getting bottlenecked and which PRs are the ones that are causing those bottlenecks.

28:59All right.

28:59Dan Lines:So let's move on to the last pillar then. And this is the X developer experience. And this the purpose of this is just to ensure that any gains you get from AI are sustainable and human centered. You know, the humans are the ones that have to operate these systems. And at the end of the day, they should be satisfied with how they're performing. And that's why we picked the North Star metric for this to be developer satisfaction. So, you know, Apex kind of treats DevX as almost like a guardrail, right? So if your cycle times are improving, but your developer satisfaction drops, like maybe that's an indicator that your productivity gains are actually just fake.

29:39Dan Lines:They're illusory or unsustainable. So let's talk about developer experience, Dan. Like, like how, how does this work within Apex?

29:47Ben Lloyd Pearson:Yeah. Like we said earlier, the A, okay. The AI leverage, the P, the predictability, the E, the efficiency, and now the X. To me, they all relate together. They all, they all relate. And the way that I like to think about it is if I'm leveraging AI in the right way. And what we talked about is making sure that AI has the right impact, not just adoption,

30:13Dan Lines:right?

30:13Ben Lloyd Pearson:the right impact if i'm improving my predictability devs want predictability i mean when i when i was a developer like chaos kind yeah it's chaos might be fun on a friday when i'm experimenting but not when i'm actually like on the hook to deliver my sprint on time i want less chaos so i want to deliver great work i want it to be in an efficient way right i don't want to create a bunch of PRs and have them get stuck in the review process or not be able to get deployed. That's the E. And when these things come together, yeah, I think kind of like the final check is, okay, let's just validate that if we're doing well with the A, the P and the E, the satisfaction is there.

30:59Ben Lloyd Pearson:And usually what I see is like, if the AI leverage, the predictability and the efficiency are looking good, the satisfaction is usually there. Now you might catch like a red flag or something like that. But anyways, I kind of think that's okay. Maybe that's like the final check to make sure that we are listening and we're on point with the other aspects of APEC. Yeah.

31:20Dan Lines:And I think it's important to remember that satisfaction can be measured at multiple levels. So you can look at like the overall satisfaction, like how, how, how do you feel about the way we do work and your job here and all of that. But then the qualitative side of this is really great for like digging all the way down to the specifics. So like looking at the teams that are frustrated with AI and getting direct feedback from them about why they're frustrated so that you can address it or looking at the teams that have been incredibly successful with AI and see what are the things that make them flow really well that you could maybe take from them and apply to other teams within the organization.

31:57Dan Lines:So that's I think what I really love most about this last one is how you can both measure it at the high level, but then go all the way down to the individual teams and individuals themselves who are being impacted by AI and just get direct feedback from them.

32:13Ben Lloyd Pearson:Yeah, well said.

32:15Dan Lines:So let's talk before moving on just about connecting this all together. So we've been introducing this framework to a lot of people so far. I think a lot of people look at it and they just wonder like where do i start you know there's there's four pillars here uh which one's for me so so dan is it is it a matter of like i have to pick one of these that is the most important to me or do do we linear b have an opinion about which one you

32:41Ben Lloyd Pearson:have to come in and pick first or how does that work no i i think the leather picks you it's like uh i don't know like in harry potter when you get selected to one of the houses you don't pick it It picks you.

32:53Dan Lines:You get a nice leather shoe and it's like, this shoe was for me.

32:56Ben Lloyd Pearson:It picks you. And what I mean by that is what I'm seeing is, okay, when customers come in and they're kind of like, let's say that you're early on in the AI adoption journey. Let's say you're kind of at the starting point. Usually the P and the E, the predictability and the efficiency is the place to start for you. That's how Apex selects you. Why? Because still at the end of the day, you got to be predictable. You got to be efficient. That's the business of engineering. Deliver value on time and do it in an efficient way. And then you work the A in later. Go ahead.

33:31Dan Lines:And more importantly, if we know that AI is an amplifier, if you have bad efficiency, if you have bad predictability, AI is going to make it worse rather than better. Great point. Right. Yeah.

33:42Ben Lloyd Pearson:So I think that middle is like the core of it. Now, if you are on the other side and you're saying, hey, you know what? I made a bunch of promises to the business around AI adoption and the impact of AI. Or, hey, my P and my E are really solid, like you said, Ben, and I'm really looking to amplify. So I'm getting that AI leverage. Then you start with the A. And then you work your way to the right. Okay, more AI adoption. And I'm going to make sure that my predictability, my efficiency, and my satisfaction remain constant.

34:18Dan Lines:Yeah, I like that approach. very, very flexible, kind of meets you where you are rather than trying to form fit you into it. Right. It selects you. Yeah. All right. Well, I feel like we've gotten a really great breakdown of what Apex is, why we brought it to the market and why we think engineering leaders should be leveraging it today. Before we close out, I want to cover just a couple more things related to this. And the first is how you put Apex into operation. Right. So as a part of this, we have a guide that that we'll include in the show notes with this. But we also included a recommendations for a specific rhythm of operations around these.

34:56Dan Lines:So for example, you might want to track your AI, your AI leverage on a weekly basis, because it's changing so frequently, or you might want to look at predictability once per sprint, because it's just a natural cadence to analyze that. Whereas something like DevEx or your metrics that could be monthly or quarterly, depending on like how you as an organization feel about this. So why do you think, Dan, is the cadence so important to Apex actually working within organizations?

35:23Ben Lloyd Pearson:I mean, I just feel like the business of engineering, the whole part, like one of the main, I don't know, tenets of engineering is to have a repeatable cadence. Engineering is like a machine that needs to be operating in a smooth way. When the cadence gets broken, engineering is broken, the business is broken. And so like you said, yeah, like predictability, the P of Apex, go with the flow of your sprints. Maybe it's two weeks, maybe it's three weeks. If you're working in Kanban, it's weekly. On the efficiency side, yeah. I mean, I like to look at it, let's say twice a month. Some organizations look at it monthly.

36:04Dan Lines:On the AI leverage side, that's a monthly mode for me right now.

36:09Ben Lloyd Pearson:And I can see actually some some of our customers are even like, okay, we're kind of under the gun here. So it's like weekly reporting on all of these. Yeah. You put it to the business, you owe it to the business. It's a quarterly executive report or if you're going to survey developers, yeah, you do it at a quarterly cadence. But the whole key to me is like the business of engineering runs on cadences and therefore like the apex framework, I think fits into the natural cadence of how engineering orgs operate.

36:39Dan Lines:Yeah, that makes a lot of sense. And it's one of the things I like the most about the framework is it's very easy to just, you know, whatever cadence works for you. Like you said, if you want to keep tabs on AI like every day, like because you're moving that fast, like do it, you know, it makes sense for you. But if you're an organization that's moving a little more slowly or uses it more for executive reporting, maybe a monthly or quarterly cadence makes a lot more sense, but the flexibility of Apex, I think is what one of the things that makes it really powerful. All right. And then I want to close out by just talking about how we actually operationalize this data.

37:14Dan Lines:So, you know, a constant theme we've always had at linear B at dev interrupted is that, you know, visibility doesn't matter if you're not taking action upon it. So, you know, I'm curious, Dan, from your perspective, like what's the first step to taking for an engineering leaders take advantage of APEX? And what should their goal be by adopting this framework?

37:36Ben Lloyd Pearson:Yeah, great question. I mean, the engineering leaders that I work with when we first start together, it's getting to a benchmark.

37:42Dan Lines:Let's come benchmark against APEX.

37:44Ben Lloyd Pearson:You got to see where you are. Yeah, like we said before, if you come in and you know, hey, I'm already predictable, I got great efficiency, I need to start with AI. So, okay, the A selects you. But oftentimes, times again apex is a balanced uh framework let's see where you are from a benchmark perspective and then let's decide together and uh usually when you come in okay you come in you start using linear b you know you start using the platform it will show you show you on the benchmarks it's pretty obvious where you'll uh same way the benchmark will select you you know it kind of lets you know hey let's start here and then build a progressive plan so get your flashing warning

38:23Dan Lines:signs. Here's the thing for you to solve today.

38:26Ben Lloyd Pearson:Come get your benchmark and then see where you want to improve. Set a goal. Those are the first two steps.

38:32Dan Lines:Yeah. And then, of course, I think when someone does that, they're going to encounter a lot of common bottlenecks that we see time to time. Things like code reviews, for example, or frequent bottleneck. So I'm just curious, what's your opinion on once you've got these metrics, what's the next step that someone should be taking and the benchmark. Yeah. Yeah.

38:55Ben Lloyd Pearson:Yeah. Yeah. So at linear B, we're always doing a metric and then action. Once you see all of your metrics, you're going to look at them, you're going to get benchmark and what's a natural thing to do. Okay. I see these metrics. Maybe I'm talking to like the linear B MCP. I'm getting my insights at the end of the day. You got to take the next step to action. So like for us at linear B, we would say, Hey, let's go roll out the AI code review let's put in some get stream rules let's work with uh let's turn on worker b these are the things that make the a the p the e the x actually improve um so yeah action's the next step

39:32Dan Lines:yeah and i think in particular ai when you're adopting it into your you're again putting it into your critical paths uh it's really useful to to have deterministic controls that keep it on guard, like keep it on like inside of guardrails, you know? And yeah, I think without that, it's very easy for a team that might otherwise be successful to be one of those teams that gets overwhelmed by AI slop, right?

39:59Ben Lloyd Pearson:Let's keep them in line.

40:02Dan Lines:All right, Dan. Well, it's always great talking to you about these topics. I think this is just a really great reminder today that, you know, the tools are changing very rapidly at times, especially now, but the goals really are, have always been the same. You know, we want to deliver value to customers. And I feel like we've really only scratched the surface of, of what we could get into with, with Apex and, you know, how it can impact engineering leaders. So I definitely think we're going to have you back at some point to, to talk about this. There's I think a lot of room for follow-up content that I'm hoping we'll, we'll be able to bring on to Dev Interrupted in the future.

40:38Dan Lines:But yeah, thanks Dan for, for coming out today.

40:40Ben Lloyd Pearson:Thanks for having me, man. Can't wait to be back next time. When you call, I'll be there. All right.

40:49Dan Lines:Awesome. Well, for those of you that are listening, if you want to see the full APEX operating model with the metrics and the visual breakdown of these pillars that we've discussed, you know, you can head over to your favorite search engine, look for the Linear B APEX framework. We'll also have a link in the show notes if that's easier for you. Dan, thanks to you and our audience for joining us today. If you found this episode helpful, feel free to share it with another engineering leader who's navigating this AI transition. I'm sure they would really appreciate the help right now. And you, our listener, are in a place to help them because of what you've learned here today.

41:22Dan Lines:So help us out, help your friends out, share this episode, share the guide. We'd love to hear what you all had to think about it. And we'll see you all in the next episode.

41:39Dan Lines:AI is everywhere in software engineering, but most teams still can't prove its impact. That's where the APEX framework comes in. APEX is a new operating model for engineering productivity, designed to measure AI where it actually matters, at the pull request level. It connects AI activity to delivery outcomes, not just tool usage. APEX is built on four pillars, with AI leverage, predictability, efficiency, and developer experience. Apex helps you increase throughput without sacrificing delivery confidence or burning out your team. Because speed without predictability creates chaos and faster coding often shifts bottlenecks downstream.

42:16Dan Lines:If you want to operationalize AI the right way, Linear-D and Apex gives you the system and the cadence to do it. Download the guide and start measuring what matters.