Labs: Inside the Top 10% of Engineering Orgs

0:28Hi, this is Nathan Harvey. world better. There are no right or wrong answers. We just want to hear about how you and your team develop and deliver software. And you know, considering some of the questions might just help your team identify some areas to improve starting tomorrow. You can fill out the survey at bit.ly slash 2023 Sodor Sponsors. We'll put a link in the show notes. Really appreciate you taking those 15 minutes to share your insights with the entire industry.

1:07Welcome to our latest Linear B Labs episode. So these are episodes where we dive into the latest data, we look at our research and thought leadership in engineering, and then we see how we can apply that more tangibly to our own dev organizations. This episode is actually a follow up to one of our first labs episodes where we took a look under the hood. We had about 2000 dev teams and we wanted to find out what metrics the top 10%. So kind of thinking about the elite portion of those dev teams achieve. So today, we're going to look at the most recent data we have on what makes an engineering team elite with help from a new voice on Dev Interrupted, Ben Lloyd Pearson, Linear B's Director of Dev Relations.

2:06Ben, welcome to the show. Thanks, Dan. It's really great to be here. Yeah, it's awesome to have you here, man. Ben, could you let our listeners know a bit about your background and why you love doing this stuff and talking about this stuff? Yeah, so I've worked in DevRel for nearly 10 years now, and my career has been split pretty equally between engineering and marketing teams. So I've always kind of had this innate affinity for technology like my entire life. And I love this career choice because it really lets me merge my creative and my technical sides to produce persuasive and informative content for software developers.

2:48So before I got into DevRel, I was a bit of a technology jack of all trades. I bounced between various roles in web development, systems administration, network management, IT support. And that was where I sort of built this technical foundation that I draw from today to produce all this DevRel content. And, you know, over the years, I've worked with some of the largest and smallest engineering organizations in the world from massive companies like Samsung and Verizon to small startups like Linear B. And a lot of my work is focused on helping these teams achieve high impact goals in an efficient way and tracking those impacts via metrics.

3:25So I've also been involved in a lot of open source community metrics in the past. And I really just love getting to apply that expertise to evaluating top performing engineering teams. And, you know, every team has its unique nuances, but there are also practices that I've seen over time that are pretty consistent from team to team. And the fact that Linear B can evaluate such a large volume of engineering metrics puts us in a pretty unique position to quantify software development best practices. And I think that's really neat. Yeah, that's awesome, man. I wanted to ask you, actually, I was going to ask you this before the show and I forgot.

4:04But the role that you have with this, you know, director of dev relations, we won't get into like the specific numbers, but it's a pretty lucrative career path. And like, I think a really fun career path. and for the people that are listening and that maybe are developers, but like want to try something new or like how do I get into this dev rel mode? Do you have any advice of how to like transition into that or like how you got your first job or if I am a developer saying I really like this stuff, I want to like enable communities and it seems like there's a cool path there. Like any tips for the audience?

4:48Yeah, you know, communication is probably one of the skills that I think a lot of people, particularly people coming from technical backgrounds, can struggle with the most. So learning how to do things like write blogs, get comfortable speaking to people about technical subjects, practice breaking down really complicated things into simple concepts that people can understand. The more you can learn how to communicate the value of the technical work that you're doing, the better, the more potential you have to sort of move into this type of role. And, you know, DevRel people like me are always looking for friends over in the engineering organization to help us with content.

5:26So if you have someone who does DevRel already in your organization or who publishes technical blog posts or other content, you know, it can be really helpful to get involved with them, you know, contribute to what they're doing. I mean, obviously you speak really well, like present well and all of that. But one thing that I will say, if you are thinking about this career path, it doesn't mean that you have to be like perfect in your presentation. Like, I don't know, like you would see like a sales pitch or something like that. I think really the people that do well in this role. Yeah, you have to be comfortable to go in front of an audience and put yourself out there a bit.

6:04But it doesn't mean you have to be refined. Like the more real you are about like being a developer, like implementing a solution, it could be good to get your feedback on this. but it usually goes over great, actually. It's not as polished. What do you think about that? I mean, what it really comes down to is combining synthetical and analytical skills. So being able to analyze a complex thing and break it down into fundamental components and then synthesizing that into some sort of content piece that you can share with the world. And it can be as simple as a configuration for some software you build that just helps illustrate how things work or some code examples or anything like that.

6:45So yeah, it doesn't have to be a finely polished solution all the time, as long as it helps developers and is presented in a way that is familiar to them. Yeah. Okay, great. Well, thanks for sharing that. I'll move us on to like the main point. So our producers don't get angry with us. You know, we're here to talk about this research and the data. And the first section that we have here, They have it titled The Power of Having Metrics. And the reason that it's titled that way is what we've seen with engineering organizations, engineering teams, kind of just even having these types of metrics that you're going to speak about accessible and visible makes a positive impact on how dev teams are working and actually improve some of the metrics fairly significantly.

7:37So Ben, could you take us through this research of the impact of having these real-time engineering metrics that are accessible for everyone? And along with what metrics your team should be aiming for? I'll start with the data that we drew from. So we analyzed organizations who adopted Linear B metrics and our improvement guidance last year. So this totaled about 2.8 million pull requests from more than 75 ,000 contributors. and the results that we saw reinforce many of the beliefs that we have about developer productivity. So 70 % of the organizations who adopted our product improved their cycle time, which is, this is a concept that I want to discuss here in just a moment.

8:23And then 65 % of these organizations improved their PR size. So overall, the organization saw PRs that were about one third smaller on average and their cycle review and pickup time all improves by about 47 to 49 percent each. Wow, that's awesome. And I really like, we'll probably work it in a little bit throughout the episode here. But one thing that you touched on is the PR size, which is a great thing to be tracking. I think I said it in another series that we're doing, but that's coming for me to start being is like one of those golden metrics. Like if I see a very small PR size, a very manageable PR size, usually then there's a really good experience for developers.

9:12And then kind of on the flip side, when you see those PR sizes getting like, again, like too large, like, you know, like 600 lines of change, 800, you start thinking, okay, things are probably moving really slow here. And then the other thing that I'll say as we start diving into some of these, what it means to be elite, usually, and this is where Ben, you're focusing a lot of your time with GitStream, usually those elite teams like a company with automation or things that are helping right in that PR lifecycle. How much would you say like automation factors into being a lead or not being a lead?

9:53Well, I guess the question is, how much time are you wasting on like repeatable actions that are really more of a distraction than a value add to the software development process. If you're constantly having to bug your teammates just to give a thumbs up so that you meet your one-size-fits-all review policy, then automating stuff like that can really make a big impact on this, right? Yeah, for sure. So we're going to move on to the next area here. We're going to start running through some really cool data for the rest of the episode. If you do want to take a deeper look, you can find all of this data that we're talking about at linearb.io slash benchmarks.

10:41That's where you're going to get your community benchmarks data. You can, you know, compare how you're doing against the community, that type of stuff. It's really cool content. You get it for free. So definitely check it out. And to set a little more context before we dive into each section, there's three different buckets that we're going to be talking about today. So the first one is delivery lifecycle metrics. The second one is developer workflow metrics. And the third one is business alignment metrics. And we'll go through each one of those individually. we'll start with what does elite developer life cycle metrics look like so ben i'll kick it over to you to go through this first grouping of metrics for engineering team so delivery life cycle metrics are your leading indicators for how long it takes a feature to go from programming to deployment so there are five key metrics that we track in this area first we have cycle time this measures how long it takes for a single engineering task to go through all of the phases of the delivery process from coding to production.

11:50Second, we have coding time, which measures the time it takes from the first commit until a pull request is issued. A short coding time tends to correlate with small PR sizes and much more clear requirements. Third is pickup time. This measures the time a pull request waits for someone to start reviewing it. And a low pickup time indicates that you have a team that is pretty responsive to the needs of others. Next, we have review time. This measures the time it takes to complete a code review and merge the pull request. Review time is a good indicator of how collaborative your team is. And then last is deploy time, which measures the time from when a branch is merged to when the code is released to production.

12:34So a lower deploy time is good when you're an organization who has a high deployment frequency. To summarize that, it's like Cycle time is the end-to-end result of all of those smaller ones that you mentioned. So the smaller ones are like the coding time, the PR pickup time, the PR review time, and then the deployment time. And actually, we've said it a bunch of times on this pod, but I'll say it again. Cycle time is one of those classic door metrics that everyone should be measuring. You need that at your engineering organization level, but also for every business unit, for every group of teams, for every team.

13:17But what we want to talk about on this pod is what does being super elite look like? And I think the number I have here would be the top 10 % of engineering teams putting up for these. Yeah, so there's a lot of variability in the individual metrics, but generally speaking, elite engineering teams have a cycle time of 42 hours or less, a coding time that is under 30 minutes, and pickup time, review time, and deploy time are each under about an hour. So those are like amazing times, right? So that's the top 10%. And that's why I was talking a little bit in the beginning of the episode here, that a lot of these teams that are achieving that level of efficiency, you have to accompany that with automation.

14:10like a Git stream or another tool like that, that's saying, hey, let me auto assign the PR to the right person, or let me actually do an automated review and say, hey, this is really low risk. We're not gonna have anybody physically waste their time reviewing it. That's where I see a lot of these things that you can kind of get that like, I don't know what you would say. What would you say? It's like a cheat code in video games or like jump to the next level. Left, right, left, right, left, right. A, B, select, start. Yeah, is that Contra? I don't even remember. No, I think that's like NES Contra.

14:49Yeah, exactly like that. Could you say like a word or two about GitStream or like any like tool that's like that of what it can do for PR pickup time and PR review time? Yeah, well, I think it really just comes down to unblocking reviews where you can and then adding context and appropriate assignments where necessary. So if it's a thing that isn't going to hurt you to just let it be merged without any sort of deep review, then just do that. Just let it go through. And then that way, you're giving the space for the things that actually do need additional human attention to fully understand. And then when you're bugging someone to get a review on a PR, they know that because we've implemented these automations, it's something that actually is important and I should pay attention to and I should you know give it the attention that it needs.

15:44So Ben when we're thinking about in particular the middle of that cycle time with the PR pickup time and the PR review time one of the things that I hear you talk about that's different from the really elite engineering teams and the engineering teams that are suffering a little bit more in terms of dev experience is cognitive load and the idea of, okay, I have to go on to a new, like I haven't thought about this PR in a really long time. And now I have to come back and answer a question versus kind of like, okay, it's in my mind. It's in my flow. How do you think those long wait times versus like the short wait times, like it affected developers experience?

16:33Yeah, I think there's kind of two sides to this. And the first is just the nature of the job that a software developer has. They need to have a lot of focus on the task at hand, and they also need to have a lot of knowledge present in their mind in the moment to fully understand how to approach the challenge. And because of that, it takes time to sort of construct that level of focus. So anything that draws you out of it, it's not like jumping on and jumping back off. You kind of have to rebuild that present knowledge that you can regain the focus. The other side is that this applies to PR reviews as well.

17:08So if you're looking through a list of PRs, you probably don't know a whole lot about what's in it until you click into it, look at the files that are changed, figure out what's going on, and again, build that present knowledge to understand what you need to just to review the code. So, you know, an elite engineering team is going to make sure that when you're tasking a developer with that level of focus and that level of knowledge, you really want to make sure it's necessary. It's important that you're reducing the cognitive load however you can to things like PR labels and integrations that give you details about the PR, stuff like that.

17:47So, you know, it really just comes down to like letting developers focus and, you know, build that present knowledge. yeah that context matters so much there's so much that goes into i gotta look at someone else's code and like under dive into that world versus what i was doing and vice versa so i'm really happy that you hit hit on that point yeah yeah and keep in mind if you submit a pr and have to wait three or four days to review it you may have already moved on to something completely different and now you have to come back and and you know like i said present knowledge that's yeah it like feels so bad to go backwards.

18:23The next section that we have here is around elite developer workflow metrics. So this is our second group. Ben, what do you have here for us around elite engineering teams, what they should be measuring, and then we'll get into what good really looks like. Yeah. So developer workflow metrics, these are measurements that look at frictions or inefficiencies within the process. So we have three of these. The first is deployment frequency, and this measures how often code is released. So frequent deployments represent a stable and healthy continuous delivery pipeline. Second, we have pull request size, which measures the lines of code that are modified in a pull request.

19:05A smaller PR is easier to review, safer to merge, and it tends to correlate with an overall lower cycle time. And then third, we have rework rate, which measures the amount of changes made to code that is less than 21 days old. So if this number is high, it could signal code churn and is a leading indicator that there might be some quality issues. Okay, great. And what does doing super elite look like here? Yeah, elite teams deploy daily. The average PR is less than 105 lines of code and reworth rate is somewhere under 8%. Let's spend like one more minute, just because it's my favorite one, on that PR size.

19:45So the data that you have is for the top 10%. This is still 10%, right? Top 10? Yep. Yeah. OK, so for the top 10%, we have average PR size is less than one hundred and five lines of code. If we go back to that developer experience and thinking about that, how do you think about the difference between, OK, I'm going to review something that is one or five lines or less. right versus getting up in there into like the 400 500 600 800 lines how does each one of those feel as a reviewer well there's sort of what you might theorize but then there's the practical reality as well and the practical reality is a massive pr is probably not getting the attention that it needs and no way maybe getting you may be getting more of the just lgtm and go go on of their day because it's just too much mental burden to, you know, they don't want to block a release or anything like that just to deal with this massive PR.

20:54You know, a smaller PR, it means you're probably sharing your code with your teammate much more frequently. So people will tend to be more in the loop with each other about the work that's happening in the moment, plus the obvious benefit of just not having to read a book to get through a single code review. too yeah it reminds me of like school assignments from when i was a kid it's like if i'm in english class and i have to read this huge book there's no way that i'm actually doing this i gotta go well i'll like date myself you remember cliff notes oh yeah cliff notes gotta buy cliff notes because i can't actually read this huge book but if you're telling me hey i want you to go like Like I went to school in like the night, like late 90s.

21:42OK, go read like a small Internet article and like tell me what you think about it. Yeah, I can do that. So I think it's kind of a similar thing. If you're giving me one hundred and five lines of code or less, like I'm into it. Got it. I'll read every one. I'll give like really detailed feedback. I'll block off, you know, like a half an hour or less of my time. I'll still get my own work done. if you're coming at me with like 400 500 600 lines of code i'm giving a cursory review at best and i'm really thinking to myself like what the fuck is going on here like i can't like comprehend all of this so i think like uh you know everyone that's listening to this pod it's not just about how quickly you can deliver work which it i mean it is about that but it's not only that It's like, what experience are you giving to your developers in your engineering org?

22:35Because that matters. It's like highly competitive market right now. So, yeah, man, anything else to add to that? Yeah, you know, something we didn't really talk about too much was merge frequency. Yes. We don't really have any specific recommendations on this from elite teams. But, you know, it is something that you should also consider because it does, you know, if you have teams that are that are merging daily, it means you are keeping things small, digestible. You're being much more collaborative, having a faster review cycle. Yeah. Thank you so much for bringing that up because that's also, I know we don't have like a benchmark data on it.

23:11So our benchmark data is on the deployment frequency, but the merge frequency, especially if you're listening and you're a larger organization, you might be saying to yourself, okay, like the deployment stuff is someone else's job or like another department or whatever, depending on how you work. But merge frequency, getting that code merged, that's what feels good. That's what feels good for developers. So make sure that you're tracking that. As we are coming up to the end of our time, we do have another section that I'm pretty excited about. And it's titled the state of business alignment metrics.

23:49this is a section that is I would say like new and evolving new data is coming in all the time the way you think about this type of data or way the industry might be thinking about it is probably going to change like every six months here but there's two areas that we want to talk about in terms of business alignment metrics one is going to be more around resource allocation where are you investing your time into? And the second one that we're going to talk about is planning accuracy. Sometimes people talk about capacity accuracy. That's more so, okay, am I delivering on time and that type of thing?

24:33And I think, so out of the two of these, let's first start with resource allocation. And the way that we think about it at Linear B, and you can find this in our app. And also the way that the industry is starting to think about this. I think everybody, maybe we got to see if our producers can include this in the pod or like in the details, but Iconic came out with a really good report. And there's four areas that most engineering organizations are now talking about resource allocation when they're talking to their board, when you're talking to executive peers, when you're thinking about where are our engineering time going, the four of them, I'll read them out loud here, are keeping the lights on, internal productivity, quality improvement, and new capabilities.

25:33So let's see where we want to go here. Ben, where do you think we should take it first? Yeah. So, you know, I've seen the same research and I know this is something that is relatively new to us here at Linear B in the grand scheme of things. So, you know, what do you think are like some common breakdowns of how like those four categories like manifest within engineering organizations? Yeah, absolutely. So I'll go based off the iconic report and the way that they break it down again is like keeping the lights on, internal productivity, quality improvement and new capabilities. And when they talk about keeping the lights on, that's anything that has to do with responding to bugs found in production, anything that like an SRE team would have to deal with, lag in prod, the service is down, something like that.

26:25And the way that they think about that is you just need to allocate time because it's not really elective time. So it's like 19%, for example, if you're making$50 million ARR or less, 15 % if it's more than that. So 19 % of allocation going to just keeping the lights on. And then these other three categories of internal productivity, quality improvement, and new capabilities, those are more electives. That's how you're having a conversation with your peers on the executive team or your board. And you're saying something like, okay, what percentage of investment do we want to do to internal productivity?

27:13What percentage do we want to do quality improvement? What percentage do we want to do to new capabilities? And I'll just read the numbers of the report. So 50 million ARR or less for internal productivity, it's averaging 14 % of time. For quality improvement, which is a little bit of a misnomer, what it really means is improving the usage of existing features that you've already release, that's at 32 % of that elective time. And for new capabilities, these are new features, new things that can bring in sales. The average is 55 % of time. So that's what the data says. Gotcha. And I imagine after hearing that, some organizations are probably comparing themselves to how they stack up to that.

28:04And maybe they don't really have a way of gauging that yet. But what do you think are some signs like, you know, if we're overinvesting in new features or like underinvesting in things like internal productivity, like what can an organization look for to identify where they might have issues? Yeah, absolutely. We're fortunate now because in Linear B in our app, we're reporting on this type of stuff. So I get to talk to our customers about what they're running into. and I wrote down three situations that I hear the most now based on this data. So the first situation that I hear is under investment in internal productivity.

28:47And again, what is internal productivity? It's all the things that you would do to decrease your cycle time, to increase your deployment time, to increase your merge frequency, to decrease your PR size. This is investment in automation or practices. And again, what the report says is you want to average about like 14 to 15 percent. Oftentimes we're seeing like five, six, seven percent there. And that's a situation where, OK, the developer experience isn't as good. Maybe we're not delivering on time. That's an area that you can then go back to your CEO or your executive peers and say, hey, we have this information now.

29:28we're a little underinvested into internal productivity. Our dev experience is suffering. You might wonder why we're not delivering as predictable as we want to. Well, that's why. And therefore, I'd like to have the conversation of increasing that to about 15%, which is industry standard. And that's like the first one. That's a great conversation to have. And the only way to do it is with data. Otherwise, it's really like hand wavy of like, okay, we need to do this internal stuff that, you know, is not like a new feature, that conversation doesn't go as well as if you have data. So that's the first.

30:07Why are developers so angry is what you're saying? Yeah, you got to like, it's like anytime that you're talking to an executive staff, you got to come with data. So that's the first one that I've seen under investment in internal productivity. The second thing that I've seen, and it kind of relates to that, is overinvestment in new capability. So probably from your CEO, if you're under 50 million ARR, you might be more in a startup mindset or you just might be more in like we got to compete against competition, which is all well and good. you're going to get that pressure on the product team to come out with new capabilities.