The McKinsey Developer Productivity Debate | Ori Keren & Kelly Vaughn

0:00I feel lucky that I have the opportunity to be here in this crossroad where this is being implemented. You can see this space crossing the chasm now. A lot of big companies are coming in. I mean, McKinsey's writing about it. McKinsey's writing about it. Big enterprises are coming in. To be the ones that kind of say, hey, this is the right way to roll out. I love it. I love this responsibility and kind of help educate the market. I think that you nailed it at that last point there. If there's anything I learned from working in video intelligence, it's that if you give people the tools to do something, there are always going to be people who abuse those tools.

0:33from a business ownership standpoint or as an engineering leader, when you're having an impact on what it is that your company is going to be building, you know, think about how it should be used, but think about how else it could be used, you know, in not so great ways. You can't prevent everyone from doing any one particular thing, but at least you can help kind of steer the ship in the direction. Hey, listeners, we spend a lot of time at Dev Interrupted talking to leaders about what makes them and their team successful. But how do you know what makes an engineering team great if you don't have anything to compare it to.

1:04Mark your calendars because the 2023 Engineering Benchmarks Report is about to be released. And to celebrate, Linear B is hosting a virtual event for engineering leaders. Attendees will be among the first in the world to explore the landmark data. This year's report analyzed 3.6 million branches from over 2 ,000 dev teams across 32 countries. Deep dive into the data by team, size, geolocation, and industry. Plus, see how elite teams perform against Dorometrics and get acquainted with brand new benchmarks like Investment Profile introduced this year. The best part? Just for registering, you'll receive a free pre-release copy of the 2023 Software Engineering Benchmarks Report.

1:45Don't miss out. Be at the forefront of engineering excellence. Register today at linearv.io slash events or use the link in the show notes. Hey, everyone. Welcome to Dev Interrupted. This is your co-host, Connor Bronson. And today I'm delighted to be joined by two of my favorite guests, Linear B co-founder and CEO Ori Keren and Kelly Vaughn, director of engineering at Spot AI. Ori, Kelly, welcome back to the show. Thanks for having us. I'm particularly excited to be doing this with Ori because we were talking a little bit earlier that we've listened to each other on this podcast multiple times, but this is the first time we've ever actually been able to do this together.

2:20Same goes here. Excited to be here and to be part of this panel. you know it was the the same thought i had coming into this where i've talked to each of you separately about a lot of these topics that i know you're passionate about i've talked to you each about in fact shows that you've both been on and you know or he's been like oh yeah you know kelly said this thing and i was like oh yeah yeah yeah she recommended a book to me and and you know kelly you've been said oh you know this thing on developer productivity around entering metrics like i heard or he said this that was really interesting so it's it is really great to have you both actually on the same episode.

2:51We need to do more of these. And the impetus behind this episode is kind of a controversial one, actually. There was an article that was released titled, Yes, You Can Measure Software Developer Productivity by McKinsey. And it immediately ignited debates within the software development community with Kent Beck, software engineer and creator of extreme programming, calling the report so absurd and naive that it makes no sense to critique it in detail. That said, he did, to some extent, go on to critique into detail when Gergely Oro's The Mind Behind the Perkmatic Engineer cataloged what was happening and reached out to Kent to join forces.

3:31Together, the two of them wrote a two-part series as a response to the McKinsey piece, definitely worthwhile reading as well. We'll link all these articles in the show notes. And as I alluded to at the top of the show, you're both very passionate about this topic. Ori, you've obviously dedicated much of your life, including founding Linear B, to many of the principles discussed in this debate. And Kelly, you are one of the most vocal and outspoken engineering leaders I know. We've talked about this topic multiple times. If there's a cause to stand behind what's best for engineers, the two of you stand there 10 out of 10 times.

4:02You truly have what's best for the community in mind. You're both authorities on the subject. And we want to start by asking you both, and Ori, we'll kick off with you. What were your first impressions as you were reading that McKinsey article? Yeah, the initial thought was great. It's great that more people care about this. More people want to express their opinion and want to participate in this debate. And I think it's an important space that we need multiple views. and actually, you know, reading through it, the first parts made sense. Like, you know, the motivation was there, the reference, you know, Dora, which is like the de facto standard now for some of the things.

4:44I also think they came in with McKinsey as the advantage of coming in and represent, because we've all been there, the voice of the CEO, CFO exists. We can ignore it or we might not want to listen to it, but it exists in this conversation. So those are like the parts that I like, but the parts where it started breaking for me were the area where people started talking about individual metrics of developers. That's the area where I feel strongly that you cannot use a system or definitely not the same system to measure individual metrics. It's almost like creating an autoimmune system that attacks its own culture.

5:29So in a high level, again, I like the motivation. I like when more people want to express opinions. But those are the areas where it started breaking for me. Kelly, I see you nodding along on that. Yeah, no, I love the analogy of the autoimmune system. That is so incredibly true. Yeah, I mean, I can echo that same sentiment. When I started reading that, I'm like, okay, I mean, this is a very hot topic. Why are people kind of so upset about this intro? Or I'm just in the intro, I should say. This seems fine right now. But same exact thing. It started breaking down for me as soon as we started digging into the individual because I've seen it time and time again.

6:06It does not work. And then I got a good laugh once I got to the very end of it, which we'll probably dig into eventually here around letting developers do what they do best as far as avoiding the non-coding exercises of things that they could be doing so they can focus on coding. I have a lot of opinions on that one as well. Kelly, it sounds like you have a strong opinion around this, what devs sort of focus on piece. I want to talk more about some of the things that two of you alluded to, this autoimmune response we're talking about and how it can be dangerous, the fact that CEOs and CFOs do want this and we have to acknowledge that.

6:43So why don't we just jump in there? Yeah. So talking through the example, I can read it word for word so I don't kind of fumble exactly what it said here. So it said, for example, one company found that its most talented developers were spending excessive time on non-coding activities such as design sessions or managing interdependencies across teams. In response, the company changed its operating model and clarified roles and responsibilities to enable those highest value developers to do what they do best, code. When I read that, I actually laughed out loud. Because, okay, the best engineers I work with are cross-functional.

7:19The best engineers I work with understand those interdependencies. They understand how their work impacts the business, how their work impacts the others who are also doing work in the same general space. And if you focus entirely on what you're doing as far as just coding, you're going to miss so much context. And that's how we start to go down rabbit holes of we're introducing features that the customers don't actually want, for example. Or we're introducing something that somebody's already working on. Or now you're introducing two very different types of patterns of coding into the same repo where you could have avoided doing that.

7:50How do you have these interdependent conversations? I'll pause there for a second or introduce which is the way your take is. Yeah, I totally agree. And I would add another thing. I think the best engineers that I've worked with, they're the best readers of code, which is a very rare quality to have. and one of the best things they can do to contribute to a team and we should talk a lot about, again, I love sports, so the sport analogy about how a team works is the best developers can read code. They have patience to read code. They sell code. They can do the best reviews. So, yeah, it started to feel a little bit like one-threaded thought or one-threaded mind there because, yes, the best developers can do a lot of things.

8:39And if you just let them code, you miss some of their greatest qualities. Completely agree. I would be terrible. I would actually be absolutely awful if I were the developer on the receiving end of this and all I was told to do was just code. Because where I thrive is on the people and the process side of things. And you would miss all of my greatest qualities, all of my terrible jokes you would never get to hear because I just so focused on coding. Ori, based on what Kelly's saying, do you think that same challenge would face these top engineers that you've known over the years? Yeah, absolutely.

9:14So I think, like we said, top engineers bring much more capabilities and qualities of just coding. And they have this overview of the system. They can be the best reviewers, the best system designers. I also think that, you know, there are a couple of other things I saw there that got my attention. One was, there was an inner loop and outer loop that we all know that developers have. But outer loop, kind of like the deployment part was part of the outer loop. And we say now, you know, you build it, you ship it, you own it. And I think, again, in a utopia, in a perfect process, you build the thing, you code it, you test it.

9:58If you can ship it right away, it's still part of your inner loop. So I think it's also like something they missed with the outer and inner loop. And the last thing that caught my attention there that was, you know, there's a lot of buzz around Gen AI and what it brings to the table and everything. And it's a topic that I'm really passionate about. I think one of the things the article said is like, you know, it was proven that developers can go 2x fast. Now, what I would want to really emphasize here is, let's say you can stream code 10x fast even, right? If your pipes, development pipes, are still the same pipes that exist, they're still the old ones that are narrow and you don't enable improving your development pipeline, you won't move 10x fast.

10:49It's still going to be stuck there. So that's an area that I'll be happy to develop later on in this conversation. Kelly, were there other key points of the article that stood out to you that you want to highlight? I do think the inner outer loop, I went back and read that again after I went through the article the first time because that was a little bit surprising to me as well. That the idea that all of these quote unquote outer loop activities are more of a nuisance necessary piece of it. But there was also a section in here, and I don't remember exactly where it was, but it was, it was, and I think it was in relation to this, where it almost came off as the roles that are manual QA, the roles that are around, you know, the deployment processes.

11:31Like these are, these are, these are important roles for a reason. And it should be treated as such and not as this extra task that you just have to do. So to both your points here, I think there is this consideration of the software development lifecycle sometimes that overly focuses on just, oh, the coding part and not the how do you get the code into production? How do you QA it? How do you make sure it's getting reviewed by the right people? How do you ensure that it fits within the context of what your team needs? How do we ensure it fits within the context of the business needs? And I know I've talked to you both individually about elements of this.

12:06And frankly, it's part of why CEOs and CFOs are going to keep caring about engineering metrics and developer productivity. We can't get away from this conversation. That's why it's so important that we frame it in the right way and that this conversation is engineering led so that it doesn't damage culture like you alluded to earlier, Ori. And I know that's part of the concern that Gergely and Kent had in their critique. What resonated with the two of you about the critique from Gergely and Kent of McKinsey's article? Yeah, I really liked the mental model that they use throughout parts, like both parts one and two, where they're thinking about it as effort, output, outcome, impact.

12:46And I had to reread that multiple times because I always forgot the order that they were talking about. But it makes complete sense as far as where a lot of these metrics tend to lean into and why you end up missing that greater picture. The outcome and the impact are so incredibly important to the work that you're doing. And when you start to dig into some of the developer metrics that you're looking at from not only these systems, when you're thinking about lines of code and story pointing and things like that, you're starting to look really far into the effort and output side of things. And you're completely missing that outcome and impact.

13:20And it kind of drives us back to that beginning conversation around developers are more than just the code that they're outputting. The developers need to know the context so they know the impact that they're going to have. They know what outcomes are actually coming from the work that they're working on. So that part really, really resonated with me. I totally agree. I love that framework that the effort output outcome impact. I love that they push back on the individual metrics and talk about those areas. I love the sport analogies. They did references to that as well. I do think we can talk about some things that I kind of in disagreement with them.

14:00But overall, I think it was a great response to kind of like paint the perspective of how people who were coming from this industry are thinking about it. I had the privilege to speak to Kent Beck once, the founding father of the Agile Manifesto, etc., the extreme programming person. So I really like the response. I really like the framework. So what are the parts you thought they missed, Ori? Again, it is a not big miss, but I think one of the things that caught me really strongly is when they took the Dora metrics and they tried to map them into that framework, effort, output, outcome, impact.

14:39And I think if I remember correctly, I don't have in front of me, they kind of positioned the Dora metrics in outcome and impact. So it's almost like I think CFR was in impact and the rest were in outcome. And I think that the Toro metrics are actually output and outcome more and not impact. And I see this, you know, when engineering leaders come and present Toro metrics to the business. The business can't understand like a CFR as an outcome, as an impact. So it can understand, oh, so the outcome is you roll back less versions, you have more successful rollouts. That's a great outcome. but impact here is the impact on the business it's still hard sometimes for the business to kind of tie between them and that's I think a little bit of a blind spot that people that come like us that come from the space tend to have like over we think hey Dora Metrics will cover and it's enough to kind of like paint a picture of how our engineering organization performs so that's one thing that really stood up for me which I thought could be improved.

15:51I do think that's a little bit of a side effect of trying to fit one mental model into another mental model. You know, I'm in agreement. I think, you know, when you're asking, when you put change failure rate into impact, the question you have to ask yourself is, why does this matter? And the why is not change failure rate. The why is the impact resulting from a decreased change failure rate or an increased change failure rate. And that is why these all need to be kind of shifted by one because you're still talking about what is the impact of that. This speaks to something that we hear a lot in business where it's, is this an operational metric for the business, for your business unit?

16:30Or is this an actual business metric, the impact piece? And I'm hearing that from both of you. I know, Ori, we've talked a bit about what we can learn from other business units, you know, like sales and marketing, they're kind of farther along the approach to metrics. And I also know there's some things we should avoid there. What would you take away from how other business units are approaching their metrics that we can apply to this developer productivity conversation? They spoke about these analogies and they talked on some interesting aspects of it, which I really appreciated. By the way, some of them, I think, you know, there was some description about like their sales reps who only care about their things.

17:14I think like... Ken had some certain opinions of sales. Yeah. Development is definitely team sports. After running a business now, sales is also a team sports. It's almost like, I love basketball. So maybe you want to compare it to professional basketball and college basketball, where in college basketball, you work more together. And in professional basketball, you still try to win as a team. But yeah, it's more about the individual. So there's still some analogies to that. But here's the thing, the biggest, biggest, biggest blind spot that I think everybody that talks about this space is missing is the big opportunity.

17:51And we can talk about, by the way, they talk about why you shouldn't measure the effort. And I think it's a risk measuring the effort if you talk about like individuals, metrics, but measuring, like mapping the entities in the effort and looking for inefficiencies there. And by the way, these are metrics that are only interesting to the engineering side. is really, really important. You can, this is developer experience, right? If you have a long build time, if you have like bottlenecks in your effort, you want to uncover them. So that's one thing of the second biggest opportunity that I see here.

18:25I like the analogy to sales because if you think about Salesforce, so you can see the funnel in sales, in Salesforce sort of like the cycle time in engineering, right? That's a great analogy. But those systems took off when, And because you map the entities, now you could apply policies to actually accelerate the workflow. And nobody's speaking about that. Nobody's speaking, okay, yeah, there's arguments how we should go about it. But hey, let's take this huge opportunity that we have that now we're kind of mapping all the entities in the universe. And maybe, you know, code reviews could be 10x faster.

19:03And maybe CI and build times can be shorter because we have the data around them. we can find the inefficiencies in them. So that's like the analogy that I like with sales and what I think there is opportunity to learn around engineering efficiency from how sales efficiency evolve throughout the years. I completely agree with that. You know, when sales is looking at optimizing their time to close, how long it takes from outbound or inbound lead to actually converting this into a paying customer, you know, we're doing the same thing in engineering. So for example, for my team, I know that we can do a lot better on how long it takes us to actually close PRs from the time it takes to open a PR to actually deploying this into production or into even staging before we get to production.

19:52There's a lot of work that we can continue to do in our own development funnel and deployment funnel that we're actually looking at right now around our API deployment process, for example. You know, these are metrics that you can be analyzing, you can be using. Where things start to get dangerous is around when you start using these to measure the effectiveness of any individual engineer. And so that's why it's really, really important to map these to the entity and not to the individual. Couldn't agree more. Yeah, Ori, I know you and I have talked about this before, but a big focus for us is saying, let's look at team-centric metrics.

20:25Because engineering is a team sport, as you alluded to. we need to win together and we're all reliant on each other's workflows. One point I think that I'm not sure I hit it all the way. Once you map the entities around your process or your effort and you see where you have problems, theoretically, you can automate much more. That's like what Salesforce did with like, or even Jira, like how you input data and move things around. so and again i don't want to push our our own product but this is exactly what git stream does right since i have pull request and i have its context because we mapped all then it's now maybe 20 of them can be auto approved or the opposite like if if it's one that's really important in the roadmap and touches a very important things let's bring more reviewers or let's run only a selective portion of the CI, you can actually automate things.

21:28And that's the blind spot. Everybody's speaking about the metrics. This is like true workflow improvement that can come from all of the movement in this space. I think that's important though, because much like the engineering space is ever evolving, so do our workflows. And we're going to have to continue to invest time and resources into our workflows, into our reporting, into our process to identify these bottlenecks. and as technology continues to evolve, we're going to find ways to automate more things. And that's exactly what you're talking about there. Yeah, or you alluded to this earlier when you said, even if we're moving 10x faster on our coding, if there's a blocker in that software development lifecycle that the workflow's messed up and code just sits there for days, things like PRs, other issues, that's not going to really solve the problems of developer productivity.

22:17So I think it's a great point to bring up. we've seen some I think negative examples in the last year or two around people saying oh lines of code is how I'm going to measure which devs I keep and which devs I lay off these other decisions that as you alluded to at the start of the show become really toxic for an organization yeah absolutely I think especially if to use their framework especially when you move into the effort zone I agree with Kelly measure the entities and the process and if you have to, if you think it's right to also kind of benchmark and look at other aspects, other parts of your organization are doing, stay with teams if you have to.

23:01Never go to individual metrics, but stay with teams because with teams you can find like, hey, maybe this team has some sort of inefficiency, like Ellie said, in the PRU process. For some, it's about, I don't know, they're leasing every two weeks and they don't have CD. Okay, so let's fix that. So if you stay with teams and you focus on the entities and the process, there's still a lot of great things you can do in the effort area, even before like outcome. And Ori, the way I know that you're passionate about this is because I know that Linear B has actively stepped out of deals where certain companies have said, hey, we want you to drill down to these individual metrics because you're so bought into this long-term belief in the health of the engineering org.

23:50And you're saying, look, companies that are going to drive into individual metrics are the ones that are going to fail to actually achieve the outcome and impact they need long-term. This isn't the kind of partner we want. Yeah, yeah. This is a great question and it's always a dilemma because think about like DORA and all the academic research that kind of like lay the foundation for companies like us come and productize this. And it comes with responsibility. And I'll be very open and honest. Maybe in the beginning, you can't stand firm, but now we are big enough to say, hey, this is our philosophy.

24:27This is how we think you should roll out. And in some extreme cases, they say, hey, it's probably not a good match. It's fine if that's your philosophy, but we're probably not a good match. There are other tools. And this is how we recommend rolling out these type of solutions. So I think I said it in the past. I think I see this as, well, we're building a business and we, of course, want to make tons of revenue and be great. But I'm also super happy and feel lucky that I have the opportunity to be here in this crossroad where this is being implemented. You can see this space crossing the chasm now.

25:07A lot of big companies are coming in. McKinsey's writing about it. McKinsey's writing about it, big enterprises are coming in. So to be the ones that kind of say, hey, this is the right way to roll out, I love it. I love this responsibility and kind of help educate the market. I think that you nailed it at that last point there. If there's anything I learned from working in video intelligence, it's that if you give people the tools to do something, there are always going to be people who abuse those tools. And so the best way you can do from a business ownership standpoint or as an engineering leader, when you're having an impact on what it is that your company is going to be building, you know, think about how it could, how it should be used, but think about how else it could be used, you know, in not so great ways.

25:51And you can't always, you know, prevent people from doing like, humans are going to be humans. People are always going to try to game systems, they're always going to try to find a way to use something you don't want them to do it that way. However, when you can really, really, you know, have a firm standing on this is how we intend for this to use or to be used. And this is how we think you should be using it. And you provide those educational opportunities and educational materials about it. You can't prevent everyone from doing any one particular thing, but at least you can help kind of steer the ship in the direction.

26:24Absolutely. And I think this is a really important and nuanced piece of what we're talking about here. It's great that Gurgle and Kent talk about Dora metric space frameworks, I think these are important things to apply. But if we're not considering the long-term implications of where we're measuring on engineering teams, we're falling into the same trap as McKinsey. And this is where I think we see a lot of positive intent from this debate where people are saying, oh, well, we see this negative strain and some of the stuff that McKinsey's putting out, how can we do it better? But it feels like there's still an internal debate happening within the intern community about how to move forward.