Labs: Setting Engineering Goals and Reporting to Stakeholders

0:00We need to remember that less is more. So don't come with 50 numbers. It's not going to work for the business discussion. Focus on what really matters and always show how that aligns with the business. So if you're talking about predictability, yeah, this project is landing on time or 90 % of it is landing on time. And here's why. And here's the choices that I've made. That was when you start overcoming those glazed eyes of, oh, some numbers on something that I don't understand. You're seeing success when people are asking you to drill down. They're asking you smart questions about the data that you're showing.

0:37Gartner just released their market guide, showing that software engineering intelligence platforms help engineering leaders significantly improve both team productivity and value delivery. Through Gartner's in-depth analysis on the critical features of SEI platforms and how they can be used to drive engineering excellence, Linear B was named as a representative vendor, and therefore we're giving away a complimentary copy of Gartner's SEI Market Guide. Head to the link in the show notes to download your complimentary copy and learn how you can unlock the transformative potential of software engineering intelligence for your team.

1:11Hey, what's up, everyone? Welcome to another Labs episode of Dev Interrupted. I'm your host, Dan Lyons, Linear B COO and co-founder. And in our labs episodes, we dive deep into data and best practices that we find. And today, we're talking engineering goals and reporting. So what we're going to do, we're going to unpack the data and strategies that elite engineering organizations use to set goals that drive predictable software delivery. And also how you can report on your progress to senior leadership. Very, very important thing to do. And I'm joined by a voice who will be, I think, very familiar to many of our listeners, an expert who has helped countless organizations use goal setting to drive engineering improvement, Linear B's own CTO, Yishai Barry.

2:14Yishai, welcome to the pod. Thanks, Dan. Great being here. Awesome to have you on, brother. We're going to dive right in. Our first topic in general is around goal setting in the dual mandate. I know our listeners are likely aware of the meanings of some of these acronyms that we're going to use here. There's a bunch of them. But I think they're crucial to the topic we're about to discuss. And so just to make sure we're aligned, I want to quickly maybe recap some of this terminology. So we'll probably talk OKRs. So OKRs, Objectives and Key Results. KPIs, Key Performance Indicators. These are like the most business-oriented acronyms ever.

3:01Then we have SMART Goals, S-M-A-R-T, which are Specific, Measurable, Achievable, Relevant, Time-Bound Goals. So those might be some of the things that, you know, some of the acronyms that we use. Now, one of the key things we've noticed in goal setting for engineering teams is a shift in what goals need to look like. So engineering leaders today have a dual mandate of both expected to be elite in operational excellence. So when we think of operational excellence, I almost think of this as like maybe where engineering teams started. It's like the deliver the stuff in high quality, really fast. Do your engineering thing.

3:43We've been doing this, I don't know, 50 years now, 30 years now, ever since I was doing engineering. That's operational excellence. Like, be really good at engineering stuff. But the other side of it, and I think this is newer. I know it's newer. And I know the best engineering leaders talk to me about it all the time. A lot of our customers are doing this. It's alignment with business priorities. That's the second part of the dual mandate. And I think the reason is most organization has recognized, okay, our engineering team is like the lifeblood of the company. There's a product that goes along with it.

4:21We sell that product. Maybe it's a SaaS product. And if we're aligned on business objectives, we make more money. I think that's where it's coming from. And that's the other side of it. So that's kind of setting the table. So, Yishai, how has that dual mandate impacted how R &D leaders need to think about goal setting? Yeah, so definitely a shift, like you said. If you go back, you know, when I started working in engineering management and leadership, there was always the focus of, are we running a tight ship? Are we getting the most out of our developers and that spend? And are we effective in translating the cost of having a dev team into new features?

5:10Velocity is a very old word in engineering management. But also quality. Are we good at what we do as an art, if you like? And I think the strong engineering leaders always had the notion of being aligned with what the business needs. But it's more recent that this is surfacing as something that's measurable and something that needs to be tied to how we set goals, how we measure our organization, ourselves, how we report to our peers, to managing up. That is a little more recent, and there's good ways to think about are we aligned with the business? Are we not just running a tight ship and an effective operation?

5:56were also tuned to what the business needs, which means this is about delighting customers. It's about understanding the things that move the needle for the business and for the business to grow or to take the next step in its journey. We know software is eating the world, and every company now is a software company. I think a lot of that shift came from that. It's no longer one of the departments which happens to dabble with writing code for some either internal or even a product, all the companies are becoming software companies. If you run an airline, you're a software company. If you're selling hamburgers, you're a software company.

6:41So having the engineering department, it's rapidly becoming stronger. More of the spin is going there. decisions which used to live outside like in IT or even in other business processes have become engineering because engineering, everything is software. So now even business processes in many cases are run by engineering. All that burden now means that we have to really focus on are we building and investing our time and energy in the right places, which is the business alignment part. You know what's so funny about what you're saying? I was thinking a lot about it. I think it's really smart.

7:24And it's almost like this dual mandate, honestly, has kind of been there forever in a sense. Right now and for the forever future, it's like mandatory because of all the reasons that you said every company is a software company. Even if you sell hamburgers, you're a software company. But if I think back to when I first started, even in development, very, very long time ago, the best leaders, people would say, oh, yeah, I love that person. Oh, yeah. Why do you love that person? Well, that person can talk about when is the project going to be delivered. They can talk to the CEO about what the CEO's needs are, what are the most important things.

8:06They were almost like amazing translator. They had all of the engineering practices of efficiency, quality, you know, low change failure rate. And they had this other ability to say, OK, I know exactly what the business needs. And I can talk to all of the go to market team about how my team engineering is going to hit those goals. Like those were the best leaders. Now, I think it's just like mandatory to be like that. Would you agree like the people you think about who are like great? I think it's both becoming mandatory or a sign of, yeah, this is a strong leader for engineering. It's also becoming much more explicit.

8:46It's not just a soft thing that you need to have. You need to be talking about very specific items and interact with the rest of the business in specific ways. because if you're not going to be predictable and empower the go-to-market side of the house, then there's a built-in delay between when features and capabilities are ready and when the field can actually sell them, for example, or support them. So it's becoming much more structured, I think. Yeah, you're right. I mean, and that, I think, leads us to the data and the goals. So now that it is mandatory and it needs to be explicit, how can engineering leaders set goals that help them achieve both sides of this mandate?

9:38You know, we're now taking this to a more tangible step of, okay, I need to set goals. Like in every business unit, I'm not going to measure everything or goal around everything. It's too much. You need to apply some focus. Realizing and understanding the dual mandate means that, okay, I need to think about at least the two sides. I need goals around my operational excellence, about our productivity, but I also have to have some goals around the business alignment and predictability. The first actionable step is make sure you have a goal on each side of that dual mandate. A goal on the business alignment part can be around your predictability?

10:24Am I being able to deliver on promises or be predictable about delivery in a consistent way? It could be about are we spending enough on building new value? Are we able to focus enough of our efforts in building and creating new value and not just chasing, putting out fires, just keeping the lights on? Do we have enough of our spend on innovation? Or it could be about the key projects, the key initiatives that the business really cares about, and making sure that we are investing enough in these. Not just in our plan, but also in actuals. If I needed 20 people on that project, am I actually getting 20 people on that project?

11:13These are the kinds of goals we can take for the business alignment side. And then on operational excellence, there's things like cycle time, like understanding our developer experience, our quality. There's a bunch of goals and metrics depending on where you are and where what's really important for you as an organization. I think these things are a little easier to measure because they've been around longer. There's a little more practice. And even on intuition, engineering leaders tend to gravitate to those areas. So have one on each as a good starting point. Yeah, I think what we want to do, of course, what I want to do on this pod, we have a lot of pods in Dev Interrupted that are about the Dora metrics, cycle time, change failure rate, deployment frequency, MTTR.

12:04Then you have all your leading indicators, PR size, rework rate, review depth. I'm going to put it aside for a second. Let's talk about where the industry is going. and you said the one that caught my ear the most was predictability. When I talk to engineering leaders and what I know from my past career, the question that's always asked on the engineering leader is like, when is the thing going to be delivered to me? When do I get my value? When do I get to close a big deal because you have this new feature that came out? When do I get to retain a customer? So if you don't mind, let's focus in on predictability.

12:45And one of the KPIs that we talk about at Linear B would be the planning accuracy. It sounds cool, and I think it's a great KPI. Maybe you can start there, but what are the other things that come to mind for you when you think delivering projects on time, predictability? Yeah. So I'll start by saying you talk with engineering leaders and with business leaders, the CEOs and, you know, the peers around the table, like the marketing leaders, the sales leaders. And as you look into more larger companies, larger businesses, forward thinkers, they're starting to trade volume or velocity for predictability, right?

13:31I can take a little less production, but at a better predictability. Why? Because software is going agile. It's all about trying to move fast, even break things, tight learning cycles. But then there's the whole surrounding part of the business that needs that predictability. You're not going to be able to pull up a marketing campaign in a day just because the release, the agile software team just delivered. You have to prepare. You have to line up things. Not everything in the world can be as agile as software delivery. So predictability becomes more and more important as you grow as a business, as you have more customers, more varied set of customers.

14:19and with that in mind, looking at the specific metrics and the specific ways you can measure and goal around predictability becomes really important. So planning accuracy, that's a good place to start. This is about understanding how we deliver sprint over sprint. Is our planning and delivery, are they aligned? Are we able to deliver what we promise first to ourselves, then to our surroundings? on a sprint-by-strint basis. If you're able to master that, you're on a path to being able to predictably deliver a project, which is like multi-sprint, maybe a quarter, a bigger chunk of work. And then you have, when you look at, you know, you zoom out, you look at a project.

15:03Now you have, you can talk about how much of the project scope did I hit? Is it getting in on time? And it's not just about golding and reporting. you can also make some decisions before the project ends to optimize what you're landing with. You're not always going to deliver 100 % of everything. We know that. There is a time when you look at the data and you say, okay, I need to change something. Maybe give up on these scopes, some items, focus my energy on the ones that are almost there. So understanding the burnup, if you like, for a project or for your whole delivery on a quarter or similar planning cycle, making adjustments, but also tracking quarter over quarter.

15:49Are we delivering enough of what we promised, enough of what we planned in good quality? That would be a great goal to track and improve. Yeah. I mean, you talked about kind of the maturity, I would call it, of the space, maybe moving from, I always have like the CEO in mind because I have this like CEO archetype of like, just give me all the stuff, like do it fast. Yeah, yesterday. I needed it yesterday. A little more maturity of like, okay, I value predictability because I understand my marketing team can be in sync. My enablement team for sales can be in sync. My enablement team for CS. The way that we go out with this product matters to me or this feature matters to me.

16:34Maybe you have like a Gartner interview coming up. the predictability matters a lot for the whole business to be in sync. And we have this in our Linear B product. It's like, I remember the CEO used to ask me, okay, is the project going to deliver on time? And I would say, yeah. Because it's like, what else can you say? And then it's like, why would you want to say no? Then it's like, you have no confidence. Like, yeah, it's going to deliver on time. But then it's like, yeah, but you said that last time with the last project and it was like three months late. And it's like, oh, yeah, sorry about that.

17:12That's like a very like qualitative conversation. Now, here's the new conversation. You pull up, you know, our feature is called forecasting. But you pull up your project forecasting module and you say, yeah, I do think it will be delivered on time. There's some risk. I want to talk to you about it. Click into the project. Yeah, it also moves from yes, no to the project is not a monolith. So there's a bunch of things. These are golden. These are in risk. It's a yes, but. Or we have decisions now. I think it's like, yes, but, and let me show you. Let me bring you in. You know what? The planning accuracy for this project looks good, let's say.

17:58It means that the teams are able to deliver on what they say they're going to deliver on. I'm doing kind of a conversation. Let's pretend you're my boss or the CTO. I'm an engineering leader. The planning accuracy looks good here. So that's a good thing. It's giving me confidence. But there's something that is concerning me on this project. And I want to show you something. It's our cycle time on this project. I'm seeing bottlenecks here. And I'm seeing some bottlenecks because it looks like only one person is doing all of our reviews. It's a senior. And we have a lot of juniors on this project.

18:32Now, I know that this project's due in six months from now, but I want to call out this people risk to you. It's a very mature conversation. I'd like to add another senior to this. And the reason I want to add a senior is to increase my probability to deliver on time. And even better now, I know this is like futuristic stuff, but this is what we're working on with some of our customers. Let me show you a Monte Carlo simulation that if we were to do this, why I think it will help us deliver on time. Now I'm talking with data. I'm talking with risk levels. I'm talking with simulation to the business of like, let me bring you in of what it really takes to forecast.

19:12I think that's really cool and kind of like next level mature conversation. What do you think about that? Yeah, I really like the ability to take this from, is this on time, yes, no, or why isn't it on time, right? That's the common question. Why are you guys late? You know, using data, we can move the conversation to, let's make some decisions. It's probably not just magically create a new resource, but let's move someone from a project that is doing well and maybe has some capacity that they can contribute here. because I have a bottleneck here. The data is showing me the kind of risk that I can alleviate with moving a resource and maybe not lose so much on the other project because the signals there are different.

20:01When the conversation moves to making choices and you see the choice, you see the outcomes, it becomes a much stronger conversation. Okay, so let's do a little summary. on the predictability side we have some kpis right we have planning accuracy that's a good one we have capacity accuracy that's a good one we have project forecasting with simulations that's a good one and now imagine having that for all of your projects you see them all in one view that's awesome now there's kind of this like what you're saying i have to make decisions of what what to do Can you talk to us a little bit about goals around resource allocation?

20:47Because sometimes a lot of those decisions might be, ah, I don't know. We just were understaffed on this project. Is it really the top project? I could move this one faster if we could delay this other one, but I got to shift some. Talk to me about resource allocation, investment profile, goals around that kind of stuff. So both resource allocation and investment profile are ways to look at the actual spin. Where is my energy going? Where are my people deployed? Not just on paper, on plan, but what is actually happening? And I think a very common cycle, and we've all seen this where management sits at the beginning of the quarter, lays out a plan, allocates resources.

21:31Yeah, let's take 20 people for this project. Let's put this team on that project. This is important for us. It's a new product or a new feature. let's concentrate some work and resources there. Great, the plan looks good. Then reality hits, right? So this 20-people project, yeah, on paper, they're all there. But three you haven't even hired yet. Two are busy with an old project that they weren't able to ram down quickly enough. Another one is pulled to put out fires elsewhere. Reality is different. and then the quarter ends and why have we missed? You spend time to realize that actually we didn't have the 20 people.

22:15So resource allocation is a way to look at what is the actual spend across projects or initiatives or any kind of slicing the pieces of work in your dev org and course correcting. Yeah, this is what is actually happening. This is what we wanted. We can now course correct. We can help people ramp up more quickly or ramp down more quickly if a project needs that. Or we realize that we have a gap. Now we can make a choice. Okay. Reality is a bit different. Let's make a choice about what to give up on, what to invest more in, and so on. Totally makes sense. And now we can have kind of that educated conversation with the business.

22:59Okay, so now that we talked about kind of both sides of the dual mandate, some of the KPIs that you can use for engineering efficiency, we have our DORA metrics, we have some leading indicators. We talked a lot on the business and alignment side, predictability, very, very important. I want to ask you in terms of actually setting goals against these KPIs, where in the org structure should I start? Is it like a top-down thing, a bottom-up thing? What are your thoughts on that? That's a good one. I think there's probably not one answer. First of all, there is the org culture that you have to take into account.

23:42In some respects, and I think mostly on the engineering, like operational efficiency, there's a lot of sense in doing this bottoms-up and letting the teams decide on which goals to take and how to tune the goals. because in many modern development organizations, you want to give autonomy, and you have the right people that you can give autonomy. The team knows how it's delivering. They have maybe even between the teams, there's a difference in how they're working. Some could be using Scrum. Some could be using Kanban. Some could be using a combination. Some are building SaaS front-end. Some are doing mobile apps.

24:25So very different ways to measure and to benchmark. So giving autonomy and the bottoms up approach makes a lot of sense. And also very strong culture, positive of, and it's not like a big brother watching over your back and measuring you. It's more like this is a way for you to measure yourselves and report up. And it still makes sense in these cases to have a top-down focus. Yeah. We know as a company, as a dev org, we want to focus on being more efficient. Let's make sure we look at cycle time top down because it's a good efficiency and bottleneck discovery kind of metric. If we have a quality issue or a quality initiative going on in the company, we can make that a top down and bake that together with bottoms up.

25:18On the business alignment side, I think it's typically running top-down with org level or VP level and then departments or groups aligning either to the product lines or the way the org is built. You want predictability to be measured across your teams or groups and then aggregated up. That makes a lot of sense coming top-down. So it's kind of like a mix. One thing that I've seen, especially with the larger organizations, let's say you have 100 developers plus, 250 developers, 500 developers, and then I don't know, I'm working with companies that have 6 ,000 developers. I think that there does need to be something top down around standardization, I'll call it.

26:06Now, of course, you want bottom-up adoption, but when I think of standardization, it's kind of like, here is how we're going to measure cycle time. This is our standard practice of when cycle time starts versus when cycle time ends, and that's the standard for our business. Here is how we're going to measure sprint planning accuracy. This is how we started either one day into the sprint or right when the sprint. So what I've seen is when you bring these top-down standards, it kind of says, when do things start and stop? And what are the KPIs that we care about? Now, after that, we care. Yeah, we care.

26:49Now you have the standards. We do care. I think it's more go deploy this with your teams and set your goals around this. Here's our high-level goals. You go make it happen. That's where I've seen it works really well. You have top-down standards. You say the KPIs that the company or the business unit cares about, and you give some high-level guidance on the goals. Now you go make it happen for your teams. That's kind of what I've seen work really well. And I think that lets the teams or the lower parts of the org focus a little on the leading indicators, sometimes on more technical ways to look at the problem.

27:30So you're thinking about cycle time as a top-down, but then for a team to improve their cycle time, they could be looking at smaller PRs. They could be looking at how long does it take me to pick up a PR for reviews, a pickup time. The ways to implement that goal are a little more detailed. And another team may look at different, they have different bottlenecks. So they're going to take a goal around a better review process. Or can I automate away some of my reviews? Absolutely. Now, what I want to do in the last few minutes of this pod is talk a little bit more about the business stakeholders and what it takes for a VP of engineering, a CTO, a director of a business unit, maybe even you're communicating you know to the board we have you know from linear b board slides that you can use to communicate you know these kpis but what can you tell us maybe an overview or some tips and tricks i'm an engineering leader i'm measuring this kind of stuff how do i approach business stakeholders i've never done it before what do i what do i do yeah so um first i you You know, a lot of us hit that wall where it feels like no one cares at the beginning.

28:57You know, everyone is about the sales numbers and the marketing numbers. They're used to seeing numbers and data and measurements for those departments. And they're used to having engineering say, I think this feature is going to land in two months. That's so there is some education. Oh, there's actually good data to look at. There's actually a way to drill in. so come prepared to not just show the data and the goals but also explain why it matters why cycle time matters so much because it lets me you know learn much faster from the feedback that I'm getting from our customers when our new the new code hits the users by reducing these bottlenecks I can get so much more done and improve the quality So be prepared to explain why it matters.

29:50I think as engineering leaders, and many of us are engineers at heart, we need to remember that less is more. So don't come with 50 numbers. It's not going to work for the business discussion. Focus on what really matters. And always show how that aligns with the business. So if you're talking about predictability, yeah, this project is landing on time or 90 % of it is landing on time. And here's why. And here's the choices that I've made. I think that was when you start overcoming those glazed eyes of some numbers on something that I don't understand. You know you're seeing success when people are asking you to drill down.