Reimagining DORA Metrics & Leveraging Feature Flags | Split's Ariel Perez

0:00The reality is actually that about 70 % of everything you put in production has either little or negative value. 70%. Imagine that. That's crazy. 70 % of everything you ship has little, no, or negative value. So only 30 % of what you're doing actually has value for the customer. It's unbelievable, right? So how do you think about that idea of value? So the Dora metrics and even the reimagined world of Dora metrics with feature flags and that granularity help you ship faster. at Devinterrupted, we work to give engineering leaders actionable ways to improve their teams. That's why we're producing a three-part summer workshop series with Linear B.

0:47Each of the three workshops will explore the processes that elite software engineering organizations and executives use to deliver better business outcomes and reduce cycle time by 47 % on average in just 120 days. You'll learn how to assess your current performance and benchmark it against industry averages, streamline processes through automation, and improve business outcomes through resource allocation. Learn from the best and take your team to the next level. Visit LinearBee.io slash events to learn more and secure your spot today. Hey, what's up, everyone? Welcome to Dev Interrupted. This is your host, Dan Lines, LinearBee co-founder and COO.

1:26And today, we're joined by Ariel Perez, a VP of engineering at Split. Ariel, welcome to the show. Thank you for having me, Dan. I'm very happy to be here. Yeah, super awesome to have you on today. Really been looking forward to having this conversation. You're really passionate about engineering effectiveness, which for me is something that I care a lot about. We care a lot about at Linear B. And I know you also have thoughts on DevOps and some Dora metrics, and of course, do the split feature flags and how they play into these topics. But before we jump into our main topics today, can you give the audience a background about yourself and how you got into engineering?

2:17Definitely. Thank you for that. Well, I'm born and raised in New York. Background, I got into engineering, even I think as far back as high school, dating myself a bit. Those were the days where DHTML was a thing. So it got me really into playing with CSS and JavaScript and making things move dynamically on a page and really pulled out my passion for programming. So I went to college for programming. I was a computer science major at that time. Jobs in New York City when it came to technology were primarily in the financial services space. Facebook had just started. Google had only just been around.

2:54So So people didn't really go to Fangs just yet. Amazon was barely new as well. So I worked in financial services. Did that for many years, grew to many roles, individual contributor, managing teams, lead engineer, got to travel around the world quite a bit working with international teams, then decided to change it up and start going, go back to startups, smaller companies. So bounced around a few startups, built my own startup for a while, was consulting, then came back to big company. I was working at JP Morgan Chase on the Chase side. That's what got me really passionate about feature flagging because I got my opportunity to build my own feature flagging platform there for use at scale.

3:33Fast forward a few years, got to take that to the UK and built Chase in the UK. Awesome experience building a digitally native mobile native bank and then landed here at Split, you know, really putting together my ability to understand feature flags, having built my own and a passion, particularly for measurement and learning. that's amazing i mean one one thing that i really like about that background i mean oftentimes we'll talk to people more so like on the west coast and you know that startup vibe and maybe i'm jumping from company to company until like i hopefully get like a big hit but you were saying like in you're in the new york city area so maybe some of those like financial firms and larger organizations what did you learn the difference between a startup and a much larger company?

4:21And the reason that I'm asking you that is now, of course, you're one of the VPs of engineering, right, at Split. And a lot of people want to progress their career to get to that type of opportunity. So any tips on how we can do that based on your background? Definitely. I think, you know, we can speak about broad, you know, in broad strokes, but in general, one of the key differences I saw in my first move from big company financial services to small startup. There's only you, meaning there aren't more people coming, there aren't more people available. So you tend to do everything. You become a jack of all trades, which is can be challenging at first, maybe daunting, but also you will learn faster than you've ever learned before versus generally being in a larger company where your role is a lot more clearly defined.

5:09You're in your particular area and they have experts in different areas, lots of specialization. So that's the first big thing. I think the second big thing that I tend to see is bigger places tend to move a lot slower. Decision processes take longer. Approvals take longer. Getting projects defined, they might not be as nimble or agile in how they define projects and get them developed. While in a startup is cool, can you ship it tomorrow? And you figure out every possible way to ship it tomorrow. A lot more ability to move fast and remove a lot of processes while also maintaining a lot of rigor in your decision making.

5:43And if I were to say one last thing is that large companies really, really care about efficiency, productivity and efficiency are the things that really stand out to them. So they'll start thinking about how to invest in tools that help make engineers much more efficient because they're so effective. They really care about the bottom line. Startups maybe don't care as much, or at least in this climate now, they definitely do, right? This climate of not growth at all costs, but very paced growth and measured growth. Efficiency and cost is becoming a lot more important for smaller companies as well.

6:18Yeah, that's actually the perfect lead-in to our first topic. And thank you for sharing some of those insights in your background. And I couldn't agree more, especially with those, you know, if you're scaling, let's say your engineering team, you're growing your engineering team or you're working at a large organization. Certainly the business is going to ask of you, especially in this economic climate, what are you doing for efficiency? Where could we cut costs? How do we make people more productive? Because the more engineers that you have on the team, that efficiency can multiply by 100x, 500x, 1000x.

6:59So yeah, I think that's a wonderful point. Some of the things that we were chatting about in like our pre-production meeting and before we got on here, you said something like teams are often pursuing productivity and efficiency at the expense of effectiveness. Can you unpack kind of what you meant by that or that type of mindset? Absolutely. And actually it ties very closely to what you just said where at a bigger company, you might say, let's hire more people, add more engineers, get more value out of that. There's a big aspect of large companies, especially, but also it's bled into every other company as we think about software development.

7:41One thing that has ruled the day for the last, I don't know, almost 100 years is this idea of Taylorism. Taylorism came out of factory work, right? How do we get very efficient at getting the most value, squeezing out the most value out of every single part of this factory that is generating things, building things. You know, that worked in a world where everybody was building widgets, right? And every widget is exactly the same. I cast the die once and I ship the exact widget over and over. Yeah, this is manufacturing, right? Exactly. Manufacturing. I need to manufacture a widget for a billion people.

8:18What do I do? Right. Exactly. So a lot of those ideas came into management space and how we think about managers and how we think about people and people or resources. So how does that translate to the engineering world where, well, I need more engineers that ideally at a lower cost and I need every engineer working individually so I can maximize how much I get out of each engineer because they're an expensive resource. Now, the key thing that that misses. So, yeah, you might be very efficient doing that. Might be. And we can talk about it. Your product is great at highlighting where you're not efficient, but you're definitely going to be productive.

8:55We're going to be busy. Let's keep them busy. Let's keep them maximally utilized. Now, some challenges that occur when you do that, right? You know, and any people working in engineering, you'll know what happens when your machine and your CPU is at max utilization. You know what happens when your RAM is at max utilization. Things fall apart very fast. So the first key thing is software engineering and in general, knowledge work is not like manufacturing work. You cannot reproduce the same exact widget over and over again, you get the same people, the same problem, the same time frame to try to build the same thing.

9:32And you're not going to get the same result because the context is different. It's always different. Software engineering is complex. That's the key piece that we fail to understand. Manufacturing is simple, might be complicated, but software engineering is complex. And what complex means is that you don't know how the parts work together and how they interact with one another. And you change one thing, you don't know what's going to happen. You add one thing, you don't know what's going to happen. You can try to guess, but you will not know. So with that mindset, then saying, let me maximize utilization for every engineer and let me try to reproduce the exact same thing doesn't work.

10:05What kinds of things do you run into when you try to do that? Well, let's maximize work in progress, right? Five engineers take five stories in a sprint, assuming scrum, and they all work individually. They're maximally utilized. But then what happens? Engineer one opens a PR. Now engineer one has to wait for engineer two or three or four or five to go review that PR. So they sit and wait. What do you do when you sit and wait? You go start a new task, right? Because you want to be productive. You want to be utilized. The other engineers have to stop what they're doing to go review that PR. So now I stop.

10:37I go review a PR. In the best case, PR closes and we ship. Most cases, you go back and forth a few times. So imagine how many times we're interrupting each other and we're waiting for each other. So there's a lot of wait, wait time in there when you have max things in progress. So maybe reduce work in progress. Yeah. Just a comment there because honestly, you're, you're, you're hitting on a problem that's like near and dear to me. So I have to make a comment here before you go. I mean, the other thing about the PR and the situation that you're talking about is every pull request is different. It's not like you're getting the same change set every single time, like in a factory, for example, a thing that's being passed from machine A to machine B is exactly the same.

11:23Every PR is completely different. The risk of the PR, the size of the PR, who needs to review it, what does it impact? And that's why I put a bunch of my passion into this GitStream workflow automation tool that I've been working on because of that exact problem. And that's where I think probably where you would say there's a science to engineering, like to software development, and there's an art. And the art is identifying, okay, everything is actually different. What can we do for each one of these different sets? So just had the comment on there. Back to you, to where you are going. Yeah. Thank you.

12:08And again, actually, I'll continue on that just for one moment. Talk about GitStream, right? The things that GitStream is aiming to do, trying to find, define how simple, how complicated or how complex this particular PR might be and help you optimize that. Right. Now, here's the thing I'll say is a drawback, not of GitStream, but this whole PR based flow. It still requires generally asynchronous review of code. One engineer works on this code, a separate engineer when I'm done later reviews my code and we'll go back and forth. So what do many tools try to do? to optimize that and make it more efficient?

12:44Well, give you rules, make the PR smaller, reduce the scope. There's all these things about make these things smaller so that that review and that asynchronous out of context review is easier, which is great. Let's improve that because you know what? 99 % of teams, this is what they're doing. So let's help them do that. But then that gets me into the second thing, beyond work in progress. It's actually a lot more effective, and I'll talk about why effective, to have engineers working together. So what approach might be, well, can two engineers in real time review the PR together? That's often much more effective than I open my PR, I walk away, you come review my PR and you have asynchronous communication.

13:25Already with the asynchronous communication, there's wait time, we miss each other. We might not catch what we're saying. If we just sit and talk through this thing together, so much more effective. You close that PR a lot faster. But let's take that even a step further. Why have PRs at all? Now, that might sound like blasphemy to some teams, but how can I potentially remove PRs while keeping the things that we want from PRs? Quality control, sharing best practices, learning. Well, what if two engineers work on a story together? That sounds like a radical idea to the Taylorists because no, no, hold on.

13:56I have two engineers. I pay them a lot. If they're both working on one story, I'm getting one story out of them. What they're thinking about is the very short term transactional costs of two engineers working on one story. What they're not seeing is that software engineering, you pay the cost at the beginning, and then you have maintenance costs for the rest of the life of that software. That software will be changed by someone else again. And if you have two engineers working on it, the likelihood that you will reduce the lifetime cost of that code drastically goes down. Because here's, I think, something great I've seen in your product.

14:29you can, you know, the rework rate, you'll probably find that that rework rate drastically reduces when you have more people working on that story at once. So you might pair, you might even ensemble or mob program on that thing, three to five engineers. Oh my God, super blasphemous, right? Five engineers working on one story, that cannot be efficient. That cannot be productive. That's right, it's not, but it's effective. When they ship that story, odds are they'll never touch it again from a bug perspective, from an issue perspective. So when it's done, it's shipped, It's out onto the next thing.

14:59No PR needed, merge street to trunk because you have that level of quality and teaching across the board. You know what? It's kind of like you're proposing a different mindset, I think, than most of, I would say the end. You know what they reminded me of? I think something like either pair programming or extreme programming. There were some of these movements early on in Agile to say, hey, let's get some of these engineers working together. Let's put the cost more upfront instead of later on in the life cycle. Do you have any, and it sounds great, right? It honestly sounds great, but at a large, I don't know, an engineering organization that's not used to doing that, right?

15:47I'm not used to doing that. I'm working at, like you say, in New York City, like a large, I don't know, financial company or something like that. And by the way, I don't know if you do or you don't, but like getting to a situation to getting more of that upfront cost earlier, you know, earlier on, have you dealt with that at all? Or like, is it a transformation? Is it a mindset shift? All of the above. It's not easy, right? I think so. There are a few things that we've done in this industry and engineering on top of the fact that most management in any company is also Taylorist. Engineers, you know, We tend to stick to this trope that the engineer is that person that likes sitting by themselves and doesn't like talking to people.

16:29I work as an individual contributor. You've got this trope of the 10x engineer who comes in and builds everything by themselves. And like, I'm amazing. And I saved the company. A lot of these things push people to a culture of heroes and individual heroes of people who don't like working with one another. And we kind of self-select and hire those kind of people. So that's one challenge you run into. Just your teams. I am an engineer because I like working by myself, right? That's a trope and you self-select for that. So that's one challenge to work through, how to get you to work together, different ways to work together.

17:02You can define working together and truly collaborating as opposed to cooperating. That's one thing you have to contend with. You definitely have to contend with the management of fighting that idea that, hey, five engineers working on one thing is better than five engineers working on five things. That's a hard one to sell. Five things sounds better to me. I'm a CEO. Exactly. I'm doing five things. Now you're telling me to do one thing. I'm going to do one thing. Exactly. With all these expensive engineers, right? It's this thing about, you know, this fallacy that seeing progress is better than seeing completion.

17:34I got things shipped as opposed to I'm doing things. I'm busy versus I'm actually having impact. So in general, for anything like this, whether it's to management, whether it's to the engineers, the whole organization, the things that really help is this is an experiment. We're going to try something out. Let's test an idea. Just like any team that has retros, the idea is the way you improve is continuously. You introduce new ideas as something to try out. And what's the key thing about something to try out is you define key success criteria. What do I want to see out of this at the end of it to say if it's successful?

18:05Or we want to continue it. And when's the stopping date? Everybody, generally, everybody can do something that they don't like. If they know, they can see an end in sight. So if you couch it as an experiment, let us try this out. And then what are the evaluation criteria to continue? And you make those experiments small, you find that you can start making those changes. But these are radical changes to both the organization and to engineer. So you introduce them in small steps and keep trying iterating on different things. While you were talking there, there's a few things that came to mind to me if you're someone that, you know, is kind of relating to these concepts and want to try it out.

18:42And I'm thinking about the business because one of the things like all of us engineering leaders kind of like deal with is the business. These are very, very smart people, non-technical people. Oftentimes, you know, you're talking to CEO or you're talking to like head of sales and you're talking about new value and that type of stuff. And you did mention that those types of people usually think of their company in like resources or like the allocation of resources. So a few tips that I could give the audience here or what came to mind to me when I'm thinking about resource allocation. One, that's one thing that you as like an engineering leader should be having a conversation with, like with your CEO.

19:28So how are our people allocated? And a common mistake that I'm seeing within the industry is once you have that allocation report, to your point, Ariel, oftentimes I see, okay, we're working on like 10 different projects in parallel with a very small amount of engineers. And therefore, each project is kind of progressing at a very slow or a non, I guess we would say effective pace. and so what you can do is show, okay, we're really spread thin. Let's make, I propose a change here. What are the top three? And I'd like to allocate more engineers to those top three and take one of them and introduce the concept that you're talking about as an experiment because hey, business, this is a project that is most important to you.

20:25I want to deliver it as effectively as possible. And the other thing that came to mind for me on the allocation side is what most engineering organizations are doing today is thinking of their allocation is like, okay, there's keeping the lights on, there's new value delivery, there's enhancements to the thing that I already released. And then there's like internal productivity, like working on your dev. work. If you see that your enhancement percentage is very, very high, which means, hey, we thought that we released this thing or we did release this thing and we're still actually enhancing it, working on it like six months later, a year later.

21:11That's also, I think, an argument for your case of, hey, we're now paying this long cost, this long term cost instead of like doing it more upfront. So a few tips. I don't know if any of those resonate with you. Oh, well, absolutely they do. And I think the first one, you know, one thing I found that becomes very easy that everyone understands, forget for a second, speaking about 10 projects, 10 engineers, the simplest layman's term version of this that even my grandmother can understand is like, what happens when I need 10 houses? I can start building 10 houses all at once right now. What happens to my risk?

21:48They'll all progress. And if I run out of money in a year, I might have 10 incomplete houses. Instead, maybe I focus on building three houses right now. And in a few months, I'll have three houses. And in a few months, I'll have maybe six houses. I run out of money at the end of the year. I'll have six full houses instead of 10 half built houses. It's the same reality in any project allocation is work in batches, work on smaller things, get more people focused on them, get them done thoroughly and effectively. So same way to explain that same concept. oftentimes the way to present like an allocation of resources is by like, yeah, and you're seeing that enhancement be very, very high.

22:29It means that you're paying a long-term cost instead of the upfront costs that you were talking. Absolutely. And, and the way I think about that, there's two things. One, you know, as an engineering leader, we're pushed to talk about in these terms, how much you're allocating to here, to here, to there. A lot of that often comes from finance, right? You know, things that engineering leaders learn is about, you know, amortization and cost and the kind of, you know, we talk to your accounting and finance people, talk about taxes. You break it down that way. What's lights on versus what's innovation.

23:01But in reality, when, if you start thinking about how you organize your team is get them together to solve the most important problems. And yes, you talk about the long-term maintenance is what do you want to invest upfront? Do you want to focus on quality and value or do you want to work, focus on efficiency and cost? Focusing on efficiency and cost too much early on guarantees that your costs go up over time, guarantees it. If you focus on efficiency and cost, because you're going to get the cheapest resources, spend the minimum amount of time possible rather than spend the time to get the quality and value and reduce your cost of change for the rest of the life of that software.

23:36If you invest upfront to reduce the cost of change, every incremental change, and it will come, will be much, much cheaper over the long run. Yeah. It's almost like there's like a education process here. Like if you are a VP of engineering and usually you need to like justify this way of working again to a CEO or to your senior leadership team a little bit, because when you did the house example, it was amazing. That made sense. But if I just come to you and say, and you know, I'm reporting to you, Ariel and you're the CEO. Yeah, instead of doing 10 projects, I'm only going to do two, three. That doesn't sound good, right?

24:17So there's also kind of like how you present it, the approach, come with some of that data. I think it helps a lot. I'm excited to transition us because I see our second topic has a really interesting title. It's titled Reimagining Dora Metrics and Leveraging Feature Flags. What does that mean to you? I think a lot of our audience knows about the Dora Metrics, right? So we have cycle time, we have deployment frequency, we have change failure rate, we have MTTR, mean time to restore. What does it mean to like reimagine these Dorometrics. Got it. So I have two versions of that. One is the more straightforward one.

25:04One is the more radical one. I'll leave the radical one for later when we get to it. Right. That'll be, I think, I like the radical. We got to get to the radical. Right. But the first one is this idea, right? You know, Dorometrics came in a world where we were just trying to ship faster. Right. And in a world where deployment and release was the exact same thing in order for you to ship a change to a customer, it was a release and a deployment. Same exact thing. I had to put the software in production. And the world has changed a lot since the Dora metrics were published initially, right? And again, Dora metrics are amazing at trying to figure out which teams are very efficient at shipping software.

25:44Shipping software that doesn't break. Shipping software that's high quality. How good am I as an engineer organization at building stuff and shipping it? But the key thing is what happened with the world changing is over the last 10 to 15 years, but even more, the industry really even less so over the last 10 years, feature flags as a term, as a concept for building my applications, right? And not assuming that everyone knows exactly what a feature flag is, although it's a lot more ubiquitous now. It's a toggle, it's a switch, it's a feature flag. It's basically at its core an if statement around your code.