Wiring The Winning Organization pt. 2 | Gene Kim

0:00Are you an engineering leader and tired of constantly being asked when will it be ready? Stay one step ahead with Linear B's project delivery feature. Our powerful dashboard lets you visualize key milestones, forecast delivery accurately, and align stakeholders effortlessly. With Linear B's product delivery, you can confidently showcase your planning, prioritize work effectively, and even make data-driven cases for additional headcount. Say goodbye to delays and missed milestones. Sign up for Linear B today and answer, when will it be ready before anyone even asks? Welcome back to Dev Interrupted.

0:37I'm your host, Connor Bronson. I'm delighted to be joined by the esteemed Gene Kim. You may have heard of him, author of The Phoenix Project, The DevOps Handbook, a new book, Wiring the Winning Organization. I just spoke with his co-author, Dr. Steve Spear, and I'm very excited to talk to Gene as well. And then he's also, of course, the program chair for IT Revolutions, DevOps Enterprise Summit, founder himself. Gene, welcome back to the podcast. Oh, so good seeing you again. And by the way, I'm dying to ask, what was it like hanging out with Steve for that interview? I love Steve. He is such a gem.

1:06I have to say he's got me kind of convinced on bow ties now. So it was fascinating learning about your research. It was great to just kind of chat with him. He's a really cool dude. I can see why the two of you wrote a great book together. Yeah, it was so cool because, you know, the question we set out to ask was, well, let me just set the stage a little bit. You know, so I spent, I guess, 23 years studying high performing technology organizations. And so we know that the organization have the best project duty performance and development. They have the best operational reliability and stability.

1:35They have the best posture of security and compliance. And, you know, the state of DevOps research we identified has something to do with their technical practices, their architectural practices and cultural norms. And Stephen Spear, he spent 30 years studying, you know, the Toyota production system, engine design at Patton Whitney, you know, the safety culture at Alcoa. Submarines, incredible career. Exactly. So it was so wonderful that when I met him 10 years ago, you know, this quest that started to emerge was, you know, what is in common between the Toyota production system and safety culture and DevOps and Agile and concepts like psychological safety?

2:12And it was exhilarating to see that, you know, they are potentially all incomplete expressions of a far greater whole. And, you know, I have to tell you, it is it was one of the most challenging thing I've ever worked on. Really? One point in the book where, you know, I almost quit. Should I tell you what? Please, yeah. That's a great story. So it was about a year ago, and we're just, I thought we were at a point where we were very stuck. We had a theory that we knew it had to be three things, slowification, simplification, and amplification. But we couldn't come up with a simple model to really demonstrate all three principles.

2:49It felt too hard to boil down. Yeah, exactly. And so I told my wife, I'm going on a walk and I'm not coming back until we have something. And so six miles later, I'm pretty convinced I'm not smart enough that I don't understand software development enough. And I try to come up with these much simpler scenarios like movie theater operations, but realize I don't understand movie theaters well enough for that. How about restaurant operations? Like, I don't understand restaurants well enough. and uh yeah it was a very uh it was a kind of a one can manage a sort of distressing moment where you started something three years ago and you might not be able to finish it because you can't explain it you don't understand it and um uh luckily you know i think that that's when the notion of uh using this vignette uh that we had written you know a year before around you know Steve and Gene moving a couch together.

3:46Right. To be expended to, oh, Steve and Gene, you know, being asked to, you know, help their spouses renovate an old hotel. Right. So they have to do, you know, remove the furniture from room, paint the room and bring the furniture back in. Right. And it just, it was just so exciting because I was like, oh yeah, we can, we can use that to show how even having just two functional specialties. Right. Doing, you know, not the most complex activity can be thoroughly screwed up. if you organize them wrong. And so it was just even more exciting when in a 30-minute call with Steve, he instantly got it.

4:20And so I was like, ah, that's a wonderful moment. Yeah, so it's been such a fun adventure. And my genuine hope is that this will not just resonate with technology leaders, that they'll find something instantly familiar, right? This is a recast, like, everything that we've learned over the past many decades about, like, how to do great software well. But it'll also resonate with manufacturing leaders, you know, people doing bio research. And so I find that something, I find that to be just a wonderful hope that, you know, that's one of the common language to talk about, you know, how we talk about systems and leading systems.

4:52Steve talked about this in a very similar way. He's felt like it was a kind of culmination of this, you know, decades of work the two of you have done. So it's very cool to hear you kind of view it similarly of like, I've had a lot of the pieces and now I have kind of this grand unifying theory of how to build high performing teams. Totally. And I love this quote. you know, anyone can make the simple, be complex. You know, it's a lot harder to take something complex and really prove to yourself that it is something simple. And so maybe I can, instead of talking in the abstract, how about how we talk about in the concrete?

5:23Let's do it. Yeah, there's like, essentially it's saying there's three mechanisms to create great performance. And you cannot, you know, create great performance without, you know, all three. Well, it's much harder at least. Yeah, exactly, exactly. And so the first is slowification. and, you know, that word we made up because there's no word in English that we were able to find after asking GPT-4, like, you know, 100 candidates. There's no word to sort of capture this concept of like slow down to speed up or, you know, stop sawing to sharpen the saw or, you know, slow is smooth, smooth is fast.

5:56I guess the Germans do. Of course the Germans have a word for it. It didn't fit on the cover. Yeah. So, you know, in our world, you know, we know this as like, you know, Amazon game days or like, you know, or chaos monkeys, you know, to perform brilliant under in production, to be able to handle that with resilience and ease actually takes a lot of planning and preparation. And so the classic stories in April, 2011, you know, the first AWS East outage where, you know, essentially AWS's largest availability zone goes down. That's where they put all the customers, you know, most of the big customers go down except for Netflix, which is very confusing because Netflix was running entirely in the Amazon cloud.

6:37how did they not go down? And so, you know, that led to the famous Netflix blog post, you know, that described how they did it. And essentially, there's really two design decisions they revealed. One was, you know, as they migrated out of data centers in 2009, they could have no single point of failures. And their largest single point of failure risk was AWS. I think they even say, you know, they will never be there when we need the most. And so that led to the development of Chaos Monkey, which randomly kills computances in the cloud, right? And it kills services in the middle of the day. And so developers got very good at fixing those issues.

7:15And so no wonder that when that availability zone went down, Netflix services still ran. So that's an example of slowification. You don't want to, if you're doing all your learning in production environments that have high consequence that you can't undo actions, it's impossible to learn. So you have to do it in the more safer environments of, you know, planning, preparation. So simplification, so that first one really makes problems easier to solve. The simplification is all about how to make the problems itself simpler so that we can solve them easier. And same concept of mathematics, right?

7:53Where it's like, okay, this fraction is large, let's break it down so we can make this a little easier for people to break it down. Oh, totally right. Let's separate like terms and there's all these things we can do to sort of get our heads around the problem. That's a great example. I've never heard that example. That's exactly - I can be in the next book. Oh, please. Just throw a credit if you don't mind. Yeah. And I think what's exciting about that is that in our space, you know, what that should bring to mind is the state of DevOps research finding, where we found one of the top predictions of performance is, you know, to what degree can teams, is architecture.

8:21To what degree can teams work independently of each other without a lot of fine-grained communication and coordination? And not just technical architecture, but people architecture as well. Oh, absolutely. Right. The socio-technical system, without a doubt. And so the classic example of that is in Amazon, they started off with two categories of products, books and music. By 2004, they have 35. How many are we at now? So imagine how difficult it was to get things done. If you were one of those 35 product teams and you had to deal with the product page team, the ordering team, the shopping cart team, the returns team.

9:01right and then you have every team connected together and to get even small things done required a huge amount of coordination yeah and so uh dr verner vogel cto at the time he said in fact i just only learned about this quote a year ago despite reading this paper 10 years ago he said there was this absurd situation where amazon uh digital so i was video and kindle to fulfill an order they had to provide a physical shipping address uh and there was no way around it So, you know, they had to go to, according to this article, right, those teams had to go to 60 different ordering teams and say, could you please give us an alternate, you know, order path?

9:38And they're like, didn't budget for it. You're out of luck, right? So they're stuck. And so that was the real genesis behind, oh, and then the fact that they couldn't deploy code anymore, right? You know, most deployments didn't finish because something went wrong. Bit of a challenge. Yeah, exactly. So, of course, that led to the$1 billion re-architecture of Amazon and those APIs, which separated them into modules that can now work independently. They can independently develop, deploy value to customers. That whole concept of just having APIs run everything within Amazon has really enabled them to scale like they have.

10:13And it's interesting that the first two examples you bring up are both related to Amazon because it shows the depth and the success they've had at building a very high performing organization and scaling it across the world. Yeah, no, absolutely. And, you know, there's two things that they did brilliantly that modularization allows. It allows the reduction of design time coupling so that, you know, those teams can do what they need to do without communicating, coordinating with, you know, the 60, 100 other or now thousands. We have an API for that. Exactly. Exactly. As long as I don't change the API, right?

10:44No, I don't need to coordinate with anybody. And then as you mentioned, right, they reduced or eliminated runtime coupling. I can scale my service without having to scale everybody else as well. So it turns out that's one of the three ways to simplify systems. And there's something equivalent that you do with sequential processes, like the Toyota production system. Each one of them enable independence of action. Dependency reduction. Yeah. Dependency reduction, exactly. And the benefit is that I can do what I need to do without having to get permission from 35 different other people. So creating autonomy within the organization.

11:17Yes. And the ability to then execute, which again, there's other research that's been shown that when someone feels like they can be impactful, they do better work too. Another indicator here. Oh, absolutely. And it allows them to get more faster, more direct feedback on their work. Right. Absolutely. So all those things come from modularity. We're seeing this all add up, these little steps that then like start to compound together. Yeah. I know there's a third dimension as well that you mentioned. Yeah. In fact, you know what? Maybe just you just reminded me. So that linearization, right? The cousin or the orthogonal cousin of modularity is the Toyota production system.

11:52But that's also CICD. When you line up sequence activities like requirements to dev, to test, to deployment, to ops. Oh, let's simplify the friction points all along the way. Absolutely. Let's tie them together. so that we don't do large batches and have to open up tickets. Wouldn't it be great if we could automate, linearize, and then automate those processes so that we can get single piece flow through the systems? Yeah, exactly right. Well, I was just talking to Dr. Andre Martin about his book and how his viewpoint is also like, when we have these situations, we are freeing up cognitive load so people can do their best work.

12:28And I see that same theme in your research. Oh, yeah, absolutely. And I think what's really exciting to me is that Dr. Nicole Forsgren was here. She presented on her research. And what I love is that the concepts in Warned the Winning Organization, they line up right with the state of DevOps research finding the door. Absolutely. Because linearization is all what allows us to get fast deployment lead time because you can't deploy within minutes or multiple deployments a day if every deployment requires threading your work through 3 ,500 different steps across 60 different teams. of which maybe the testing step will take six weeks, right, can't be done, right?

13:08So linearization should also be familiar to technologies. And these are all important, like, sub-concepts of that simplification thing you talked about. Yeah, absolutely. So, yeah, you were mentioning the amplification, the third one. Right. Right, the notion is that in any system, when something goes wrong, you want to be able to generate the signal, transmit it, receive it, and then hopefully have someone react to it, right, and then solve the problem. and this should be familiar to us in technology. It should remind us of production telemetry. You've got to see the problem. Also continuous learning.

13:40Continuous learning. Let's find the problem. Let's keep going on it. Totally. Iterations. Iterations, you're right, because once we see a problem, we want to fix the problem. We want to be able to, in any complex activity, it helps us see what's going on and helps us confirm that what you actually did resolved the issue. In fact, for that matter, it also reminds me of now that you mentioned it, like putting developers on rotation, just like ops people, right? Because you got to share the pain or, you know, you got to - Share the learning. Share the learning, right? Yeah, yeah, yeah. I love it.

14:12So how should I construct my team then? How do I wire that winning organization? Great, I have these concepts. How do I apply them? Oh yeah, that's a great, that's a really great question. And, you know, I think it defies easy explanation, but I think we know when it will work or when it won't. I think we sort of simplify it down enough. So let's say dev and ops. Configure it one way where dev does all their work and hands it over, throws it over the wall to ops. And the only way that ops can talk to dev is if there's a live 7-1 outage and you have a ticket number. And the only way that ops can talk to dev is if there's a project code.

14:54And I feel like, you know, that is just not our winning recipe because the communication channels between Dev and Ops are so few and thin that it's just hopeless to transmit enough data, you know, to get the system to work as a whole. I mean, does that resonate with you? It absolutely does. And there's plenty of research that shows that when there is high degrees of communication and feedback, the quality improves. Yeah. So, I mean, this absolutely resonates. And I think the metaphor that we use for that is to try to educate leaders on that is it's like Steve and Gene moving a couch. And it's just a metaphor for how we do joint problem solving and joint cognition.

15:36And so one might think that moving a couch is all brawn work and there's no brain work. But when Steve and Gene move a couch together, there's actually... How do you get it through the door? Yeah, exactly. Where's the center of gravity? How do you get down the stairs? Who goes first? I was actually telling Steve when I was younger, I had to help a cousin of mine move. And we realized we couldn't get the couch out down the stairs. And we had to actually hoist it off the roof to get out of the building. So I'm like, yeah, there's absolutely some complications that come in here. Oh, absolutely. And as leaders, you know, there's all these things that we can do to make it harder for Steve and Gene to move his couch.

16:10Right. We can put in a ticketing system. We can put in like, we can prevent them from talking directly to each other. We can turn off the lights so they can't see what they're doing. and we can put in a lot of background noise, right? So they can't hear each other. So they can't communicate and coordinate. Okay, so that maybe the background noise would be inefficient systems that don't flow well, to your example earlier about CICD systems and trying to create this linear flow. Oh, totally, totally. And so the reason why the first system doesn't work well, the reason why I think we were able to say, oh, that doesn't work well because Dev and Ops are trying to move a couch together, but they're not actually allowed to talk to each other.

16:44And so, you know, when we do things like co-locating them together, having a liaison of ops into the dev teams, you know, and vice versa. You know, these are now much closer to Steve and Gene moving a couch together. Right. Because there's actually joint problem solving going on. How am I doing? Does that resonate with you? No, that definitely resonates. And it reminds me of an example Steve gave about Toyota and their efforts to ensure that we'll fly as much as possible so that individual workers on a production line can focus their energies on key tasks instead of, you know, worrying about these other contextual pieces that it's like, hey, we've already figured the context.

17:17We've figured out the flow. Let's free you up to do what you do best. And I think it's the same thing, whether you're a writer, whether you're an engineer, it's like, okay, let's make the systems that work for you as easy as possible. You know, some people mention AI here. I'll throw it as a buzzword. Got to do it once in the talk. And then we're going to free you up to focus on what you're most creative at, what you're problem solving, your strategy, these things that give you depth. And as a leader that this release speaks to is like, so much of your focus should be, how do I make it easier for and resonant for everyone around me to do their job, to feel passion and to go deliver.

17:51That is amazing. I double, triple underscore that. What I learned in this book is that the job of the leader is to ensure that everyone can do their work easily and well. Yeah. And you can thoroughly screw that up by not having enough time, slack in the system, because that means you're not slowifying. Every mistake is being made in production when you can't learn. or you've organized a system so that, you know, no one can move a couch together. Yeah. Or maybe to get things done, you have to open up tickets with 35 different teams. That's the opposite of easy and well. Or if you create a system that somehow suppresses or extinguishes entirely important signals, not like weak failure signals, right?

18:33As opposed to amplifying them and acting on them directly. So I found that to be very satisfying. I love the way you phrased that. Yeah, honestly, I take a lot of inspiration just from what I've read of your book so far. And my interview with Steve, this conversation around social circuitry and the idea of how do we program the society we live in and particularly like our organization to work better. And I think that's a it's such an impactful concept. I already told my producers I'm never repeating this for the next year. They're going to get tired of me hearing me say it. It's so difficult to argue against.

19:05And what I'm hoping that people reading the book will give them a language to be able to say, no, my leadership has not made it easy for me to do my work easily and well. And there's got to be either one of three things has gone wrong. And what I hope it gives leaders is especially ones that come from a technology background or are engineers, the same theories and intuitions and experience that helped us build great technical systems are actually perfect for us to think about, you know, the socio part of the socio technical system. Oh, yeah. Okay. So system design, either way, like architecture design, but also applying that same thinking, breaking things down into components.

19:46Again, understanding that we try to simplify, amplify the key pieces so people can spend more time on key tasks versus less time in, you know, status meetings, for example. 100%. In fact, can I provide that up? Yeah. Yeah. So I love the way you talk to that, like an architect, right? When Steve and Gene are moving on the couch together, they are coupled. Right. What affects Gene affects Steve, vice versa. But it means that they are coupled together. And there are times when we want lots of coupling, like when you want dev and ops pairing together because you want that incredible flow of information to be as wide and fast as possible.

20:24The problem is, is that in order to make a change, involve both Steve and Gene. Right. So that's great. But it can get to a point where you have 35. imagine a couch that 35 people are attached to and none of the 35 can do anything and they've lost independence of action. The autonomy piece we kind of alluded to earlier. So maybe it's time to divide up the couch, right? Without destroying it, right? But somehow re-architect it so that, you know, maybe it's 35 smaller couches. And what are the pieces that can actually, you know, be done independently of each other? And the benefit of that is independence of action.

21:02So the other thing that we can get wrong, what we got wrong in that part is that When we are overly coupled to the organization, not only does anyone have independence of action, but we waste a lot of time. You're listening to things that have nothing relevant to you. Small things require vast amounts of effort to get everyone's permission. And then it's not just permission, you have to schedule together, prioritize together, worst case, deploy together. If you're watching on YouTube, you can probably see me shaking my head right now. I'm thinking of some examples of when I've dealt with similar concepts before.

21:37This is really wonderful. I love the way you have broken this down into these simple explanations that then build outwards into real-life scenarios people are dealing with. Are there other key concepts from the book that you want to highlight for our audience? You know, I have a super fun example that actually a group of people came up with that I just fell in love with. Imagine a fictitious telco. that the most important thing that they could do is just present a checkbox to the 30 million customers that would allow them to opt in to like a$5 a month service for email, watch movies, et cetera.

22:12The problem is that it has to cross 40 different teams across four different customer channels, digital, store, support, filling, and it will require CEO minus one support. It requires daily war room meetings and it will take about nine months. estimate is about$20 million. And most people think it will have a 20 % chance of success. Why? Because it didn't work the last two times we tried. Doesn't sound ideal. Like the nice concept, but yeah, okay. And what's awesome about this example is that it's not because it's technically challenging, right? It's because somehow the organization is wired wrong to make it almost impossible to get this work done.

22:55And so, yeah, I think it's just a great example of, it just so vividly shows how we can easily create systems where to get small things done require like super heroic efforts because the coordination cost is so high. And these coordination costs often become even more and more impactful as organizations scale too, because, you know, when you're a startup, maybe you're 20 people working in the, you know, one office together, you're all fired up about the idea, you're passionate. It's easier to ignore these coordination costs because it's a smaller group, there's less people you have to talk to.

23:29But as organizations start to grow, it's so crucial that you think about these concepts and ensure you're being intentional about them because by that like 100 person mark can definitely be on once you're getting in thousands. There can be such a tax on coordination within your organization if you're not careful. Yeah, and in fact, one of the more moving parts of the book was showing the examples outside of technology. There was a section that Steve wrote about how when one of his daughters was six years old and got in an accident on the playground, had to go to the hospital. And so they show up into the ER and it takes hours to get through the paperwork.