Helping Government Software Teams Move Faster | Carnegie Mellon's Robin Yeman

0:00The Department of Defense and the intelligence communities really started looking at the environment and saying, hey, we have to go faster. Right. Currently today, if we were to launch a satellite, it takes 90 months. That's just too long. Right. And we've got new entrants into the market like SpaceX or Relativity, and they can do it so much faster. Why is that? How is that? My goals and objectives are really to enable government organizations to become digital so that they can do agile, they can do DevOps, they can do model-based systems engineering, they can build digital twins, they can bring in artificial intelligence and machine learning.

0:41I usually refer to this as all of the tools in your digital engineering value stream. How can you unlock the full potential of generative AI for your engineering team? Getting started with your Gen.AI transformation initiative isn't enough unless you understand the impact you can have. With insights from industry-leading CTOs and VPs of engineering, Linear B's Gen.AI Impact Report highlights how you can measure Gen.AI's impact on your software development lifecycle. Discover which use cases are delivering the most value and learn how you can track adoption, value-based benefits, and mitigate risks.

1:14Don't miss your chance to stay ahead of the curve. Head to the show notes to download the Gen.AI Impact Report today and start measuring the impact of AI in your development processes. Hey, everyone. Welcome back to Dev Interrupted. I'm your co-host, Connor Bronson, and today I'm delighted to be joined by Robin Yeeman, Space Domain Lead at Carnegie Mellon's Software Engineering Institute. Robin, great to have you with us. I hear you're also a former TEDx speaker. Yes, it was actually a fantastic experience. I did a talk on human collisions. So it was right during that time during COVID where we were like, yeah, everybody's working from home.

1:49But what things weren't happening, right? We hadn't actually gotten good at this whole working virtually, really engaging with one another. And so really talked about making sure that we can still do that, right? Because some of the best ideas happened when I ran into you at the lunchroom. Totally. Not necessarily when I was like banging away at my keyboard. We have to be really intentional about the social design, the social circuitry of these hybrid or remote organizations, because you can create that kind of extemporaneous collaboration. But if you're not intentional about it, you fall into these negative patterns as you bring up.

2:25And this, I'm sure, spans from your expertise over the years. You've spent over 28 years in software engineering with a focus on digital engineering, DevSecOps, Agile, and you've been a leader. You've built large, complex solutions across multiple domains. In fact, you have a pioneering research that you've done in collaboration with IT Revolution and a number of other industry experts over the last five years. And you're releasing a book now, is my understanding, but with a set of proven success patterns. And it's under the title Industrial DevOps. Can you tell us a little bit about the book?

2:55Yeah, absolutely. So we've seen huge success over the last, let's say, two decades with Agile, right? People have been able to go faster. They've reduced lead time, improved communication. DevOps expanded that, so now we've got the tooling and the automation to really deliver capabilities at the speed of relevance. I wanted to be able to do that for safety-critical cyber-physical systems. So during my time in engineering, I've spent a long time building systems like submarines or satellites or aircraft, things of that nature. And they also need to be able to deliver faster, have high morale, build quality in.

3:42So what happens when we take those practices that are best for, let's say, software with Agile and DevOps, and we apply them in the arena of cyber physical systems? And what we've seen is actually even bigger results than maybe just the software alone. And your book's coming out October 12th, so it should be live the time this episode is. Where's the best place for folks to find your book? Amazon. The best place, go to Amazon. It's on pre-order right now. Or IT Revolution. You can also get it right from Gene's website. Perfect. And as you mentioned, this book has been an offshoot of your work over the last 25 years at Lockheed Martin.

4:24designing everything from secure submarines to satellites. And you also have a co-author who has experience in Northrop Grumman and beyond, correct? Absolutely. Dr. Suzette Johnson is a fellow at Northrop Grumman. And actually, we started collaborating together almost a decade ago. And I was working at Lockheed. She was at Northrop Grumman. And a lot of people said, hey, how unlikely collaborators, right? Because inherently, we're competitors. But we really refer to it as competimates because in these large systems, say F-22 or F-35, that's not a Lockheed alone system. That is a Lockheed, Northrop Grumman, Raytheon, right?

5:08All of these contractors are in there because these systems are huge. So this is really fascinating. I'm curious how it led to your work today at Carnegie Mellon. So I spent a long time, became a senior technical fellow at Lockheed. and I really liked it. However, I wasn't done yet. A lot of times you get to that senior technical fellow role, it's kind of the last role or it has been for many technical leaders. But I wasn't done yet. In 2018, some really interesting things happened. So the Department of Defense and the intelligence communities really started looking at the environment and saying, hey, we have to go faster.

5:49Right. Currently today, if we were to launch a satellite, it takes 90 months. That's just too long. Right. And we've got new entrants into the market like SpaceX or Relativity, and they can do it so much faster. So why is that? How is that? And that was a perfect opportunity to come work at Carnegie Mellon. Why? Because it's an FFRDC. And for those of you who don't know what an FRRDC is, it's really an organization that works with the government. They support the government needs. And my goals and objectives there are really to enable government organizations to become digital so that they can do agile.

6:31They can do DevOps. They can do model-based systems engineering. They can build digital twins. They can bring in artificial intelligence and machine learning. They can make sure that they bring in cyber. And so I usually refer to this as all of the tools in your digital engineering value stream. And this is exactly what you're doing with the book as well, right? As you're expanding on this and going in depth. I know you're giving a talk later today around this and the approach to industrial DevOps. Where should folks get started as they start to approach these concepts? There's a couple things.

7:03One, have a growth mindset. We always say that, but what does that mean? If you were to look, let's say, five years ago, even 10 years ago, we may not have had some of the tools to do this. But now we're in a place where we can take physical systems and put them into cyberspace, which means that I can build them at the same speed and get the feedback loops that I could in software. So the first thing you want to do is recognize what didn't exist before these new tools in your toolbox. ensure that you understand the tooling that is required, meaning I have to have test labs much earlier, right?

7:43If I wait for tests to the end of the system, it's way too long and it's actually a lot of rework. So we need to invest earlier. Look at your org structure. Inherently, when we have organizations, especially from like industrial age, they're all functional based, right? And you You know you have a functional organizational structure if you've got program managers, software engineers, systems engineers, test engineers. Here's the interesting thing. They don't speak the same language. And communication follows the org structure inherently. So if I'm a program manager, I know maybe something about lean, lean startup.

8:23If I'm a systems engineer, I probably have heard of things like system thinking and design thinking. If I am a software engineer, I have probably been engaged in the Agile community or leverage DevOps. If you talk to folks within the test community, they're going to talk about things like shift left. Awesome. And operations is big into what I would say ITIL or IT infrastructure library. Now, the interesting thing about all of these approaches is they all are designed to optimize the speed of delivery while maintaining quality through your system. But we've got org structures where we're talking past each other, right?

9:04You're telling me about Lean Startup. I'm trying to tell you about Agile. We aren't talking the same language, meaning we can't actually deliver around that value stream. So how does the research for your book come together into this concept? I mean, there's clear variety here. There's a couple things. So my experience at Lockheed was building really large systems. So I can tell you with pretty good accuracy how to build a satellite. I can tell you, you've got command and control, you've got telemetry, you've got mission management, etc. One of the things that I got to see with these large systems is actually this lack of communication.

9:47Interesting, I could see that the program managers, while they were talking about this thing, software was talking about this, and we weren't actually collaborating. You'd be amazed at how many times there would be a gap in the system. obviously it got fixed during, you know, let's say rework cycles or overlap, meaning multiple teams, right, addressed it from a different approach. Most people, many people never actually see across these large systems. You know, Orion is another good example. It's a space vehicle. There are hundreds of teams. It's a very large system. And so a day in the life of most engineers is going to be my piece.

10:30But the day in the life of a tech fellow is I'm going to see how all of the pieces come together. And that experience allowed me to see how the problems with the organizational structure as well as our architecture was leading to us not delivering systems as fast or as best as we could. Do you feel like the industry, particularly in the defense side of things, the government side of things, is still struggling to address these challenges? It sounds like you do. Oh, yeah. I mean, this is a huge challenge. Now, I would say it's a Department of Defense thing, except that I've had the opportunity to do some collaboration with some automotive companies and some healthcare companies and things like that.

11:10And actually what I'm seeing is anybody who's building cyber physical systems in highly regulated environments, whatever that is, are experiencing very similar problems, right? Making that leap from, you know, industrial and manufacturing to this digital age has resulted in a whole new way of working. And they haven't adjusted how we communicate, how we organize, and the tool set to be able to do that. So what are some of the proven success patterns that you've discovered where companies or organizations can adapt and be successful? First one, and this is pretty well known, really look at your work structure, right?

11:52Cross-functional teams. Now, I usually would say that that's kind of a grid, meaning I've got teams that are focused on delivering the outcomes, and I've got folks that are concerned about their specialization, right? So I really want my software engineers to have the latest and greatest understanding of test-driven development, things like that. So is that what you're referencing here is like a platform team, a developer productivity, devx team, that kind of thing? Absolutely. And so the benefits of that are huge. Now, you could say that you're creating a different silo, right? Because we're saying, hey, we're creating a silo around the stream of value.

12:30But it's a tradeoff. You're never going to have the right and perfect organization. You have to understand what your goals are. And in a conversation with Adrian Cocroff from Amazon, he said they make a very intentional approach to organize around their value streams, to organize around throughput, even though they know in some cases they probably run into things like redundancy or repeats because their number one focus is their customer. Right. Right? So that's really what many organizations' number one focus today is too. Or often should be. Or often should be. or you're probably not going to be in business a super long time.

13:09Okay, so this reframing of organizational structures, what would be your advice to an engineering leader that's looking at their organizational structure and maybe building a cyber-physical system as you bring up about how to reframe and refactor it so that it is more effective? The first thing is don't do it in a silo, right? So when I look at the org structure, I could just say, hey, all right, everybody go to cross-functional teams. Right, but it becomes the buzzword, right? And the system's just going to break down. I have dependencies. So I have to do two things. I've got to look at my org structure in lockstep with my system architecture.

13:43So a contextual piece. Yes. And so when I look at my architecture and my organizational structure, I'm going to do a step. We would call it a reverse Conway's law maneuver. A step where I refactor the architecture to remove some of those monoliths, maybe create some more modularity. and then change that particular piece of the organization. So it's got to be done incrementally. If I was to just go into any large company, probably Boeing, and just say, hey, everybody, cross-functional teams, they're not going to get planes out anytime soon because currently I guarantee they have a lot of legacy architecture and so there's a reason why we're organized the way we are.

14:29So you've got to do that in lockstep. Look at those two pieces. Interesting. Okay. So this is a fascinating question of organizational design. And I know Gene Kim and Steve Spear, his co-author on their new book, Wiring the Winning Organization, but using this phrase, social circuitry, to talk about rewiring. Is this a concept that you see as really crucial here when you're creating these new alchemy of teams? Yeah, absolutely. I had the opportunity two days ago to go to their workshop, right, rewiring the organization. And as much as I think that I already knew, I learned some stuff. And I was like, okay, so this approach of simplicity, slowification, amplification are things that I actually need.

15:18So while I think I kind of get to some of those items, they articulated it brilliantly. So actually, I was on a conversation with a customer this morning and I said, hey, I want to refactor this workshop to bring in some of these concepts that I learned from Gene and Dr. Spears' book to basically make sure that we're doing those things, right? Because these systems need exactly that. Simplification, well, we need modularity, linearity, all the things that they bring up. And that's really hard. Slowification, we really need to have more time to practice and understand. A lot of these big systems, we never have any time to slow down.

16:01We have a tight schedule. We got to get it done. But the funniest thing happens, we don't have any time to do it the right way the first time, but we always got time to do it the right way the second time, right? Because we do rework. And so I think that some of those concepts kind of galvanize the things that we have in our book. So I'm anxious to be able to take some of that and actually bring it together. What are the other archetypal approaches that are in your industrial DevOps book? Other approaches are going to be planning. So if you talk to a pure, I don't want to say purist, but maybe pure agilist, they're going to say, hey, we're going to plan every two weeks.

16:41We're going to plan two weeks at a time. So if I'm building something like Fleet Ballistic Missile, which is a 50-year program, it's not going to work for me. It's just not. And the system won't come together. We refer to what we call multiple horizons of planning, meaning I need that big, long plan. I might need a 10-year plan. I might need a five-year plan. And typically, I do. And then I take that and I break it down into an annual plan, which then I break down into a quarterly plan and a sprint plan. Now, the coolest thing happens when you do that. I get empirical data, right? So I get empirical data on how fast I'm able to do things.

17:19I get empirical data on the rework metrics, all kinds of things. And I can take that data and I can use it to inform that next horizon, right? So for easy math, if you plan to get 10 things done today and you got six done and you did it tomorrow. It's a typical day for me, I think. All right. I have those days. And you did that tomorrow. Well, in one week, I can pretty much say that it's likely you're going to be 60 % complete. Now, if you keep planning to do 10 things a day, you're basically providing inaccurate data, right? People that are waiting on you. You're eliminating predictability. Exactly.

17:55So I can't predict the future, but I can constantly use that empirical data to inform my plan. And so I can do that with the annual plan, the five-year plan, etc. So when you're looking at these long-term plans that you're then breaking down into bite-sized chunks, so to speak, are you largely using quantitative data or are you also bringing in survey and other qualitative forms? Both. But I see the gap primarily around the quantitative. I have seen many projects where what we call the integrated master scheduler, so the person that's got the keys to the schedule, is completely divorced from, let's say, the teams who have the work in their sprint schedules.

18:37They don't match. They don't even slightly match. And the problem is a traditional, let's say, master scheduler isn't in the Agile tools. They've never done that. And a traditional agilist or a scrum master, they're in the tools, but they don't actually recognize the gold in the data they have. So it's that silification effect you talked about that we need to address. And that's an org design issue in part. Absolutely. So bringing them together to have these multiple planning horizons and then understanding the importance of that data and using it to further inform has been huge. And it sounds like you think this particularly applies when you're doing industrial scale work.

19:21Because to, I guess, to look at a counter example, like a startup, for example, can't plan on a five-year horizon. They can maybe think about it a bit, but they have to be much more iterative early on because they're usually pre-product market fit or still trying to figure out their approach. Right. So the thing you got to take away from there is context matters. Definitely. Absolutely. So there are cases, if I'm building an update for an application, even something within the government, a small application, every couple days, that's perfect. I can get fast feedback. If I'm trying to build an aircraft, I have to order hardware.

19:56I have to make sure that I've got the supply chain coming in at the right order so that I can do the assembly line. There's other factors. Many dependencies. Exactly. So context matters. It depends on what you're building. Yeah, and while software obviously has plenty of dependencies, especially if you're bringing open source projects, all these things, that software security supply chain is much easier to pull from the cloud, to move much faster. So this is a good explanation. I appreciate it. What other patterns are you seeing in organizations? So the first two have been fascinating. What else is there?

20:32The other patterns that we're seeing is technology has exploded in many areas, right? So materials. files. Now I can do additive manufacturing or what we call 3D printing to get fast feedback. Digital twins. A digital twin is technically a cyber system connected to a physical system with sensors and data. And this is a full simulation in a cyber environment, similar to, I'll use an example, Amazon does this with a warehouse, for example. Absolutely. And so does, you know, Tesla, each and every Tesla coming off the line has a digital twin. Got a chance to talk to Joe Justice, who worked there for a while, and he said not only that, they take those digital twins, they create a digital ecosystem, multiple digital twins, and then they use artificial intelligence to find patterns.

21:21Why? Because in these large safety-critical systems like a car, there's something called emergent behavior. And when I bring together a lot of these different digital twins and let it run through a lot of different scenarios. I find things that I as a human would take hundreds of years to find. Interesting. This is a fascinating concept. Are there other key concepts in the book you want to share with us? So some other things are going to be, we say shift left. We have provided some real practical examples on how to shift left and how to begin with test. In my world, at one point in time, I had the opportunity, well, many opportunities to respond to what we call RFPs or requests for proposals.

22:05And these proposals, they have these requirements. And I read through one and it looked perfectly reasonable, like all the requirements. And so just for grins and giggles, I thought I would go through each requirement and try to validate how I would test it. How would I prove that this happened? On the front end, before I even started building anything. And the interesting thing was 32 % of the requirements weren't testable at all. They seemed reasonable, but when you said the system shall be performant or it shall be scalable, to what? And those are the things which allowed us to really reduce that rework, to build it right the first time.

22:42There's other things like we talked about is that architecting for speed. Modularity, standardized interfaces, huge, both in software and hardware. And really bringing together kind of the different cultures. because we're continuously learning. My youngest son just graduated computer science degree from UCF. Congratulations. Yeah. And so, you know, he's out looking, you know, getting his first job, things like that. And he's like, oh, it's like you have to keep on learning. And I'm like, yep. That's called the suck it up buttercup concept there, dude. You pick technology. Yeah, yeah. It changes every day.

23:21That's the fun part, yeah. It's the fun part, but like, yeah. Fantastic. Well, Robin, I really enjoyed this conversation. Thank you so much for sharing all these insights from your book, Industrial DevOps. I'm looking forward to checking it out. Awesome. And thanks for coming on the podcast. It's been great. If you're listening to us here, check us out on YouTube, too. You can see Robin and I in the midst of this giant plastic dome. Very cool. Thank you. Yes, we'll clip that. That's very cool. That's perfect. And thanks again, Robin. All right, cool. Thanks.