How to measure and improve developer productivity | Nicole Forsgren (Microsoft Research, GitHub, Google)

25:21Basically, you're doing great. That's kind of the message of this framework at least. And if you're not doing it through like brute force and killing yourself, now can I jump in here? Because then people are like, but how do I do this? So let's say that you're not in that category. And you're like, because this is the next, this is the piece of criticism I'll get about Dora. Right? People are like, well, all you've done is make me feel bad. You gave me these metrics. You've judged me. Now I feel bad. And then I'm like, so there's Dora, I wrote a book called Accelerate, right? Which is like the first four years of the research compiled and put together and expanded in a few things.

26:06And I'll joke, there's a whole rest of the book, right? Dora is best known for the four metrics, but there's an entire research program supporting it. So it's not just these four metrics. What we find is that if you improve a set of capabilities, I love your question around what is DevOps? DevOps is not the tool chain you buy. Marketing teams label tool chains DevOps. They wanted your money. DevOps is a set of capabilities. They're technical capabilities. They're architectural capabilities. They're cultural capabilities. They are lean management practices that predict speed and stability. And then speed and stability gives you money.

26:46Right? Because like, it's your ability to create these features that give you money. So when you work backwards, if you want money, like you get the features fast, if you want the features fast and stable, you do the things. And the things are technical capabilities like automated testing, CICD. And CICD is continuous integration, continuous deployment. Is that right? Yes. trunk -based development using a version control system. Right? So do you have good technical practices? Do you have good architectural practices? Like do you have a loosely coupled system? Are you using the cloud or like if you're not in the cloud for whatever reason, are you using the underlying architectural pieces that enable good cloud to do the cloud, right?

27:39Or if you're in the cloud and you're not realizing benefits as it goes, you're doing the cloud, right? Do you have a good culture?

27:48So you don't just like magically go fast and have stability, right? So working backwards, which pieces are you struggling at? Now, you kind of noted down the benchmarks. If you go to doora .dev, the team at Google was lovely. We worked really closely with the team and they're keeping this updated. You can take a quick check. There's a button there that says quick check. You can plug in where you kind of think you are. Like I said, you can hunch it. And it'll tell you where you are in the benchmarks today. And what industry you're in and then the cool part is it'll say, now, like you'll want to ask yourself, like where am I struggling?

28:25But it'll say for your performance profile and for the industry that you're in, statistically over the last several years, these are probably your constraints. Okay, these are probably the things that you're struggling in right now, right? Like for people finance who are high performers, they tend to struggle with these four things, right? Whether it's like culture or continues integration or whatever. I love that you're getting tactical with how to actually improve these already, which is the bread and butter this podcast. And so we'll link to this quick check. It's just doora .dev slash quick check.

29:03And by the way, they do not collect your name. They do not collect your info. There is no, there's no lead, any lead gen, anything. Everything's just there. And then there's deep dives into every single one of the capabilities. Amazing. And also your book talks about all these things. So people should go check out the book obviously. It's on Amazon search accelerate. Is that right? Yep. Okay. So we were talking about doora. This may be a good time to talk about space, which I think is a different framework. You recommend. What is that all about? Okay. So space is a way to measure. We say productivity, developer productivity, but it's a little bit more than that.

29:39Space is a good way to measure any type of complex creative work. Now, how do they relate? Let's say you go through the quick check. It points out like four things and you decide you want to improve continues integration and culture. Right? Well, now you're like, cool, but how am I going to actually measure them? This is where space comes in because space helps you figure out space gives you a framework to pick the right metrics. Now, some people are like, well, space, you didn't give me the exact metrics. People love dorks. It's like, here's the exact for you need. Well, space is like when you want to measure something that's complex creative work, maybe like developer productivity, there's also an example at the bottom for incident management.

30:30When you have something you want to measure, it says within your context, within the metrics, you have available to you. Here's how to pick. That's what space is good for. Now, we called it space because it stands for the five dimensions that you want to measure. So S is satisfaction and well -being. So satisfaction and well -being is kind of self -explanatory. Now, some people might jump in here and say, oh, well, you're just, you know, touchy feeling. This actually matters because we find that satisfaction and well -being ends up being incredibly highly correlated with all of the other dimensions of productivity doing things well.

31:09And as soon as satisfaction and well -being, things like sustainability, if you're satisfied, as soon as that starts falling off, other things start to break. So this can be an incredibly strong and important signal. P is performance. This is going to be the outcome of a process. So reliability within Dora, the MTTM or Change Fail Rate, those are both performance metrics. And so you pick one to kind of measure as performance. Yep. A is activity. Anytime you have a count or a number of something, these we see all the time because they're super easy to instrument and automate, right? A number of pull requests, a number of check -ins, a number of something, that's A.

31:53C is communication collaboration. This can be how people work and talk together. It can be meetings, it can be collaboration. It can also be how our systems communicate together. It can be the searchability of a code base. And then E is efficiency and flow. So this is going to be the flow through the system. It can be the time through the system. If we think about SRA or your incident management, it can be the number of hops a ticket takes until it reaches the right person. Now, to use space correctly, we want to use at least three dimensions at a time because that helps us balance. Turns out Dora is actually an implementation of space.

32:32So Dora would be space for mostly that outer loop. So again, once you found something that you want to improve, find the metrics that make sense to you, try to have them be in balance or intention so you don't like throw something at a whack, but pick three. So when you say Dora is an implementation of space, one has five buckets, one has four, how does, how do you actually think about that? So space is there to help you think about how you want to pick metrics, right? So a lot of time I see people, so we have to have step back. I used to advise people on how to pick metrics, right? For years, people would pull me in to advise on Dora or accelerate, right?

33:15They would ask me questions, but it ended up being metrics questions a lot. How do I pick the right metrics to improve what I'm doing, right? Like I said, they had the Dora numbers, they would pick their constraints and they wanted to improve, but how do I improve? How do I measure this? How do I show improvement? And so we would start thinking really critically about which metrics were the right metrics to pick. And I would always say make sure you pick balanced metrics, make sure you pick metrics that are intention. And I could say it, but people have a hard time wrapping that around their heads because they kept picking things like number of lines of code, never picked number of lines of code.

33:57Number of putt still every month, again, these numbers, number of pull requests, number of commits. And I was like, these are all activity metrics. And so finally, I pulled a few of my friends together and I was like, let's come up with a framework to help people think about it. And so there are five broad categories, pick three, because that will help force you through the mental exercise of what could I possibly pick? You don't need all five, right? This isn't, we're not playing bingo, we're not playing different dimensions. Now one example here, I was working with a group that wanted to improve their pull requests, very generally.

34:40They just said improve pull requests. So they were thinking about pinging someone every 15 minutes and I was like, oh, this is going to be bad, because we know from other literature and research like nursing, you'll get alert fatigue where people will just start tuning out alerts. Either they'll turn them off or they will just stop hearing them. So like, number of alerts, right? They're like, let's just think about number of alerts. And I said, well, if we think about efficiency and flow, how much time do you have to work on your coding? So those two are balanced. So we need to protect time to work as well as code review time, right?

35:26Pull requests time. And so sometimes we can think about those and then, and then we, I think we added a satisfaction metric. Are you satisfied with the pull request process and the selection of the reviewer? How do you go about actually capturing and measuring the say satisfaction? So for satisfaction, I would, I would generally ask, right? Go ahead and ask. Now the ones that you instrument, you can instrument and pull out of systems all the time, right? Go ahead and grab that string for a satisfaction metric. I would only pull that periodically, right? Once every few months, like a survey to your engineering thing, like a survey.

36:02Yep, absolutely. Awesome. And don't, don't discount what people say, right? Sometimes I hear actually that sometimes a lot of times I hear people say, oh, but people lie. First of all, what is there incentive to lie? Why would they lie about having a bad system? Because it's bad and they want it fixed, right? If it's absolutely a hostile work environment, they might lie and then tell you it's good, then you have bigger problems, right? Also, do we ever see bad data for my systems or incomplete data for my systems? That's a lie. But we find ways to deal with it. We see it, we acknowledge it, we look for holes in our system data, we try to deal with it, right?

36:49That's also a lie. So I think there are better ways to think about and deal with that and then try to work with it, right? Because then I wrote a paper with Mick Kirschen on this several years ago on how data from people and data from systems are really important compliments because we can get certain insights from people that we'll never get from systems, like let's look at lead time from changes, for example, from commit to deploy. The speed might be fine, but people might tell you it's taking absolute heroics, it's some ridiculous Rube Goldberg machine. The system will never tell you that. Or you could get data on your version control system.

37:36I worked with a company several years ago and we found out that there was a significant portion of code that was just not going into any version control system. You're never going to find that out from your systems because it's not in the systems and it was mission critical. I can see why people come to you asking for advice on metrics because you have this framework of here's the type of metrics you want and then I think, especially for a engineering team, there's going to be this like how do I optimize and make sure I'm doing the right thing and measuring the right things for someone that wants to do this and an hour long podcast isn't going to give them all the answers.

38:12What do you recommend they go read or go do or look at to help them figure that out? One, I hate to be this person, but I'll point to a few of my papers because I will say I write things down because I get asked them so often and I want to make sure it is broadly applicable or broadly available, I guess. This space paper for sure it's an ACM and I think the year we published it, it was like the most red paper at ACMQ. Yeah, so we tried to make it as readable as possible so the space paper is nice because it outlines this framework and it gives examples of metrics in every single category and so hopefully people can look at this and they can say, okay, here's an example to use here, right?

38:57Here are some of the things that I could possibly use and we're seeing that space is being used lots and lots of different places. Another good one could be the paper that I mentioned with Mick Kirsten and it was about, we talked about using data from people and data from systems. We wrote it up in the DevOps context because I want to say this was written in like 2016 or 2017 or something, but it helps you think through what types of data are good in which situations, right? Because you will never find yourself in a situation when you don't want both types of data. Even teams that I've worked with that are the most advanced, they have absolute instrumentation in every possible scenario.

39:45In the most detailed way, they will still survey their developers at least once a year because you can get new insights, right? One book that I love, it's a little dense but it's really interesting that I love is how to measure anything and it's by Hubbard and there are parts of it that are like real stats heavy, but he has this portion in the front that's like in covering intangibles and it's like what happens when you don't have data? You have no data, you're starting from nothing, what are good ways to hunch data? And I really love that because he he covers some really good ground there. Today's entire episode is brought to you by DX, a platform for measuring and improving developer productivity.

40:40DX is designed by the researchers behind frameworks such as Dora, Space and DevX, including Nicole Forsgren, who is my guest for this very episode. If you've tried measuring developer productivity, you know that there are a lot of basic metrics out there and a lot of ways to do this wrong and getting that full view of productivity is still really hard. DX tackles this problem by combining qualitative and quantitative based on the very research Nicole and her team have done, giving you full clarity into how your developers are doing. DX is used by both startups and Fortune 500 companies, including companies like Twilio, Amplitude, eBay, Brex, Toast, Pfizer and Procter & Gamble.

41:19To learn more about DX and get a demo of their product, visit their website at getdx .com slash Lenny. That's getdx .com slash Lenny. You also mentioned offline that you might be working on a book that will answer a lot of these questions. Is that something you're up for chatting about? Yeah, absolutely. So as I mentioned, I tend to write things down when I get asked questions on it a lot. And so this is one in particular. So we'll be covering, you know, I'm starting to go through and I'm covering some of these. And I think some of the important topics in particular are, you know, starting with what is your problem or what is your goal?

41:57And being super, super crisp on it, right? Like what is it that we're trying to answer? And I would say this is a bigger challenge than most people recognize or realize. Like I'm making this setup, right? Like 80 % of the folks that I work with, this is their biggest problem. Even at like executive levels, they'll ask their team or teams will come back with uncertainty and they'll say like, well, you told me to improve developer experience. And like, okay, great. What do you mean by that? And then teams will have gone off for several months and they're tackling something and they'll come back and they'll be like, oh, well, that wasn't what I meant.

42:38And like, okay, what do you mean by this? Are you talking about inner and outer loop? Are you talking about friction? Are you talking about culture? Because sometimes they're talking about culture. And if you're talking about culture, this is an incredibly valid answer. But if you're talking about culture, this is totally different than if you're talking about friction in toolchains, right? And if you're on different pages, you're heading in completely different directions. So like that, that's one thing we cover, which seems obvious, but trust me, it is not. And then even like how do you, we're going to do kind of a rough version of like, how do you start measuring from nothing?

43:17And also the measure majority, right? Like how do you think about the tradeoffs between and the proportion of measurement between subjective data, right? Data from people. So you have interviews and you have surveys and objective data, stuff you get from systems. Because when you first start off, you'll be relying much more on data from people because you can get it relatively quickly. But as you kind of transition through this measurement journey, you'll get more and more data from your systems because it's scalable. It can be engineered. You can be doing, you know, much more with it. And also you should be thinking about, you know, don't let the perfectly be the end of me the good, right?

44:00So like how do we think about this very, very strategically? How do we transition through this? How do we think about what each piece of data is for? And also like lots and lots of examples, right? So I have included like example interview scripts. How do you select people? How do you screen people? Example survey scripts? What are some of the analyses we should do? And trying to make this incredibly accessible. So like basically anyone can do this. So you do not need to be a data scientist. But if you have one on staff, like you could hand them some of this and just like let them run. I think this book is going to do extremely well.

44:34Definitely come back on when it is out. I think you said maybe your ish kind of time frame. Yeah, probably about a year by the time we get all the way through. If people want to be notified when it's out, can they sign up on your site for a newsletter or anything like that? Is there anyway kind of being on in the loop as it approaches? Oh, yeah, absolutely. Yeah, I'll add, I'll add a link for that. Also, if anyone is doing some of this work now, if they have, you know, major questions that they would love to hit me to answer, if they have success stories, if they have case studies, if they have anything that would love to be included, I remember when I wrote Accelerate before, there were a couple folks that reached out after and they would like, oh, I wanted to have something included.

45:14Now I've looked today off learned, right? If there's anything that folks would love to be in discussion with me about, I'm always eager to chat and nerd out about DEVACS and especially measurement and measurement journeys. Awesome. I usually ask this at the end and I have more questions, but while we're here, how would people reach out to you? What's the best way to contact me? On my website, I've got info .nicolefee at Gmail. Okay. Awesome. Okay. A few more questions. Awesome. Thanks. What are the most common pitfalls that companies run into when they're trying to roll out any sort of developer experience developer productivity system measurements improvements?

45:53I think one I just mentioned, right? Like not being clear or not understanding what it is that they're looking for because then you can have a thousand flowers bloom and everyone's kind of running in a different direction. I think another one is not pursuing this in both a top down and a bottom up structure, right? And I think that can really help drive success and having good communication throughout is super, super important, right? So getting your ICs bought in and helping them understand that this is for them, we want to understand what they're doing, knowing what vocabulary they use, what terminology they use is super important, and then chatting with leaders, right?

46:45And understanding what their motivations are or helping them understand what the motivations could be, you know, this kind of harkens back to one of our earliest chats on why I even got into this and how I see two different sides to the conversation on like, why is DevOps even a thing? Why should we even ship faster? Like, there are so many people that I talk to that are super passionate about DeVex right now and they're like, how can I convince my executive team this is important because their developers are just completely burning out or they use computers in anger every day. And so it's like, how can we have the right tools to socialize this to our leaders as well, right?

47:25Because this should be a priority, this needs to be a strategic piece, and how can we help pull together the right value points to communicate this and to understand what their priorities are so that we can see how this fits in, right? You've been working in this space for a long time, probably longer than anyone that has ever worked on this area of developer experience productivity. What have you seen change most from the time you started working in the space to today? What kind of progress has been made? We have these increasingly large complex systems, right? So like 10 or 15 years ago, like the internet was around, but like things were really different now.

48:06Almost every company has a really large complex system, right? We also have a shortage of developers, or at least a reported perceived shortage of developers. More companies are technology driven, or at least they understand their technology driven. So like I understand a handful, I remember a handful of years ago when I met with a financial institution whose CTO insisted to me that he was not a tech company. Like that, that's not real anymore. That doesn't happen many more companies are like, we have to be better at this. And that was not always the case, like five to 10 years ago. I used to have to really explain why this was a pressing concern, why it would continue to be a pressing concern.

48:58And now in the last six to nine to 12 months, we have this AI moment happening. And it just poured gas on top of everything because now what's important, like we've always said that like ideas aren't important executions important, but now this is absolutely true because it's not just about what it is that you build. It's about creating absolutely novel, incredibly new experiences and doing them at a speed that no one has seen before. And the only way to do this is to have the software pipeline that is fast and is safe, and is stable and is reliable. And that's where we're seeing this really interesting convergence and pressure isn't quite the right word, but it's really forcing the discussion.

50:03And strategy and prioritization, right? I'm glad you touched on AI. That was actually exactly where I was going to go next. Yeah, obviously productivity, AI engineering, something that's top of mind for a lot of people, there's a lot of layoffs that have been happening. There's a lot of talk of we don't need as many engineers. I actually had dinner not too long ago with few, I'd say 10X engineers. And those are folks that people sometimes say like they don't need co -pilot, they're not going to use any of these tools, they're already amazing. And they were the opposite. They're like, this is making me 100 % more effective and efficient than I love it.

50:39So clearly good things are happening there. I don't know what the question is specifically, but I guess have you seen the impact of AI on engineering productivity? And is that shifted? How do you think about developer experience and productivity beyond which you already just shared? Absolutely. So yes and, right, I think this is a super interesting open question. So can I answer it just with a whole bunch of questions? Absolutely. We're absolutely seeing an impact and we continue to explore this. So like I have an interesting question to see like how it'll change the space framework. What's open here?

51:17I think a few things will remain, right? Satisfaction is still going to be there, performance is still going to be there, activity is still going to be there. How you communicate with people and with the tool, efficiency and flow is still going to be there. I believe it will change and add a dimension like trust or reliability, right? How do I rely, can I rely on it? Well, I have an over reliance on it. And what we're seeing is that probably unsurprisingly, people really fundamentally shift the way they work when they work with an AI enabled tool like GitHub Copilot or like Tab9 or others. Because now instead of just writing code or like having a short auto -complete, you spend more time reviewing code than writing code, right?

52:01There's this wonderful paper out that uses the CUP's model. I'll share it with you, a team at MSR did it. The finds that, you know, about 50 % of your time now is spent reviewing versus writing. But it'll be interesting to see how that changes things longitudinally, right? Because other, you know, some of my colleagues also did a paper that showed that, you know, you can do certain tasks like building HTTP server, you know, 50 % faster. But I don't think that's what productivity is about when you're using an AI tool, frankly, right? Anyone who's looking at that and like, dear CEOs or whoever who are like, now I can lay off my workforce.

52:40That's not what this is about, right? It's not about taking a task and cutting your time in half because now what we've enabled is your ability to do certain things faster and then free up some of your cognitive space so that you can do harder things with this new Copilot sidecar or something, right? But also because now you're accepting text and then reviewing it, we've changed what your mental model is. So we've changed the friction model that you expect. We've changed the cognitive load of what you expect. We're changing reliance on code. So what does this mean for reliance or over reliance?

53:22What does this mean for learning? What does this mean for novices versus experts? How do we measure productivity, right? There are a handful of us that are having these discussions on what does this mean and how do we communicate it thoughtfully? Again, we really need to have these kind of holistic balance metrics because if we, if it's an oversimplification, we really risk losing the force for the trace, right? But it's also like super interesting and super compelling, I think. How can we think about, you know, learning or like onboarding to new code bases or new languages for folks who already know computational learning?

53:57I think it's also very different for folks who are just learning programming languages and don't already know things like computational thinking. If someone was excited to kind of go down this road of, we're going to focus on developer experience, we're going to focus on helping your engineers be more productive. What are the next step or two that they should take in your opinion? Just broadly knowing that you don't know any specifics about the say the company that's thinking about this right now. I think if you're like walking away from this podcast and you're like, you know, I'm already working on this or I think this is a thing that's happening, right?

54:31I would say just like go check your work basically, right? Like, has this been written down? Is there a clearly defined challenge problem, something? Start there. Absolutely, right? Because that is going to be the thing that produces confusion. The best, right? Absolutely. And then see if there's any data. And data can be very loosely defined, right? Is there any signal that is related to the problem? Like, I'd start there and you can do that. You can do that a week. You can hunt something down. Sounds like something you could do in a day. Yeah. Well, depending, depending on how scattered things are.

55:16Are there any companies that you look at as good models of they do this really well? I think Google does this incredibly well. And sometimes I hesitate to mention Google because they're you know, some people like, well, we can't be Google and we aren't really advanced. But the thing I love about Google's approach is that they've really taken kind of this measurement phase approach to things, right? Even when they roll it out in new places. They're very systematic in how they measure things. They have incredible telemetry and tooling and instrumentation. And they continue to invest time in developer experience surveys and they try and do like them.

55:54And one thing that I also love being able to point out here is if there is ever a disagreement between the surveys and the instrumentation, which is incredibly advanced almost every time, every time that I've ever heard of the surveys are correct and not the instrumentation. Amazing. I have just a couple more questions unrelated to this topic. Is there anything else that you thought you think would be useful to share or leave people with around this general space? I would say that like thinking about what it is you want to do is always important, right? Like getting crisp, the ability to communicate clearly is always one of the best things.

56:36One of, I think, one of my superpowers and one of the things that I've been working with my teams on doing and kind of teaching them is, and one of the things that's really leveled up our work in general is making a work incredibly accessible and accessible, not necessarily like the accessibility definition of the word, but making it very easy to understand what you're doing for your key audiences. And so thinking about doing that for anything that anyone who's listening for all of your work is super important, right? So who is it that your audience is? What's their role? What words resonate with them?

57:14And then always being able to translate your work into a few sentences or a paragraph or left or less. I love it. A lot of the listeners of this podcast are product managers and this is so core to the work of a PM. So I think this is perfect. Speaking really directly to a lot of the listeners. Okay, so just a couple more questions. Before this podcast asks you a few questions, including just like, what are people asking you for advice often around? And are there any other frameworks that you find are useful? And so there's a couple of things I just wanted to try and see if there's something interesting there.

57:46The first is, you have this framework that you call the four box framework. I'm curious what that is and what it's all about. Yes, I love this four box framework. I've used it for years. I actually pulled it out first when I was a professor and I still to this day get LinkedIn messages from my students saying that it's like the most useful thing they've ever used. So here's what it is. I literally pull this out on napkins at bars at conferences to this day. So here we go. Draw four boxes on a piece of paper, two on the top, two on the bottom. So there'll be kind of aligned. The first two to the left to them write the word words and below them write the word data.

58:34And then between the two on the top drawn arrow between them. So it'll say words box arrow box. Right. That makes sense. Then on the bottom it'll say data box arrow box. Okay. So on the top half, this is where if you want to think about measuring something or testing something, you have to start with words. So as an example, let's just say I think that customer satisfaction gets us more money or customer satisfaction gets us return customers. Let's do customer satisfaction. So the first box you'll put customer satisfaction inside the box. You'll put return customers in the second box. Now always start with words.

59:20Do not start with data. Always start with words. And then you'll go around to a couple people, stakeholders, managers, others, and you'll say, do you agree with, do you agree with this? Is this actually what we're doing? It can turn into a sentence. And then in the boxes below it, this is your data. How are we going to measure customer satisfaction? It could be a survey. And so like this is where you'll go and you'll say, like what data points do we have that could proxy for? What could be our data points for customer satisfaction? And this is where it gets tricky because you could say, well customer satisfaction could be return customers.

59:57But we think it leads to return customers. So we can't use that here. But return customers could be. So like that's where you kind of like roll this out. So how's putting measure customer satisfaction? I made this hard on myself. Like a C sat score or C sat. Yeah. C sat NPS. We could say the amount of money that they spent. It's a stretch. Okay. Now return customers. Let's go to the next box. How are we going to measure return customers? Depending on our context, let's say that this is an online business. We could say that it's return customers as measured through the website. We could say that it's return customers.

1:00:39We could ask them, right? Maybe we have a follow -up survey. Return customers. Maybe we're going to do a stretch here. Maybe we say it's a referral link. This helps us get super clear on what it is we're going to measure. Now the reason I like this is because if some of our data, now this data analysis, we'll just do correlations here, right? If we applaud two and over time, that's fine. You can hand this to like a data scientist. You can hand this to someone. You can say, what data do we have? Let's go run this. If something here falls apart, now you can point to the data boxes and we can get mad about the things in the data boxes.

1:01:18We can say, what's wrong? Is the data poor quality? Are we missing data? Was this a bad proxy? Proxy stands for something else. Was this ridiculous? One of the things I made up, it was just a bad idea. Instead of getting mad at Lenny for his really stupid idea or getting mad at Nicole because this was a really bad idea, we can say, this was problematic. What's wrong here? Or we can go back up to the words at the top and we can say, this is not actually something that is probably going to hold. This is not something we want to test right now or this is something instead and it makes things incredibly clear.

1:01:58It helps you communicate what it is you want to do fairly quickly. I love it. Here's my phone. Check it out. It's ugly. I'll zoom in right now. I will say advanced mode. You can start with the same four box framework and you can say, what data do we have available? What do we think the relationships are? Then you have to go back up to words and then say, for these data points, we think that they represent something and we think this is the relationship between them. What do they represent? If I turn this into a sentence, what do they represent? Then you want to double check because it's spurious correlations.

1:02:42One of my favorite websites instead of charts. You want to go chat with someone, interview, make sure things are actually right. The challenge is I will see people run every correlation they could think of but they haven't turned it into a word or a sentence that you could communicate to someone else. They don't do the check and they don't do that before one before running the correlations. Two, if it's there, all of our data is so interrelated that we quite often will find spurious correlations but it can be really helpful just to have that laid out, even if it's just not a post it, to say, what are the things I expect to see?

1:03:23What is this actually testing? What relationship do I suspect is there? There's actually a newsletter guest post on how to do a correlation analysis and regression analysis. That's so great. Plug -in play all kinds of makes it easy for you. Take away from this, this is an awesome framework especially for thinking about a hypothesis you may have. In this case, it's like customer satisfaction is going to lead to more return customers. Here's how we're going to measure it and then you basically run the test and see that it's true and if it's not, maybe you need to pick different metrics, maybe you need to pick a different conclusion.

1:03:59Within the Dora framework, we would say if we want to improve our speed and stability, we think improving build time would help and then how would I measure build time? These are the data points that have available to us. Yeah, to circle back. I love it. It's all connected. Last question, ask you what advice people often ask you for and you said that it's around making decisions and I'm curious what advice do you give people about making decisions? Yes. This one comes up in business but also comes up personally and among my mentees. Many times it starts with being very critical about your objectives and definitions but then it comes down to really clearly defining what your criteria is, what's important and then among that criteria what's most important.

1:04:52Some of my friends know I have a decision making spreadsheet that I have shared out with and full of friends. Should you take a job? Where should you move? What are the different things? Really useful. It is. It's funny though because what's interesting is many times I will like, I'll share it with someone and I've got a couple that are just funny, right? But walking through the spreadsheet is often all you need to do in order to know what the decision is and by that I mean, so we walk through the decision, I had one where I was like where should I move next or like what job should I take? When I started Dora, I did this starting Dora.

1:05:32I thought was my lowest once I walked through the spreadsheet. It became my high. So what you do is you outline of all of your options, what do you want to do? And then you say what are the criteria that are important to me? So if it's for a job, is it something like total comp, cash money, prestige, team, job predictability, work like balance, identify the criteria that are most important to you. Now it's really interesting because sometimes I will only get that far when I'm working with someone I'm mentoring or coaching and they will say I know what my answer is. We don't even get to the next step.

1:06:16But just identifying the criteria that are important, is it? Now when I was thinking about where I wanted to move next, it was proximity to an airport, the relative texting, the food scene, that was real high for me, you know, a handful of things, that was important. Now, the next thing I do is for each criteria, what's their relative weight? What's their importance? And I make it add up 200%. And then I, this is the easy part, right? You just put it in a little spreadsheet and then I give everything to score and I just multiply it out. Now, this is where I'm data informed and I'm not data driven.

1:06:56There have been times I make a decision where the whole flip of coin and whatever it's, wherever it lands on, what your reaction is tells you what it should actually be. There have been times where like I multiply it out and then I'll actually like fudge the numbers to get what I want, but it's still slightly off. That's where your data informed. Same thing in business, there are many times where you actually run the numbers and it'll give you a class or a category of things and then you choose. Now, this is where, you know, one of my favorite quotes I heard somewhere about strategy comes into play, right?

1:07:34And that's that the key to having a good strategy is knowing what not to do and the key to executing a good strategy is actually not doing it. So you can have many options, right? As a leader and as an executive, we have many options and we only fund some of them. If you fund everything, things are going to fail. So being able to think through and identify what your criteria are, identifying that criteria, what's your selection criteria, what's your evaluative criteria, ranking them and then deciding what the cutoff is, is important. You can't fund everything. You don't get to pick everything.

1:08:18Amazing. I love the spreadsheet idea. I've made versions of it, but it's always, I think like you said a lot of times that exercise is just tell you what you already think and just gives you like, yeah, all right, you're right. You probably should just do that thing. You already thought you should do. Yep. Have you thought about making a public template of this spreadsheet, even though it is simple, I bet it would be really helpful to people. I have and this actually might be a good forcing function, maybe. Okay. Awesome. So if, if you do it, I'll put in the show notes. It'll probably be near the bottom at the end of the episode, but that'd be awesome.

1:08:50Is there anything else that you want to share before we get to our very exciting lightning round? No, I think that's it. Well, welcome to our very exciting lightning round. I've got six questions for you. Are you ready? Absolutely. All right. First question, what are two or three books that you've recommended most to other people? We actually had the perfect segue because the book I've recommended absolutely the most is called Good Strategy, Bad Strategy, by Richard Rumeilt. Another one is Design Your Life by Bill Burnett and Dave Evans. And the last one is probably Enders Game. Of course, it's got a card.

1:09:26No comment right now on some of his political commentary, but I used to have extra copies in my office when I was a professor and I would just hand it out to my students. It's a fun, like, just easy nonsense read, but I absolutely love it. Such a good pick. Haven't read it in a long time. And are they making a show that at all? That'd be that'd be some. They made a movie and I was afraid I wasn't going to like it, so I just didn't read it because I wanted to ruin the book. But at least Harrison Ford was in it. Okay. I'm not going to check it out. They're making a movie of three -body problem. I don't know if you read that, but that is.

1:10:02I'm really excited for that. It's on my list. Oh man. Best sci -fi ever. Next question. Actually, very correlated. What is a favorite recent movie or TV show? I've been going through like some real just easy fun watches lately. I'm rewatching suits again, but Ted Lasso is a favorite. And I just tore through Never Have I Ever, which is fun because John McEnroe narrates it, which is hilarious. John McEnroe, the tennis player? Yeah, it's a riot. Yeah, it's so funny. I love it. Next question. What's a favorite interview question that you like to ask people when you're interviewing them? I love questions that I can kind of spin around hard decisions that people have had to make and how they made them.

1:10:49I love hearing their thought process. And I get a little nervous when people just like yellow and shoot from the hip constantly. So what is it you look for there that gives you a sense that they're someone you may want to hire, work with? I just like hearing if they have some sort of process, right? If they have some kind of decision -making process, if they have criteria, if they have, how do they do evaluation? What is a favorite product you've recently discovered that you love? I have a big one, a little one. My big one is probably sleep eight. So I live in Arizona. It gets hot here sometimes.

1:11:21Oh, eight sleep? Yeah, eight sleep. Yeah, the other way around. Yeah, so that was fun because it makes the bed cold and also gives me some data, which is probably like a little bit off it in the approximations fun. And then Korean face masks, they're just fun. Yeah, you can get some pretty good ones for like just a couple dollars and that's always fun. So care. First mention of that one of Korean face masks. Right. Listen, everyone get on board. I just did the TikTok. There's a filter now where you could see how you look when you age. And I'm not happy with how it turned out. And so I'm my look into this.

1:12:00I had some basil cell cancer on my forehead a few years ago. And so I am much more careful with my skin. And you can get like one of my favorites is Cossar X. You can get 10 for like $15. So it's fun to just like chill at the end of the day with a good face mask. I was going to ask you for a specific pick. And so we got one. Yeah, amazing. This next question I ask everyone. And it's especially appropriate to you. But I don't know if you'll have an answer. What's something relatively minor you've changed in your product development process that has had a big impact on your team's ability to execute.

1:12:33And I feel like you have a big perspective on this. So I'm curious what you have as an answer. I think I alluded to this earlier. I would say that it's helping everyone. So I've done this before. But I think it's helping everyone to ask who's our audience and how we'll be sure this now. And it's sort of interesting because right now I'm wearing two hats. One is at MSR Microsoft Research. We need very ambitious research. Right. So like H2H3. Maybe what is H3? The third half? Oh, Horizon 3. So it's supposed to be like five to ten years out, which right now is like who even knows, right? Because like in computers.

1:13:16Yeah, AI has like completely upended how we kind of think of horizons. And so when we're thinking really ambitiously and very, very, very forward looking, what's our check in? How do we evaluate this? And then how can we easily communicate it to our core audience? And so here, who's our audience? And how do we bring the FAR near? And then for the other hat I'm wearing, I'm working with Octo kind of across all of Microsoft to take a data -informed approach to really improve and uplevel our central developer infrastructure. And so as we're thinking very, very tactically, what is our long -term vision?

1:13:58And so there it's who's our audience? And how do we bring the near FAR? I love that final question. What is one tactical piece of advice that listeners can do this week to help them prove their developer productivity or developer experience and move it in the right direction? You know, if you walk away from this podcast right now, you could take a look at what's happening in your org today. Is it written down? Is it clear? Do you have any existing data and efforts? And if not, go find a handful of developers and ask them how they feel about their work tools and their work process and what the biggest barriers to their productivity are.

1:14:40Also pick up a copy of Accelerate on Amazon or your local retail establishment. Nicole, this was amazing. I think we're going to help a lot of companies move faster, have better and happier engineers, which is going to create infinite value in the world. Thank you so much for being here. Two final questions. Working folks find you online if they want to reach out. And how can listeners be useful to you? I'm on Twitter and Blue Sky. And Nicole FV. And my website is NicoleFV .com. And my contact information is there. And as we mentioned previously, I'm working on a new project and a new book digging into exactly these ideas.

1:15:20How can we measure better? How can we improve? And what does that measurement process look like both for one time really quick unofficial measurement pieces? And if we want to do very formal longer term measurement pieces. So if anyone is interested in that or has any success stories they'd love to share, I would love to hear more about it. So please reach out and share. I'd love to hear more. Awesome. Nicole, thank you again for being here. And thank you, Lenny. Bye everyone. Thank you so much for listening. If you found this valuable, you can subscribe to the show on Apple Podcasts Spotify or your favorite podcast app.

1:16:00Also, please consider giving us a rating or leaving a review as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show at Lenny'spodcast .com. See you in the next episode.