Managing Developer Feedback Effectively | Atlassian’s Andrew Boyagi

0:07Welcome to Dev Interrupted. I'm your host Ben Lloyd Pearson and joining me today is Dan Lyons Linear B's COO. Today, we've got a great conversation with Andrew Bajocki. He's the head of DevOps Evangelism at Atlassian. And this conversation is actually the very last conversation that will feature our host, Connor Bronson. He's since moved on to a new opportunity. If you want to learn more about that change and more of the changes that we have planned for Dev Interrupted, I encourage you to go back to the last season and listen to our holiday episode where we cover some of the changes that are happening here.

0:42In a moment, we'll play this conversation between Connor and Andrew. But first, Dan, I wanted to bring you in to break down some of the topics that they discussed. One of the things that Andrew said in his interview was that there's this misalignment between engineers and their managers on how to improve DevX. So in all the conversations that you've had with engineering leaders out there, do you sense this misalignment? Actually, that's pretty funny. I do not sense the misalignment, but I'll tell you why. And I think I know if you go out and let's say you do a survey and you ask a bunch of questions to many developers and many managers, you're going to get a ton of different answers.

1:26The thing is, all they're saying is actually the exact same thing, but from different perspectives is the best way to put it. Now, on the flip side, with our customers, with Linear B, the engineering managers and developers that I'm interacting with, they're all utilizing data. They're using data to measure productivity and then to improve productivity. And when you utilize data to do so, it's actually a universal language. There's no misunderstanding or misalignment. So for example, if you're thinking about something like, okay, you know, hey, we want to improve our developer experience or we want to get more productive.

2:08Actually, managers and developers want the same things. They want to streamline their code to production. They want to ensure that they have the right investment balance in technical debt versus new feature delivery. They want to deliver their projects in new value on time. And they both want clear expectations of what it takes to grow a career. They actually want the same thing. Now, they might verbally express that differently. That's the trouble you'll get with like a survey. But no, actually, like the customers that we're working with, since we're more, I would say, like data centric, they're actually pretty aligned on what they're looking for.

2:50Yeah, I mean, it's a great way to put it. Everyone just wants to build new awesome things, but to get there, you know, you got to make sure that you're investing into DevEx and to keeping the lights on and to maintenance because, you know, that keeps your developers happy who in turn build better products. One thing that really stuck out to me was Andrew mentioned that when Atlassian surveyed their developers, they got so much feedback that it could take him, he said, like up to like two years to dig through it all, which is a little wild. And he mentioned that a lot of that data was also process related.

3:20So how is an engineering manager supposed to manage all of this feedback from their developers? I mean, again, here's the thing. Like if you're sending out a survey, it's a very like academic, let's say academic questioning of what's happening with your personal experience. And you go and do that for a hundred. It's not even developers. Let's just say people. You're going to get a lot of information. Yeah, you mentioned there would take them two years to sort through the data to find actually what is the real problem and what should we do next, let's say, to improve productivity. Now, when it comes down to the engineering managers and maybe you start talking about things like feedback loops, there's actually a lot of good feedback loops, I think, already built into the engineering practice and processes.

4:11Like, for example, most teams have an iteration retrospective. You're getting a feedback loop every two weeks or three weeks maximum. Most engineering leaders or managers have one-on-ones with their developers. You know, for us, we usually recommend once a week. I'm getting feedback all the time. I think the more important question is not around managing this insane amount of feedback, but more so getting everyone onto the same page. Again, I'm going to be more data-centric and a data advocate. Let's look at the data. Let's see what the data tells us about what's causing bottlenecks, what's causing planning accuracy problems, what's causing investment problems into DevEx.

4:53And let's see, does that relate to the feedback? Most of the time you'll get the answer of, okay, yeah, it's actually the data is describing it really well. Now it's more so a loop on what do we do about it? I think that's like on the productivity side, at least the customers that we're with, you can kind of understand in a few minutes after using Linear B where the bottlenecks are, the bigger question is, okay, what do we do now? Yeah. We recommend to a lot of organizations starting with Dora metrics, you know, and I think that's one of the things I love most about it is they have this saying, like, get better at getting better.

5:25You know, it's more about teams working together to get better and being empowered to be better together than it is about whether or not you have enough feedback from your lowest level up to the highest level. So diving a little bit deeper into these feedback loops. So, you know, you think about feedback loops as a way to like continuously improve environments, but that can be really hard to scale, right? Like some organizations like Atlassian look at surveys as a solution to scale that. But how would you recommend that engineering leaders scale like these types of feedback loops? You got to use data.

6:03That's the thing. So you're going to have these feedback loops from developers, each team, each manager, each business unit, each organization. And the only way I believe that you can get a clear picture of that is be able to see that data. Again, I'll just go over the ones that I think are most important. You have efficiency and quality related data. That's more so what you were saying on the door metric side. You have value delivery data, which a lot of people equate to productivity. What value am I producing for the business? You can see that within project delivery. You have resource alignment data, which is kind of that feedback loop of, hey, we're not getting enough time focused on technical debt.

6:47Well, you should know what percentage you're investing into technical debt and have an agreement with the business on what it should be. Everyone already knows that you're not investing enough into technical debt. That's been around since like the beginning of time, since I've been a developer and developers have been voicing that in feedback loops throughout surveys or one-on-ones or whatever, it doesn't do anything. The difference is, can you actually quantify it and then go to the business and say, can we make an agreement with you to say at least 17 to 20 % of our time is on technical debt?

7:21That's a good agreement. And then I would say the last thing on the feedback loop, again, it's on that career development. Are you looking at data? Is it very transparent of what it takes to grow your career from a junior developer to a developer, to a senior developer, to a principal engineer? And this is the one that actually, maybe I'll just take a minute to focus on a little bit more as it might even relate back to developer experience. If I'm progressing my career in a fair way, I'm going to be having a good, it's one of the main inputs, I think, to having a good developer experience. Now, some of the complaints that I've gotten from developers at, you know, many size organizations is there's not a clear way for me to understand how to progress.

8:08And developers that are a little bit more comfortable or articulate in how they can say how productive they were seem to be advancing. And me as a developer, I'm trying to let my works, you know, speak. Maybe I'm a little more shy. So when you have like a developer coaching program that's data centric, again, I think that's kind of a fair way to do so. So I would just say, yeah, there's a lot of different feedback loops that are already baked into the development process. We have those today. It's more so about can we get on the same page about the data to actually improve? Yeah, I think the way you describe it is a great job at drawing it from, you know, high level resourcing decisions, like deciding what things you strategically want to invest in as an organization.

8:54And then saying to your developers, then, you know, we're going to establish working agreements around these priorities so that they always know, like, whenever they're working on something, does it fit within that strategic priority framework that you're giving them? It always has to relate back. I mean, imagine going to a CEO and saying, we did a developer survey and the developers say we have a technical debt problem. It's like, dude, I already knew that. We've been talking about that for the last 10 years or eight years. That's not new information. A different way to do it is you go to the business.

9:31I'll use the CEO as an example and say, hey, currently we have an 8 % effort, investment effort in technical debt. I would like to move that percentage effort to 17%. And because of that, I can still give you a 50 % new value delivery. But what it's actually going to do is decrease our change failure rate, which is very high right now. And it's going to make our customers happier. That's a better conversation. Exactly. To tie this to productivity a little more. So we've already talked a little bit about Dora metrics and how, you know, I think one common misstep a lot of organizations make is they try to use Dora metrics to compare across teams, you know, when it really should be more focused on like a team saying we're going to work together to solve our priorities.

10:23So how do you view something like Dora metrics fitting into the overall productivity picture? Because I personally hear constantly that Dora doesn't measure productivity. which I think is probably right. So here's what I would say. First of all, with any metric that you use, it's probably highly risky to compare that on a per team basis. So whether it's like the Dora metrics or amount of story points delivered, anything that you're going to compare on a team by team basis, I think there's like a lot of nuance and context on a per team basis that needs to be applied. So for us, when we're working with our customer community, We don't do that comparison.

11:03Now for your second question of how does it relate to productivity, I wouldn't say that Dora metrics are an exact equivalent to productivity. What I will say is that they're an input into developer experience that help with productivity. So for example, having a streamlined developer development process where I can easily go from coding to deployment to production, that's a very, very important thing for efficiency. And efficiency leads to productivity. It's one of the leading indicators. At the same time, if I wanna be highly productive, if I'm looking at change failure rate or mean time to restore, I don't wanna be pulled off of my value facing work all the time because there's an issue in production.

11:51That's what a change failure rate metric is essentially saying. So I would say it's an input into productivity And it's certainly, if you're trying to improve developer experience, it's something that you want to measure and then improve. So I guess like the last thing I would say on the team by team basis, it's not where you are, it's where you start and you're measuring and baselining. And now if you even make a 5 % improvement each six months or each year, your developers are going to say, wow, I actually do feel that. I see that we're focused on it and we're getting there. Awesome. Well, thanks for sharing.

12:26all of your great insight with our audience today, Dan. We'll return after the break for our conversation with Andrew Boyagi from Atlassian.

12:37On January 29th, join Linear B and Refactoring for a workshop on starting and scaling a software engineering intelligence practice for your organization. With a data-driven approach based on qualitative survey responses from engineering leaders like you, and quantitative metrics from the experts at Linear B, this workshop gets down to brass tacks with actionable examples. Check the show notes to RSVP your spot for January 29th, and we'll see you there. I'm your host, Conor Bronsden, and today I'm thrilled to be joined by Andrew Boyagi, Atlassian's head of DevOps Evangelism. We're going to dive deep into the current state of developer experience, explore the challenges software teams are facing today.

13:21and discuss what needs to change for developers to thrive in the evolving landscape. Welcome to the show, Andrew. Thanks, Connor. Thanks for having me. Andrew, let's start with the big picture. Atlassian's State of Developer Experience report has highlighted some key challenges that software teams are facing today. Could you walk us through the main findings and what they mean for the industry? Well, thank Connor. So there are three main findings that I want to share with you today. The first one is that there's a huge misalignment between developers and their leaders when it comes to developer experience.

13:51And this is highlighted a few different times through the report. One of the key findings is that the top issues that are impacting developer experience are engineers and their leaders have wildly different views on what they are. So if you ask managers, they say things like understaffing, new technology and build processes are significant impacts to their team's developer experience. But when you ask engineers, they say tech debt, core documentation and build processes are majorly impacting their developer experience. So there's a pretty big misalignment about what's affecting developer experience, but they are actually aligned on one thing.

14:32And that is that developer experience is very important. However, even though they're aligned on that, the reasons for it being important are different. So from an engineering leader's perspective, it's important. They say it's important in terms of recruitment and retention of engineers to their organizations. And for engineers, they consider it when they are making a decision whether to stay or to look for a new job. So developer experience is important, even if it's for different reasons. The second key finding is understanding and improving developer experience actually pays dividends. Our data finds that there's a correlation between time spent on understanding and improving developer experience correlates with better developer efficiency.

15:21There's a lot of companies out there looking to try and improve developer productivity. And this shows that investing in developer experience is a good investment in terms of helping developers be more efficient with their work. And then the third key finding that I want to share is that engineers and their leaders have wildly different views on the impact of AI. So again, if you ask leaders, they say that AI is the most effective way to improve developer productivity. But when you ask engineers, two-thirds of developers are saying that AI isn't helping them be more productive just yet. So this misalignment, there's obviously a lot of this misalignment between engineers and their leaders.

16:07but if you look at what causes or what the impact of this misalignment could be a lot of companies are spending some money and time trying to improve developer experience who is driving that within a company it's usually the leaders so if they don't understand what's actually impacting a positive developer experience it could lead to them investing in the wrong things and every company I speak with they have obviously some limited budget that they've got to spend on improving developer experience. And so you want to make sure that you're spending that money in the right area so that you're actually moving the needle for your engineers.

16:46I heard a bunch of great insights there. And so, Andrew, I really appreciate you kind of walking us through some of the key things that Atlassian found in this report. A couple I want to call out here. One, that AI impact difference. We're hearing that everywhere, where leaders are very much looking ahead to these efficiency gains or they see small efficiency gains that stack up across the board. But for individuals, it's a lot harder to say, oh, this has been massively impactful for me. Maybe that's going to change as we shift more to like agentic AI experiences versus simply like Socratically talking with a, you know, Claude or something to help you with your coding.

17:20But there's definitely huge opportunities for improvement there. And to your point, it also speaks to the fact that many leaders today are looking at different things. I am happy to hear at least they were aligned and everyone agrees build processes are a problem. great like that we got we got one right but if if you are as an engineering leader are not either surveying your devs or ideally both uh leveraging qualitative but also quantitative signals to understand what's happening with your development processes what's happening through developers it feels like you're missing out um because you're kind of assuming what your devs may find to be a problem without actually gathering data on it yeah absolutely and i think the ai example that you raised this in is a really good one.

18:05If you just look at AI and the promise that a lot of these products make, of course, you'd look at it and say, no brainer, I'm going to give this to my devs and it's going to improve experience and help them be more effective or efficient. But is it actually solving a problem that you have? And so it really depends. Maybe it is, maybe it isn't, but AI is a tool and just giving a new tool to developers, regardless of what tool that is doesn't mean it's going to help them. And without a good understanding of the problems they're having, there's no sense in just giving them more tools to try and improve things.

18:41Andrew, it's a lot easier for me, though, if I just give them more tools. Tools are shiny, they're fun, it's exciting. It's a lot harder to do that work of understanding the problem and aligning on the right solution and then maybe getting a tool or maybe fixing a process or both. And I love that you're also thinking about this from a, you use the words developer efficiency standpoint, but developer productivity as well. Because I really think about developer experience and developer productivity as like two sides of the same coin, where it's like, if you can improve the day-to-day experience for developers in the ways that they want, your devs will typically deliver more.

19:17Your devs are great problem solvers. Your teams are there to solve problems. It's exciting. It's fun. Help them do that more efficiently. and you're going to have that productivity game that you want. So I love that Atlassian is thinking about this connection here because too often I think the industry will kind of hyper-focus on one element or the other. And maybe if you're a scale-up, you're thinking, oh, I need dev prod. I really need productivity. I need to innovate. I need to push features faster. And maybe if you're an enterprise, you're like, oh, I have to retain people. But really, these problems are so holistically combined in so many different ways.

19:55Yeah, absolutely. I mean, if you look at what goes into developer productivity, it's really two things. You've got a great developer experience and a positive engineering culture. And they're the real two inputs into developer productivity. If you look at developer experience, it's around removing friction. It's around how do engineers feel about the tools and the frameworks that they have to deliver software in your organization. If you look at engineering culture, it's about how things are done, the decision-making process, and things like that. Those are the two key inputs to productivity. And so just saying we want to improve developer productivity is like saying nothing at all, really.

20:38It's really one of those two other things that you need to affect to improve that efficiency for developers. And I like that you're talking about it from an efficiency angle as well. I think there's a lot of folks that when they talk about measuring developer productivity, I'll use the McKinsey study that came out last year now as an example, where they tend to hyper focus on maybe some metrics that aren't the best, the best measures of actual efficiency for the team, actual improvement. and measured productivity has always been challenging, whether in software development or elsewhere, especially with the pressures of delivering high quality software at speed and the fact that devs are spending less time coding and more time on other responsibilities due to the shift left movement, rise of microservices, AI, all these different things are happening.

21:25How do you think companies should approach this given the complexities of modern development? How should they be measuring productivity? You know, I get asked the same question three or four times a week, but different companies I speak with, They jump straight to it. How should we measure developer productivity? I want to know, why do we want to measure productivity? I feel like we're more obsessed with measuring productivity than we are actually trying to help developers be more productive. And how do we measure productivity? You can't measure productivity. There's been thousands of academics who have researched knowledge worker productivity over years.

22:03and the closest they've come is to say that high employee satisfaction leads to better productivity but they have not come up with a measure for productivity. So I think it's actually the wrong question to ask. The right question is how do we help our developers be more productive? And I get asked that question far less than I get asked how do we measure productivity? And so if you ask me how do we help developers be more productive, the first tip is to actually understand what's stopping them from being productive. And from the evidence we have in our survey, you can see that not a lot of people are asking that question of their developers.

22:43A few years ago, I spent two full weeks in an organization just asking engineers, how could we improve the way software is delivered in this company. After two weeks, I got so much information from developers. It was like a two-year backlog. It would take us longer than two years to sort out the things that they gave us. And at that time, I was expecting a lot of answers around tooling, around build processes, some of the things that we've already spoken about. But there was also an overwhelming amount of feedback around the processes that the company used to track and to deliver software. And that was a real surprise to me.

23:23And it kind of completely changed the angle we were taking around improving things for our developers. So the first step I always recommend is ask your developers, speak to them, just ask them simple question, what can we do to improve the way we deliver software? And you'll get so much information, so much rich information that you'll have more work than you can possibly handle. Do you think part of the problem here is that too many companies are focusing on individualized productivity versus like broad team efficiency and where there are problems and workflows? Or do you think it's something deeper than that?

24:02What I see from the company that I speak with is if you look at what impacts developer productivity and certainly over the last 10 years as complexity has grown, it's really around the fact that developers need to do more. They need to know more. They're under a lot more pressure these days to deliver high quality software at speed, like you said. And really, there's two types of complexity that they deal with. There's intrinsic complexity, which is complexity associated with being a developer, which is things like now we need to know about dev and ops and cloud and AI and machine learning and all the things, testing, you name it.

24:42Developers need to know everything now. that's intrinsic and that's intrinsic to the role. So whichever company you work in, you're going to face that challenge. And then you have extrinsic or organizational complexity. And this is the stuff that's specific to the company that you work in. It could be things like regulations I need to comply with because of the industry. It could be processes. It could be governance things that a company has implemented. And so we have all these two levels of complexity that developers need to handle. I often see companies trying to tackle one of those. They either try and abstract a lot of complexity, intrinsic complexity.

25:21So you'll see platforms that abstract the need to know about cloud and testing and all these other things. Or you'll see companies trying to simplify their own environment by simplifying processes and things like that. Really, it's a holistic approach is the one that wins. So it's around complexity. It's around understanding how much complexity your developers face and how you can make their role in their lives easier with dealing with that. So as I start to implement that efficiency program, as I have talked to my devs, maybe I've done a survey and have begun to understand some of the pain points.

26:00Maybe I've realized some of the pain points aren't ones that I thought they were. They're different ones. We've started to implement changes. is how do you think teams should be tracking the success of these initiatives? Is it purely on the qualitative side or do you think there are quantitative measures that should be leveraged as well? Yeah, there's both. So if you look at developer experience, let's say that you've spoken to engineers and you find out there's three things, three major problems that's impacting most of the engineers in your organization. The first step would be to measure that.