Using HEAT Metrics to Bring Purpose to Platforms | Simone Casciaroli

0:00Imagine again, if you start a company, the first question you want to know is that, is anyone interested in my platform? Do they want to use it? You know, like you can't really spend months building something and then start looking for it used to start a company. I used to work in a company that built a big data platform and then they were going around and say, are you interested? Are you interested? It's a big investment. And if it doesn't work, you waste a lot of money. So adoption is probably the sort of thing you want to look at. And it sounds obvious because it's kind of like, of course, but we don't do it.

0:35Are you an engineering leader and tired of constantly being asked, when will it be ready? Stay one step ahead with Linear B's project delivery feature. Our powerful dashboard lets you visualize key milestones, forecast delivery accurately, and align stakeholders effortlessly. With Linear B's project delivery, you can confidently showcase your planning, prioritize work effectively, and even make data-driven cases for additional headcount. Say goodbye to delays and missed milestones. Sign up for Linear Beat today and answer when it will be ready before anyone even asks.

1:35happy back on the show. There was an awesome opportunity to kind of see one of your talks there actually. And I was losing my voice because we were talking to so many people. You were like doing these amazing speeches and it was really a great event. And you really caught my attention based on the talk that you gave titled, are we building engineering platforms using the right metrics? And obviously as anyone who listens to the show knows, we are very passionate about understanding engineering here at Dev Interrupted. Can you introduce this talk to our audience? Yeah, of course. So, this still comes from a number of passion of mine.

2:16So, on one end, I truly believe in the power of platforms, in enabling teams and creating an environment where basically, like, you can do, you know, your team can do their best. At the same time, I really believe in driving engineering through, like, a metrics approach. it gives so much more autonomy to teams. You know, instead of saying something like, you should implement X and Y, Z, you focus on the outcome. You focus on, you know, what you want them to achieve. I come from a business product background as well. I run two startups. I work with a number of startups as well. And for me, it's very clear that, you know, a platform team should basically kind of treat the team that are using the platform.

2:59and trying to use the same umbrella of knowledge and tools that I normally use if I had to run a company that had the specific sort of target tokens, I do the same for platform. I literally do the same product tools that I would do if I had to run a SaaS and have a sort of target audience. It's just that I do internally for the internal teams. So that was the idea. Yeah, I think platform as a term has been abused massively. And as a consequence, I really see some places where they use metrics I don't think is right. And if it was just like, you know, syntactically wrong, I would be okay with that.

3:41The thing is that I see big impact, big negative impact by not using the right metrics. So this is why I felt like it's definitely something I want to get out of my chest. And so that's the talk, I guess. What are some of those metrics that you feel are causing problems when they're being used? So this is going to be super controversial. I think, so let's talk about something that we're all familiar with that are, let's call it infrastructural or cloud platforms. This is something that a lot of companies have about. Now, in this space, as you know, this is probably a space that you follow very closely.

4:19then has been like a huge amount of progress recently we used to have ops teams that they would be responsible basically for you know deploying what the engineering team would be responsible for building and then at certain time we start to talk about devops obviously there are still a bit of confusion what devops is but uh the truth is that sometimes when we build platform teams, I think we treat the platform almost like an ops team. So we target them with metrics that are actually metrics for the whole company. I don't know. Like, are we sure that, you know, for example, the normal corporate is something like, you know, are we sure that Dora is a metrics for the platform?

5:01Are we sure that Dora is actually not, you know, for the teams? And then it trickles out to the platform. So these are where I see more struggle. or assuming that, I don't know, security. Security is another one. Improve the security standing of our company. I know potentially the platform team has all the keys to the AWS account or the Google account or the cloud account, but it doesn't mean that they are responsible. They are so responsible for security. Absolutely not. So if we take a product approach, probably you, those teams, should look at what metrics or what goals their customer have basically their internal teams.

5:41Then depending on that, they find their own metrics. I'm expecting that a lot of their customers, a lot of their teams are going to have metrics or outcome that are related to Dora. It could be that they want to speed up how long it takes to deploy its production. It could be some team have it, some team don't. It's kind of like maybe some team are releasing every couple of weeks and it's fine because, I don't know, it's a special product that they don't get to iterate quickly. There are other teams where actually, I don't know, changing like, you know, moving like 10%, reducing like time to production is key.

6:16So even that, you know, we should also build different pathway for different teams that have different needs potentially. So I guess this is where the problem I see. One side of the problem at least. The other side, no. The other side is because we focus so much on metrics that are looking mostly at, let's say, if we talk about platform, I say cloud platform, we mostly look at metrics that are related to the operational aspect. We forget that as a platform team, if the user experience or developer experience of our tool is poor, we're causing a huge amount of cost and pain to the other teams. Especially for platform teams that basically everybody uses.

7:03Let's imagine a team that has hundreds of teams. If something you could have done, like, I don't know, like in two hours instead of six, that is six, four hours of waste across 100 teams. If this is painful and it's blocking people and it frustrates people, that affects so much the company. Before I start working for the company I work right now, I work for a company called Babylon Health. And at the time, I was working with a data platform. So every time we were building functionality with that data platform, really the team morale who were dropping to zero, like kind of like it wasn't real. The reason was because the whole developer experience of using the tool was poor.

7:48They weren't doing anything wrong. It's just they weren't really measuring how their work was impacting us. You know, the only thing we could do was escalating the bad experience. So this is where I start asking questions like, well, okay, but how do you know if you're successful? And, you know, the usual metrics was like, well, you know, we're improving how long it takes for you to deploy these things to production. We're improving how long it takes you to, you know, measure a new data entry and so on. It was like, this is all great. But what about like, there is something else. These are, you know, how happy are we with this?

8:22How hard it is to onboard with your platform? What's this kind of thing? What's this kind of metrics? And I must admit, that was basically like the other part of the challenge that I foreseen. So this is where you kind of view platform as an internal product that needs to have differentiated goals, differentiated needs, because the customers you're serving are distinctly different. You know, they're the developers, they're the other teams. I remember at one point, you asked the audience at your Lead Dev New York talk, how many of the men worked on platform teams and not many raised their hands, but many more raised their hands when you asked if they had benefited from a platform team.

9:01And I'm sure others would have also raised their hands if we had asked if they'd been frustrated by a platform team. There's very clearly a shift happening right now where companies are paying more attention to these internal tooling pieces, these internal platform teams. How do you see that shift taking place in startups, whether at OnTour or elsewhere? It's a great question. First, even if you're the team that you're using, so sometimes you have, like even a team doesn't have the platform name, but you're using their tools, you're using their APIs, you basically integrate with them. They're basically building a product for you.

9:35So this is why it's so common. Now, again, as you know, platform as a name has exploded. So everybody wanted to have a platform in their name. It looked great. In fairness, it's mostly because it makes it look more like generic, multipurpose, something that they can achieve more. But the reason you can achieve more is because it's a product. It's not a project. It's not like, hey, we now ship the continuous integration server. It's in prod. Go ahead and use it. that's an iteration one. Then you need to direct with the teams and understand how to improve it. So by moving those teams into platform teams, putting that label, you have some responsibility, you know, with great power can raise responsibilities.

10:19And the responsibility is that, well, they need to start treating, you know, other teams like customers. So obviously it's very different depending on your skill. On to, it's not like a huge engineering team. You know, we are about 25, 30 engineers. So we have a tiny platform. And most of these focus on infrastructure kind of platform. But at the same time, you know, I was having like a meeting where I invited platform this morning, for example, to understand the pain points of our teams. Because again, if those are your customers, everywhere, every place where you can go to understand their pain points is a good data point for you.

10:57How can we help to make their life easier? How can we help to make them faster? That's the, that's at the end is the goal of every platform. You know, imagine if as a platform, often this thing comes with gated approach to platform. Let's assume a platform decides to do, we're going to improve security. So we're going to include like a gated approach to security. That's great. But what's the impact on the team? Are they going to make us faster or slower? If it's slower, there is something we can do because, you know, we want to make it easy to have a secure system. No. So these are, I think, the trend I'm saying.

11:35Like, obviously, the more people are using platform, the more hopefully they realize that platform, despite it, is not just a name. It comes with a number of responsibility and number of tools. Probably like another thing that is not very common, but it should be more just as a challenge, is to have someone who has some product thinking that is in the team. Very often, platform team don't have a product person. and mostly because you need like a product person that is comfortable with, you know, technology, knowledge. But at the same time, maybe the right approach would be to take an engineer and, you know, that is passionate about product and make you act as a product manager.

12:14It's very hard if everybody's wearing the engineering hat because you don't have like a friction. You know, everybody that is an engineer knows exactly what's easy and what's hard. and if you don't have someone that push and is you know like trusting for the customer then it's sometimes it's hard so it's saying well should we should we check you know with our customers how they're doing should we run a survey should we measure their behaviors these are all things that the whole thing that i think are going to come i see when i last year i wrote like a blog post on on this concept and there were many articles now i see many more articles on the topic of well let's treat platform as a product.

12:56I've been in a concept for a while because it was mentioned, I think, in Team Topology. I think it's a few years old. That's a book. But I think the whole concept of, okay, what does it mean to have a platform as a product? I think it's surprising more and more in the current year. So I'm expecting to see it much more commonly discussed. Yeah, we had the Team Topology's authors on in, I believe, season two, so a little about a year and a half ago. And really great conversation. But it's interesting to see how much more I'm hearing their book be cited now a couple years on after publishing, because it's very clearly become popularized in the public consciousness.

13:32And engineering leaders like yourself have read it and are saying, oh, like, how can we apply this? And one concept that comes up a lot, whether it's through their work or through other conversations with engineering leaders, is this problem that you're alluding to around. are we treating the challenges we're trying to solve as something where, hey, output is our goal or the outcome is the goal? Are we trying to actually solve the customer problem? And it's kind of an internal struggle for a lot of engineering teams, but particularly for platform teams, where to your point, it can be easy to kind of lose track of who your internal customer is or to think, oh, I understand the problem here.

14:10Let me just go solve it. How do you think about that challenge? It's tricky. I'll be honest, it's tricky. It's a great question, but it's tricky. The reason is it's much easier to deliver to an outcome, sorry, to an output than to an outcome. Because kind of like the output is kind of like, you know, it's still hard, but it's kind of like, well, let's release that feature by X, X meaning like a range of time that you can deliver. It's much harder to say, let's change your behavior less because normally an outcome is often related to someone else's behavior or some internal measurement of, you know, stability, security, and so on.

14:47So that's much, much harder. And that's the challenge. The challenge is that mostly we require a bit of a mentorship, I guess. I'm saying, okay, if you're optimizing for output, we're going to optimize to get something out as efficiently as possible. Now, imagine if what we release as efficiently as possible is a very long release. that, you know, is very efficient. We're releasing two bonds, but then it doesn't do anything to the outcome. I mean, what's the point? So if instead you give it to the team, the tool to say, like, look, you know, we really care about the output. Like what we care is to move the outcome.

15:31The team will self-organize, will organize themselves to start to iterate until they reach the outcome. And so, you know, maybe if you knew up front what they needed to achieve the outcome that they want, Probably that would be more efficient, but we don't. We're really done. It's a bit like when we're launching a website and we're trying to say, oh, we want 10 % conversion of the sign-up funnel. I wish we know how to do it. We know best practice, but except if you've done the exact same thing with the same target audience on, there's got to be a lot of learning. Oh, what are the main pain points when people sign up, for example?

16:09So CMA is for a platform. kind of like, well, also we are experts in the field in the platform. So also understanding when we build something, when we build a tool, that for us is obvious. It's kind of like, oh, how do we provide this on a platform? Oh, that's easy. We've got to do something with Terraform. And we already mentally, we eat a Terraform for lunch. But what about the other streams? So how long is it going to take them to get used to it? When they get an error, for us, like understanding error for time and time is like that. What about them? This is just an example. But again, like, you know, platform is just because it's partially close to my heart.

16:47But when I start trying to use these approaches to platform as a product, actually, I wasn't even working on a cloud team. I was working on a very different platform team. So it applied to every type of platforms. So the thing that is kind of a through line through this conversation is the idea of using the right metrics and customer-centric metrics. what's actually delivering for the customer, because that's what's going to drive value for the business and or for your team, right? So if it's a development team, are we actually improving development experience? If it's an external-facing team, are we actually improving the customer experience and driving revenue?

17:27What, in your mind, are the right customer-centric metrics to leverage, whether you're on a platform team or an external-facing product team? Okay. Now, obviously, the answer may change, But I can tell you what I used. So I used a framework called HEAT. It stands for happiness, efficiency, adoption, and task success. And I didn't invent anything. It basically comes from a framework that was invented by Google called HEART. I just removed a number of metrics that I didn't think was very useful in Python. I introduced some that actually think are very important for the graphing team. So this is where it's coming from.

18:10And the idea was to say, well, you know, let's recap. You know, we're building a product. Our customers are the teams that are actually like using our product. That's great. We know roughly what they want to achieve. They want to do their job better. They want to get faster and achieve their goals. So when we know that, how do we start? But normally, like where I would start, imagine again, if you start a company, where you want the first question you want to know is that, is anyone interested in my platform? Do they want to use it? You know, like you can't really spend months building something and then start looking for, it used to be like that.

18:49You know, I used to work in a company that built like a big, you know, data platform and then they were going around and say, are you interested? Are you interested? It's a big investment and if it doesn't work, you know, you waste a lot of money. So adoption is probably the sort of thing that you want to look at. And it sounds obvious because it's kind of like, Of course, but we don't do it. Obviously, like platforms start in two different situations. Sometimes a platform starts from scratch. So there is nothing like that. There is no surveillance whatsoever like that in the organization. And someone feels like, you know what?

19:25Our team really needs an analytics platform. It's kind of like, well, they're going to start creating their dashboard. It's going to be amazing. And we're going to pull all the data from everywhere. Definitely, we got to do it. And other times, instead, it's the opposite. Other times, it's kind of like there is a team or a group of people or one person that is providing a service. It's more like a job for him, let's say. Let's imagine your usual ops team. They're manually, when you say, hey, this is ready, can you deploy to production? They're already doing it. It just is not a platform. Why is it not a platform?

20:01Because it's not a service. Because it's not a product. It's kind of like it's actually human being doing it, so it's not scalable. Now, if I asked you, which one is riskier, implementing a platform from something that already exists or something that doesn't exist, which one would you pick as riskier? Probably the one that doesn't exist already. Right. Because you don't know if actually people care. so my advice in this case is always like when you measure adoption if you can move into a very like a low-key you know low fidelity kind of like service you put a team that manually does what your platform wants to do is a much cheaper and a better way to actually figure out adoption because what's the best adoption it's not like hey i promise that we're gonna use it you know i work especially in big company you know one month they say they're gonna use it and then priority shift.

20:52And then like, it happened to me more than one time when I was at Babylon, we launched some new functionality. We run down, like we got like three or four teams that were interested in using it. By the time we released it, we had two only. The other two kind of like, they changed priority. So also this is why like adoption you want to measure like in a way that sort of like give you a bit of bandwidth. So kind of don't get one customer, get more than one. And that's one. So adoption again, this is why it's probably the most important one, trying to figure out exactly and treat them well, because your first customer is likely going to be the one that you want to get closer so that you can learn the most.

21:30And then normally, depending on another thing that I normally do, so adoption is definitely the first. And then depending on what road I had picked, so if I picked the road of doing a lot of minor work, I may look into efficiency next. Let's imagine what I described. You know, like I want to do something. And at the beginning, instead of creating like a service, analytic platform, what do I do? I put three engineers that, you know, they get like a, you know, a Jira board. And people say, hey, can you create this dashboard? Can you create this dashboard? This is the data, you know, and they do have...

22:06People are using it. Yeah. And so, you know, when you figure out adoption, then the challenge is that, well, how do we scale it? Or at the moment, let's say it takes like 15 days, a month to get dashboard created. And not only takes a long time, but internally, it doesn't scale because if you have more than four people asking for a different dashboard, basically we max out the team. So it's not very efficient. So then you want to look at efficiency. Efficiency is basically a way to say, how can I reduce the time it takes to support this service? I don't know, be 30%, 50%, 80%. And, you know, like in the talk when I was at Leadbev, I was, as a joke, I said, well, I know you're thinking, you know, like if I have something like that, I just stop people from posting things on my ticket tool and I'll fix it.

22:59You know, no one is coming through. But efficiency is tricky because when you try to improve efficiency, normally you do it through some sort of automation. sometimes the you know the human aspect of doing something manually has a better user experience or potentially another thing that could happen is well i need to learn now a tool before i just create a ticket and that's it and now i need to do something myself so it's tricky because what normally what happens is when you increase efficiency all the things drop and what are the things. These are two other metrics. One I call a happiness, and one is task, success.

23:40So it's also verbatim from the heart framework. Happiness is easy. It's kind of like how happy and satisfy your customers with your software. It's kind of like, you know, if they're seeing you in the corridor on Zoom, do they smile or, you know, they want to slap in the face? If it's the second, I don't know, I think happiness is not very high. Obviously, you don't want to measure slap in the faces as a metric. I mean, I think it's quite painful as well. Yeah. So what do you do? Normally, these are surveys. Happiness is a type of survey when you kind of ask, you know, like how happy you are or satisfied you are.

24:16And also, if you are in a larger enough organization, my advice is why don't you survey? Really check with your research or user experience team, your product team, because they know much better than us how to write a survey that are not biased. So that's definitely my set. At the beginning, there is a lot of qualitative surveys. Sometimes we focus too much on the quantitative part. But I think, especially at the beginning, for a long time, you really need to get a lot of real feedback. These are questions, discussions, and so on, interviews. And that's one. The other is task success. So let's imagine, apart from the habits, general habits, how successful your teams are and achieving what they need to do.

24:57So we mentioned before how long it takes to get the dashboard ready to them. Maybe with you, it was, I don't know, like it was yes a month, but they were just waiting. And, you know, when it was ready, it was there. What if you launch it and it takes them 15 days? Yeah, it's half the time, but how do they feel about spending 15 days to set it up? So that's something you want to measure as well. That is kind of, you know, depending on the size of the organization and how many customers you have, you can... You can instrument your tools to measure some of those behaviors. But again, I would mix quantity and quality type of research in there.

25:35So get some feedback and get some measurements as you can. Again, if you have three customers, it's pointless to instrument your software because, you know, variance is going to be like quite high. But yeah, these are the four, the four one. And normally another advice I give is that don't focus on one at a time. Try to counterbalance them. So as I mentioned before, efficiency is a tricky one because you want to increase it without affecting happiness and that success. What you want to do is you want to agree with yourself and your team how much you're happy to lose. So you guys say, I don't know, let's say I want to increase efficiency 50%.

26:1250%. At the same time, I don't want to get an happiness lower than, I don't know, 10%, but that's success, I don't know, less than, I don't know, 80 % of what it was before or something like that. So that if something happens, you can stop. Okay, this is not working. This is not a solution. This is not an option. We need to find a solution. So this is how I did it. I was working on a workflow platform team. So quite different from like an infrastructure team. And we implement all of them for an year and a half. It was very cool. It was very cool because everybody in the team understood what we were trying to optimize for.