Backstage’s journey from spreadsheets to global IDP standard | Spotify’s Tyson Singer

0:06Hey there, and welcome back to Dev Interrupted. I'm your host, Andrew Ziegler. Today, we're joined by Tyson Singer, Spotify's head of technology and platforms. He's the person responsible for turning engineering chaos into harmony for teams. And Tyson's fingerprints are on everything at Spotify's internal developer experience, from backstage to the open source platforms that reshape to how engineering orgs operate across the industry. And he spent nearly a decade steering Spotify's massive tech ecosystem. And now he's doing it again with Portal, a next-generation internal developer platform built from hard-earned lessons from this whole experience.

0:46And we're going to dig into some of those learnings today as where is where your team could take it. And Tyson, welcome to Dev Interrupted. Thanks, Andrew. Appreciate you inviting me, and I'm excited to have this conversation. We're really excited, too. Spotify comes up a lot in our discussions with engineering leaders. It's a widely accepted, adopted leader and tool to look towards for getting teams aligned with their own internal practices, which is something that we talk about constantly on the show, as well as like making for a superior developer experience. So, you know, we're going to kind of dive into the journey of how Spotify ideated and created and scaled and delivered this kind of experience.

1:26But we're going to start by looking a little bit at the past from where this kind of technology even comes from. Because inside Spotify, before Backstage even existed, engineers were living in effectively like whisper networks, right? You're asking around for docs and for services. Who owns what? Where's this resource? How do I get started with this? So how every good journey starts, how did that chaos evolve into what then becomes now such a widely influential platform model for developer experience? Yeah. So one thing to maybe call out in that question is that even today, Spotify is actually a very network-oriented company.

2:08It's really important that we stay synchronized across kind of everything we do. And that has a very human element to it. But we are so large now, the weight of all the connections between all the people just doesn't scale very well. So if we look back to about 20, sometime in the 2010s, what was going on in the software industry was in order to enable scale, and I'll say scale on a couple different dimensions, one from just a compute perspective, like the number of consumers hitting your platform, and then the other from like a people perspective, we were shifting from these sort of large monolithic software approaches where all the dependencies were resolved to kind of build time.

2:53to much more smaller independent scale software components where things were resolved more upon time to maybe oversimplify it. And so at Spotify, that was really happening kind of on steroids. It was Spotify has always been a very extreme context. We were growing fast and we had a very microservices oriented architecture. So thousands of engineers, tens of thousands of software components. And, you know, we had to make sure that the system was reliably running and scaling. We started doing that, you know, kind of using, you know, sticks and wires, aka spreadsheets, and to kind of map out how things really work together.

3:42And that was sort of fine initially. but what we found was that you know as we scaled out we had to figure out who was behind the software and that was changing really rapidly and so if you're dependent on a component you need to you need to talk to the person who you're dependent on to make changes you have to find the relevant documentation to understand things and what we also saw is because we had a really really autonomous culture, trying to hire the best and the brightest. Everybody came in with their own background and how they wanted to develop software and wanted to do things independently.

4:24So we saw a lot of people duplicating capabilities in our ecosystem because they couldn't quite see, has somebody already built that? Or can I reuse it? So we kind of went from this like, all right, we have a bunch of spreadsheets to we need to be more organized about this, have a single managed repository for our entire software catalog that provided this both sort of a build time and a runtime visibility into the dependencies. And most importantly, the people who were involved in that so that we could really manage this rapidly changing ecosystem. We had, you know, we have this idea of a squad team and squads would split and the software they own would move from here to there.

5:04And we wanted a system that could move at the pace that we needed. So that's sort of the starting point. I'll pause to see if you have some more thoughts about that. No, I love how you painted the scene. You took us back in time for a minute. It's the 2010s. You're doing what every large company is doing with monolithic software. You're breaking it up into microservices. You have hordes of engineers that are owning all these small services and one-off types of implementations of things. And you're trying to reduce people duplicating and kind of reinventing the wheel, so to speak, Like while still allowing a high degree of autonomy and ownership over the engineering decisions that are making you best in class at the time.

5:47That's why all those best amazing engineers wanted to work there still do is because you were creating this environment for them. So I like that you, you know, that's a very relatable experience, I think, to any large company that they were, we were all doing that in the 2010s. And so it sounds almost to that from the beginning you tackled it not just as technical debt, but as a type of cultural debt to people are coming in and they have misaligned expectations or they want to do things their specific ways. They've assumptions that are maybe frozen. Right. And so you're trying to create this cohesive experience.

6:24So now that starts to evolve from, I guess, from a network, and you say it still is. Where does it go from that point onward? Yeah. So those were some of the sort of the initial challenges that we saw, but we needed to, as you just pointed out, we needed to get people more aligned to a similar way of doing things. And so we had a number of different tools to do that. One of the most powerful tools we had at that time was because we were growing very fast is we had our onboarding boot camps. uh and we basically indoctrinated people right then uh with our uh golden paths our golden tech our tech standards our quality standards and so this was like new people coming in trying to get them aligned to things of course all the existing people didn't necessarily have those same expectations because we hadn't set them and so you're trying to change the expectations for for people coming in still dealing with like legacy expectations but what you we couldn't see really well without more tooling was where people were diverging from our expectations and from those golden paths and the golden state and the golden technology.

7:39And since, you know, we sort of the engineering leadership couldn't see it, well, the people who are doing the work couldn't see it either. So we needed to bake in visibility so that everybody could see, okay, like, what am I even trying to manage to? And so having a tool that allowed for effective visualization, of that, really putting it at the fingertips of the engineers to say, like, okay, this is actually the easy way to do things. This is the right way to do things. And you can see it and you can visualize it. So, you know, it wasn't just about, you know, getting that linkage between the software catalog and the people, but it was also getting that linkage between incentives and awareness and visibility on what people were doing that then kind of all eventually got bundled up into this product we eventually called Backstage to sort of match with our music-focused culture.

8:35And so Backstage ultimately arises from the internal curation of that whole developer experience, from having the catalog of tools to having the network of talented engineers, but then also having the golden paths and the very visible, replicable ways of picking the picking the tool out of the toolbox, so to speak, and it's already ready to go. You can go and tackle today's problems. And so, you know, I think a lot of teams can get to that point, you know, to give them some credit. A lot of teams can think about what are those communication issues that are stopping us? How do we align our developer catalog of what we have to accomplish?

9:13And then how do we put it in front of everybody and repeat it at every town hall meeting, right? Anybody, anybody who's making software could do that with a large team. But then how do you then take it from just being this loose group of ideas into being backstage? You've ended up productizing it. It becomes a tool that other teams can pick up off the shelf, break open, and then be like, these are all of the parts we need to have a successful developer experience where all these things are laid out for us, right? Was that a conscious decision that you and other Spotify tech leaders, like engineering leaders, found yourselves in early?

9:53Or was it at the end and you were like, oh, this is something that's ready to productize? Like, how close was it to being at that point? Yeah, there were a number of points where it was quite a conscious decision. When I started at Spotify, what I noticed was I had this amazing team of smart folks. A lot of them were very focused on infrastructure capabilities and doing that really well and maybe a little less focused on the sort of customer needs. And in this case, I'll refer to the customer as sort of the feature teams across the organization. And so, you know, building out this platform was kind of like nothing special in one sense.

10:33It's just like standard product development to understand your customer needs and workflows. You have to build your quantitative and qualitative insights on their behavior. You have to develop your hypothesis and your value propositions and kind of treat it as anything that you would do. Since Spotify is a consumer-centric company, we're very data and insights oriented for that part. It was pretty easy to kind of take those skills and add them towards an internal customer. And I think that's what a lot of folks weren't doing at that time. They weren't saying, oh, I can take all these same patterns and I can apply them to my internal customers and build out internal platforms at that time.

11:14But that's what we did. And we bolstered our product management and our design team and our insights function to ensure that the work that we were doing to support the effectiveness of our engineers and our developers and the ecosystem really had a high ROI. So that was sort of an important part of just setting the framework there. One of the things that really worked for us to drive adoption very well was to make the platform something that everybody could participate in. So Backstage was built with this idea of adding capabilities into it dynamically. So yes, my teams would build out the core of it.

11:55They build out a lot of the core capabilities and workflows. But then in a feature team, anybody could come in and contribute. They could say like, oh, like I really need to understand this aspect of my software and I want to visualize it and measure it and do those sorts of things. And so they could do that. So the capability was there in order to get people to engage with that capability. We set up some hack weeks. I think this is probably in 2019 where we said like, hey, go and hack on this. see what you can do to make this platform like yours. The end of that year, I think we had more than 100 different plugins from across the organization contributing into that.

12:38And so it was sort of also applying these techniques for building a platform where you really engage your user base to contribute back to it. And since our user base was developers, they could contribute back to that in a way that a lot of users can't. They can literally shape it and build it out. and you know eventually we took that further because we open source backstage allowed the community to engage in that follow that same pattern sort of flow back into the value ecosystem and just sort of continue that that journey with our commercial product as well a bit later so spotify becomes community zero for backstage you get everybody on the platform contributing to the platform you have these hackathon days structured around you know just putting everything else down and just working on our developer experiences.

13:27By the way, a really common tactic of really high performing teams with excellent developer productivity and experience who are able to tie that work back into results. Like you mentioned ROI. That was something that's very important from the whole time building this experience is you're not making it just to make it. You're not, you know, picking out the nice, beautiful cutlery to put on the table just to set the table nice, like you're there for a purpose. And so everything had a high degree of utility while still allowing people to exercise freedom. And of course, all of this becomes an open source tool that other people can use.

14:05So in that world, we talked a bit about the past, that's like the environment that made that possible. All of the levels of iteration from like the generations of Spotify leaders and engineering folks who have been there, I've been building on backstations at the beginning, but now we're coming into the present. And Backstage itself and what Backstage is needing to be is evolving pretty rapidly. The engineering world that we are in this year is very different from the engineering world we were in last year. And so what are those hard-learned lessons of Backstage that you're carrying now with you into Portal, into the other things that we're going to be talking about Spotify is working on?

14:47You built that foundation. And then how has the last year broken that foundation and changed your expectations? Yeah, so maybe it's good to explain a little bit of where we came from. So we decided to open source backstage probably 2020. And part of the reason we did that was because we could see that this process of platformizing the product was very valuable for our context. and that other folks would be able to do that. But it was also, to some degree, like a selfish decision, which is we can see that when there is a very, very clear need that everybody has, then someone's going to come along and solve that.

15:32They're either going to do it commercially or they're going to do it in open source. And we had deeply learned experiences that replacing things and migrating to new products is really expensive. And so we basically made the bet that it would be less expensive for us to share our solution to make it the industry standard for what eventually became defined as internal developer portals than to try to go out and replace it at some point. So that was sort of almost like just our high-end investment decision from one perspective. But you get a lot of value back from it. You get a lot of value from the developer community.

16:14Uh, they, they, they engage with backstage in particular, we had one of the highest engagements of any open source project from external parties. So typically when you open source something, it's really mostly the open sourcer who does all the work, but we had very, very different statistics, like 40 % of the contributions were coming externally. So that was, that was really validating on the strategy overall and it worked out, it played out well, but we of course made some, some, I don't know if that we've been made mistakes, but we built in things that didn't necessarily work for the community.

16:49So we put a lot of effort initially in the front end of that plugin ecosystem that I mentioned before, and then kind of said, great, and integrate it with your backend system. It turns out that didn't really meet everybody's needs in the community. And so we spent basically a year rewriting that backend plugin ecosystem with the community. And that was pretty painful, when you have to do a lot of that low-level plumbing to rework things so that more people can participate in the ecosystem and in more flexible ways than we started out with. So that was really good for the community. I think over time, it was also better for us because usually when you have the opportunity to rebuild something, you build it better.

17:38We have rebuilt backstage a few times before we got it out the door as well. And so those were all useful iterations for us. And then also useful as we sort of go towards our commercial product because it provides a more sort of no-code approach, low-code, no-code approach for the full plugin ecosystem inside of Spotify portal. So those things come back in goodwill to your approach as well later sometimes. Yeah. Imagine being Spotify and being so big that you can just be like, oh, you know, it's actually cheaper for us to just define IDP as a category than it is to change or move to anything else.

18:16I love that as a conclusion. It's not that hard to imagine. We did open source a project called Helios, which is our orchestration capability. And Google did the same thing like a day before us. And we know how that went. like kubernetes uh one replacing our system with kubernetes has been extremely expensive so we we know the reality of it but yes it helps not to be a startup at that point to to be able to do those sorts of things absolutely and so you take those lessons and you i mean you you also described this anecdote of having to rebuild the back end of some parts in order to make it more broadly applicable that's a you know that's a definitely a hard lesson it's way easier to I already have the plumbing in place than to be up on the stage where everyone's watching expectantly, waiting for you to ship the plumbing, watching you pick up one pipe and everyone's going, no, bad pipe, bad pipe.

19:12And you put it down and you pick up a different shaped pipe and more people are yelling, bad pipe, bad pipe. You know, it doesn't feel like it's a good solution. So you had to work through this, like fixing it in the open, but then you get through it. And so where does then portal come from in these in this evolution because it sounds like oh you know you're checking all the boxes backstage is perfect why why started a new product named called portal yeah so on our journey uh through evolving backstage first we open sourced it uh we got a lot of community engagement a lot of positive feedback one of the you know strongest uh open source projects out of the gate and then just grew from there.

19:52And we said, okay, this is cheaper than doing a replacement, but it's not cheap. So it's great if you can sort of amortize some of the cost of that development work. And we launched what we call our commercial bundle of plugins. So basically a set of plugins that weren't things that we put into the open source ecosystem, but we thought like, great, this will help people go faster. This will be a revenue stream back into our ecosystem to help cover our costs and potentially more. And that has been really well accepted by the community. It's really grown a lot. A lot of people are interested in that capability.

20:31And then we started to see that there were more challenges that folks were having out there. So for us who created Backstage and the category of IDPs, we had a bunch of sort of assumptions like, yeah, sure, this is easy. Like we can, you know, set this up and manage it and do all these things in Spotify specific ways. And a lot of other companies were like, yes, you can. Well, we don't really want to do that. Like that's not our core business. Could you do that for us? And we said, yeah, we could probably do that for you. And so that led us to sort of a SaaS based product, which we called a portal.

21:07We can't actually call it sort of backstage portal because when you open source something, you give away your rights to the naming and the branding. So it is our version of Backstage in its sort of commercial form. So that's why we call it Portal. Okay. So then you arrive at the SaaS offering and you're offering it as a solution for other teams, other engineering orgs to pick up off the shelf, use as a service through you. So in those conversations and those relationships, how does Spotify show and measure and help teams understand the impact of those tools and to get it fine-tuned? Because obviously it's so flexible, every team that picks it up is going to have to wildly probably change it to make it fit them for the exact needs that they came to you for in the first place.

21:59So how do you go about thinking about those conversations? What do you measure? What's most important? Yeah. So what we do in our approach is we talk with a potential customer and we try to understand their context, the outcomes they have, their culture, how it differs from our culture, and explain to them how the product can be customized for their context. So we have a plugin called Soundcheck. Soundcheck is a way to help people align to whatever technology, quality, or other standards you would like in your organization. And to display that and visualize that and give the team members who are doing the actual work actionable ways to move on that.

22:46And so out of the door, we provide basically a whole bunch of defaults, which we are like, these are kind of the Spotify way. But then we realized that, of course, no organization is like Spotify. Every organization is unique. And so they need to be able to customize and adapt that. So that's sort of how we interact with the customers in those type of contexts. And it just kind of goes on for the different capabilities we have across portals. yeah i like that you acknowledge that when teams come in and they adopt it it's largely a it's part you know a cultural adoption in terms of you you look at it and you you study them and you understand what are their needs why are they coming to us what are they looking to achieve and it really acknowledges the fact that each engineering org is kind of like its own living organism and each kind of need their own different kind of treatment and help you know yeah going Going back to your question about sort of hard learnings, like sometimes we kind of, you know, not very humbly think, wow, we're like, we're so unique and great.

23:50But then we forget that we are unique. I won't say anything about the great part, but like that we are actually unique and that other companies have different requirements and needs and we hadn't necessarily understood the variety of those things. So you go through this process of building out a product, you learn those patterns. And then those patterns become useful to other people because there are actually sort of a more finite set of patterns of like, this is how companies operate. Spotify, for example, is a company that is in sort of a less regulated space than some of the folks who use Portal.

24:24We have regulatory requirements. We have SOX requirements for a certain part of our software pipeline, et cetera. So we have that, but like some customers have that in space. And so they need to apply those patterns more aggressively. They need to do more of the aggregate metrics things and feature rollouts and a little bit more of those stronger guardrails. It's a different requirement. One's not better than the other. They're just different. And so having a platform that allows people to do that is really, really important.

25:01AI has changed how we build software, but faster code doesn't always mean faster delivery. That's why Linear B is launching The Essentials Plan, your toolkit for measuring and improving AI productivity. You get the new Linear B MCP server, AI Insights dashboard, and developer surveys, all designed to reveal how AI tools impact delivery speed, code quality, and developer experience. Stop guessing which AI tools work and start leading with data. Visit LinearB.io to learn more and unlock the next chapter of AI productivity. You know, you're wise to draw circles around the groups of users and find the cohorts and then understand that those cohorts kind of have similar needs to each other, obviously at different levels of intensity.

25:46But even just by grouping them in those loose ways, understanding like, oh, this person's in a regulated or less regulated industry. This typically impacts, you know, how they use our product in this, this, and this way. I think that's a smart learning for anybody who's building a SaaS offering and who's dealing with a large array of customer sizes as the cohorts and understanding why they come to you, especially for something so ubiquitously broadly customizable as Portal is. You know, you're talking about everyone's engineering org. Everyone has one at this point, but they all look incredibly different.

26:21and um speaking of looking incredibly different i think it's time in our conversation to talk about how ai is impacting some of this stuff and it's actually kind of refreshing because this has been a nice long conversation where neither of us have really brought up ai and those are pretty rare these days but you did it i you can't possibly think i was going to let you get away from this one tyson i do have some questions about how ai is coming in and impacting this because you know first and foremost um you you get to the point that you're at you've built backstage you built portal you have this community you have so much momentum behind this experience and ai kind of comes in like a bus and hits it you know on the side how are y 'all thinking about where ai fits into all of what you've built yeah so i'll give you maybe just a few examples to sort of share the directionality of that one of the the primary areas where an idp like backstage really helps people with is discovery and so you're specifically trying to discover your software ecosystem how it's linked together how it's linked to the to people you know where like and and all the documentation and all that sort of stuff that is uh related to that that ecosystem so as sort of gen ai swept through uh the world uh what happened back in 2023 is we're like oh great look at this like amazing potential for expressing and solving that problem in just like a better way how can you instead of making someone go across the documentation and searching for it just they can just ask the question i can come back with a really high quality answer that sounds very simple but it actually required a fair amount of context engineering to look at all the different knowledge sources that were important in that that context and distill that and represent that in a really effective way uh first some of our first iterations on that like the quality of the responses came back not so great and i'll talk about in a second like what are the metrics that we we tried to focus on uh anyways we ended up building this product that we call ai knowledge assistant or aica and sort of another plugin into our ecosystem uh that tries to uh really consolidate all that knowledge uh in a very effective and context specific way and so as we thought about this particular problem we thought about what's the what are we trying to solve and so one of the biggest opportunities for improvement in the developer ecosystem is to improve the flow for developers.

29:08So I'm doing something. I'm dependent on some other component. I reach out to you, Andrew, and say like, hey, I don't understand that thing. Can you help me? I just interrupted your flow. My flow was interrupted too, because I couldn't figure out something. I had to wait for you to wake up because you're on the West Coast of the US. I'm in London. All these things are like flow disruptors. And so what we wanted was something that could solve part of that problem, at least. And so with Ica, we actually were managing to those type of outcomes. And what we saw over time, after a few iterations, was basically a 50 % reduction, 47 to be precise, in our internal support requests.

29:49We saw employees using this basically every day with more than 1 ,000 employees on a base of about you know 4 000 employees uh and uh weekly active users at about 85 86 percent of our developers so that told us like great there's some like clear adoption metrics some clear like task uh avoidance type metrics uh that really were driving very concrete and specific value so developers could stay in flow both on sort of both sides of the equation Right. So really like impactful metrics coming from what on the surface seems like, great, it's another chatbot. But there's, you know, a lot more context that goes into that in terms of having something that really allows AI to have a meaningful impact.

30:41how did you measure and understand you know you cited 47 of like interruptions prevented effectively like you're i like how you define success for aika is like protecting the flow state of our engineers our engineers are flowing towards whatever goals we've set for them we already have them aligned for roi we need to reduce their interruptions reduce the blockades that they hit along the way. Was this like a cohort? Did you start ICA small with a group of people and do before and after with surveys? And I ask this specifically because these experiments are things that people are running constantly in their engineering orgs, but everyone kind of emerges with a different level of an empirical way of understanding what happens.