Is Internal Tooling Holding Your Team Back? | DevZero’s Debosmit Ray

0:00An engineer is never wrong, even when they are. And so if you ever ask an engineer to use a tool they don't like, they won't do it. If you ask them to turn left, they'll turn right. So at that point in time, it's very important to first explain what value propositions the tool will actually bring to the engineering team. And then you all need to be very, very focused on making these tools very simple and easy to use. And once they see the value and actually derive the value in their day-to-day workflows, they will end up being your biggest champion. Are you an engineering leader and tired of constantly being asked, when will it be ready?

0:39Stay one step ahead with Linear B's project delivery feature. Our powerful dashboard lets you visualize key milestones, forecast delivery accurately, and align stakeholders effortlessly. With Linear B's project delivery, you can confidently showcase your planning, prioritize work effectively, and even make data-driven cases for additional headcount. Say goodbye to delays and missed milestones. Sign up for Linear B today and answer, when will it be ready before anyone even asks? Hey everyone, welcome back to Dev Interrupted. This is your co-host, Connor Bronston. And today I'm delighted to be joined by Debo Ray, co-founder and CEO at DevZero.

1:19Debo, welcome to the show. Thanks so much for having me, Connor. My pleasure. Before you founded DevZero, you had a really fascinating career as a staff engineer at Uber, where you worked in infrastructure and product security. You've also founded two other companies before starting DevZero. So you're a perfect guest for us, particularly because, and I want to give you a bit of a shout out here, you graduated from my alma mater, the University of Washington. Go Dawgs. Yes, the best school level. Amen. It's clear that you are someone who's very passionate in everything you do, whether it's your work at Uber or dev tooling, something that you've worked in deeply, both at Uber on the infrastructure side, you're on a mission to unlock developer potential is kind of how we've seen it defined for you.

2:06And that makes you a kind of kindred spirit for us because at Linear B, we think a lot about these concepts. And we both agree that when you get the tooling right, You make developers' lives better, and you make them happier, which the great thing is that helps engineering leaders, engineering teams produce higher quality work, produce things faster. So I want to kick off this interview with something you said before the show started, which is engineers join companies to do good work and help the business, but are held back by internal tooling.

2:47At the same time, we are also very impatient test people. And usually the path for me to launch and get some feedback on this new feature is often a pretty arduous one. And I have all of this context in my head. And I just want to have written some code and I want to see this thing working end to end. And as developers normally in companies, we have to go through these like almost doilsome dev cycles where I'm constantly like changing my mental context. and this is quite challenging and often unproductive in my opinion. And this is generally what we call the inner loop of software development.

3:40I want to constantly be in this state. That is where magic and creativity, everything's happening. But we end up spending most of our time in the other drudgerous aspects of software development, right? Be it debugging issues related to the environments that we are working in and all of the inconsistencies there, or being able to access some data so I can actually test this feature end-to-end or actually trying to figure out what the best way to call this sister team's products or services is. Or the worst one is when I actually have to get into a meeting and drive consensus with fellow teams.

4:15So, yeah. Yeah, it's tough because of the obviously crucial things for a development team. But most devs have been trained to be this kind of focused individual problem solver. And, you know, as we get into larger companies where you're relying on a lot of other people to get big products done, which is, I mean, how you achieve incredible things, great teams, people don't necessarily have the tools, training or time to make some of these team dynamics work the way they want to. And companies often don't do a good job of actually ensuring that devs have the processes, tools and help they need to be in the best position to succeed on these.

4:57Where do you see companies wasting the most dev time? In my opinion, developer productivity, I think it's a fundamental part of any tech company's core product line. And when you don't invest in internal tooling, a couple of issues start to prop up over time. There's usually this gradual increase in the friction of shipping new product updates, right? And then we also see successive product iteration cycles. they take longer and longer and more delays start to show up. Early on, when a company is just starting off, it's just a group of people that have gathered together around a certain project.

5:37And often testing is now happening in production. It's risky, but the developer experience can be pretty good early on. But right after that, as you gain more users, you start to care more about reliability. And immediately this drift starts to happen. and you end up making a somewhat divergent copy of production for your CI, for your dev. And normally as engineers, we'll always try to keep the app layer consistent because that's the layer that our end users usually get to interact with the most. But the underlying infrastructure fundamentally is forced to diverge now. And this is a gap. Once introduced, it gets consistently harder and harder to reduce over time as the company just continues on its lifecycle.

6:21So it sounds like what you think that if the foundation that is laid around dev tooling and internal processes for engineers is problematic, it's really hard to restart that later down the road once you realize it's wrong. And usually in most companies, you'll end up seeing that it takes a significant investment later on when the problem reaches a point where essentially a shipping product comes close to a standstill. Are there particular companies that are doing a great job of this that folks listening can learn from? Yeah, so in my opinion, and obviously it's a bit of a partial one, at Uber, I thought we did a pretty fantastic job with this.

7:06We had a consistent, it was almost a culture of introspection into our dev tuning and processes. is we've made a lot of mistakes along the way, but we were continually questioning whether we were shipping fast or not. And I think that, and we obviously learned from our peers over in Google or in Facebook now called Meta, at how they were also approaching this problem. A few of the lessons from our peers and industry were very helpful. So it sounds like you think there are definite lessons that companies can learn from some of these massive mega cap companies that have invested in platform infrastructure teams, in internal tooling.

7:45What are some of the specific lessons that someone, say, working in a startup or maybe a scale-up could learn? Yeah, so as I was saying, the constant focus on whether we were shipping fast enough, I thought was a very important one. And often this will fall down to one of the super easy-to-measure metrics, right? Everyone normally starts with the number of commits that are being put out, the number of lines of code, etc. which kind of start to treat software as tangible goods. Although as a starting point, I don't think that's a terrible place to be in, right? At least it shows the beginning of introspection.

8:21And like one important thing to note over here is like software falls into the class of intangible goods and its value lies in its novelty. So by shipping more features or capabilities, obviously thoughtfully applied, you actually stand a chance of providing more value than you currently provide today. Because if you're doing it right, you're iterating positively. Exactly. So caring a little bit about the cycle times, and am I actually shipping enough changes into production that are reasonably sized? What's the reverse for these large companies? What can they learn from startups about how they're approaching tooling and enabling devs to have positive developer experiences?

9:05The company will always start as a small group of people working on a project. And at this point in time, you fundamentally care about the day-to-day experiences of all of your colleagues. And you're consistently also building these little tools to fill the gaps that you're spotting on a day-to-day basis. And communication channels are as a result of being a small company. They're very simpler. But as the company starts to scale, this strategy does not scale that well. Yes, having developer productivity owned in some way is much better than not having it be owned at all. But as the company grows, it will often become a community-funded initiative, right?

9:49This is an approach that will end up falling prey to the fallacy of the company. Absolutely. There needs to be executive sponsorship or clear importance put to productivity and the experience developers are having. Otherwise, you risk these major challenges coming to light down the line and not enough focus to put on it. Because in the end, it's so easy for a business to say, oh, yeah, we want to build this tool for internal productivity to help us on the long term. But we have a short term goal. We need to focus on that. And let's shift that team over. We're working on a side project. We need to make sure we ship this for our customers.

10:29And while that might be fine once, even twice, over time, you're going to devalue the tools and the platform that your engineering team is using. And you're going to, to your point, bring your product organization to the standstill. Actually, to that point, Connor, I was reading this thing recently. There was a comment from Dylan Fields when Figma was very, very early on in its lifecycle. And apparently they had an outage. The engineering team was working on a variety of features at that point in time, which they deemed to be more important for their end users. And during this outage, Dylan was like, why aren't we fixing this right now?

11:05And the engineering manager over there was like, we have more important things to work on. Dylan's like, we have users. The engineering manager is like, not really. Dylan's, we have 10 users and some of them might be using us for the real world. So we have to fix this right now. So a lot of this is also cultural and like engineering reliability and the developer experience. It has to come from the top of something. That's a very illustrative example. And I think it kind of speaks to another ethos with engineering teams, which I've been guilty of myself, which is we usually think we're right. We're solving these problems.

11:43We're doing the research. We're on the front lines of this creative process that is kind of delivering the end value. And we sometimes maybe become disconnected from the actual users, the customers. And I think that's a really important piece of business alignment that we talk about, but sometimes don't put enough emphasis on. And I think a corollary of that to focus in on tooling is that as engineering teams, if we don't want to do something, if we're like, I don't really like this tooling, it doesn't make sense for me and my workflow initially, it's hard to get us to actually adopt that tooling, even if it's something that's crucial in the long term.

12:26So I'm curious from your perspective, how do you think engineering leaders should get their teams to adopt tooling that they know is going to be important in the long run, even if in the short term they maybe don't see the benefit. Yeah, I think taking to your previous point, an engineer is never wrong even when they are. And so if you ever ask an engineer to use a tool they don't like, they won't do it. If you ask them to turn left, they'll turn right. So at that point in time, it's very important to first explain what value propositions the tool will actually bring to the engineering team. And then you all need to be very, very focused on making these tools very simple and easy to use.

13:09Let the engineers derive value from it. And just fundamentally, right, especially now being a vendor that provides a developer tool, what we see as developers always hate talking to salespeople, right? As an engineer myself, I can completely relate. As an engineer, just let me tinker. Let me see value. make sure that engineers are able to do whatever they want to with your product. And once they see the value and actually derive the value in their day-to-day workflows, they will end up being your biggest champions. So let's say I'm a salesperson and I'm coming to you and I'm saying, look, Debo, you're a former staff engineer, you're CEO now, you're kind of leading this team.

13:52And I really think you need this dev tool to help your team. How should I pitch you? I will very rarely try out a tool just because a salesperson asks me to. I might find myself going to the website and actually trying to sign up myself, connecting my source code or whatever else to see how the tool works. A bunch of examples, guided walkthroughs are usually helpful. I'll take a look at the documentation. And then if I have a burning need at that point in time, great. If I don't, it's something that will stay in the back of my mind. and probably in a few months, weeks, whatever, when that problem arises in my life, I will remember that tool if it left a good lasting impression.

14:35I think that to your point, you need to meet engineers where they are and say, hey, I just want to share this information about something that can help solve a problem I think you might have. And if you have that problem, great, then maybe you'll go pursue it and you'll investigate it. You can dive into hopefully great documentation, a great walkthrough, a free signup flow. But companies and people who choose to kind of wall that off are doing themselves a disservice and making it much less likely that engineering teams will pick up their tool because we want to tinker. Yeah, and that's the other thing, right?

15:11And I find because I use a bunch of different tools on a day-to-day basis, one thing I've noticed is just there is this creative bit of my mind that I like to exercise whenever I'm doing something. because as engineers, I think we always like to discover things and then solve problems. And if something seems a little too magical, you end up questioning it a lot. Like as engineers, we always, like, sure, first it needs to be a value add to my life. And then I also need to understand how exactly this thing works underneath. And my sentiment is this gives me a sense of reliability that, okay, when I use this on a day-to-day basis, I have a general idea about how things are working underneath that tool itself.

15:56Okay, now I can use it. So let's switch gears a bit. Can you catch the audience up on the work that you're doing at DevZero? Yeah, so at DevZero, we build cloud-hosted development environments, primarily aimed at software engineers to do their day-to-day writing, testing, building code. Our goal is to ultimately provide a production symmetric environment for all engineers in a company, their very own environment, to do their everyday work in. Our belief is that this is the highest context environment possible and this is the closest you'll be able to get to literally coding in production. And in environments like these, it turns out you don't have to wait as long for your code to build or compile or for your test to run.

16:44and turns out you can actually run your end-to-end tests right within your dev environment itself. And you can also reuse these for CI if you want. Why is it that most development hasn't moved into the cloud yet, do you think? Yeah, so most of the tools that we use on a day-to-day basis, right, be it our communication tools or the various collaboration tools that we have, all of those have moved to the cloud. But the only notable exception to this trend has been the tools that developers literally use to write and run their source code, our IDs, etc. My sentiment is that this has happened because of two large classes of issues.

17:30One is the internet dependency. There is the perception of a lot of latency and what happens if I have to code when I don't have access to the internet. So that's one class of issues. And the other is just the overall complexity, because at the end of the day, I am moving my local host to the cloud now. And how am I going to get all of the dev tools that I'm already used to using to work on this non-local host network that I'm already used to? And most of us engineers, we don't have a very deep understanding of how networking, et cetera, works underneath, right? So the various potential value propositions that might be available as part of this, like just the first hurdle of being able to jump through that barrier might seem a little off-putting.

18:18Yeah, that initial friction can be a big turnoff for folks who would otherwise make the jump. And if I'm an engineer who has a local ID, like I'll say I do, my instinct is to keep using what I have here and expand my tooling in other areas. Maybe because it's a comfort thing, maybe because there's just these blockers. So given these challenges, why should engineering leadership choose to push into the cloud for actual ID usage, coding, etc.? The thing near corner is, at the end of the day, developer productivity is something that is usually owned by a CTO or a VP of engineering or an infrastructure or platform engineering leader.

19:06And they care about things like the performance of environments, like our developers slowed down by their local environments or from using various shared tenancy dev clusters. How easy is it for us to onboard offboard engineers to switch environments? How easy is it for an engineer to switch teams? They also care about things like security and compliance. And so my fundamental belief is that in order for an organization to truly adopt cloud-based development practices, it has to be a top-down motion in some way driven by a couple of these priorities. So it sounds like you envision a future where these tools take the Microsoft Word, Google Drive route of not necessarily having software locally on the computer, but trying to maintain a constant connection so that you can reduce local build times and offload those to the cloud and take these other approaches.

20:09Yeah. See, at the same time, Connor, to your point, right, you have your ID on your local computer. Your local laptop feels like home to you. And when you open it, you just open up your ID and stuff just works. Changing these fundamental developer workflows, I don't believe engineering leadership would ever want to do that because it doesn't make sense. You have to retrain your entire workforce. And that comes at a pretty significant cost. So what will work in this space, my belief is that a concept of combining local tools with remote compute, that engineers use ITAs, all of their aliases, dot files, etc., exactly how they use it.

20:51To our previous conversation around points of friction, remove all points of friction to the engineer adopting cloud development environments. Have everything be already set up and ready to use for them. And I think pairing this idea of local tools with the flexibility and the power of remote compute is going to be the trick. And how are you trying to solve that problem at DevZero? Yeah, so for us, we look at environments in a couple of different ways, right? There is obviously the whole idea of a hermetic environment primarily paired around one repository. We do that pretty well. But as an engineer, especially nowadays, everyone's using microservices in some way or form, whether it comes out of a monorepo or a bunch of different smaller repos.

21:43And in this world, we take a dependency on having upstream or downstream services around us. So at DevZero, we try to make that world possible as well. So we have this templating language at DevZero that enables a DevOps or a platform engineering or an infrastructure engineer to go and configure what dev environments might look like for a certain sub-team or large-team, etc., with all of your dependencies, etc. around you. And then we have a bunch of essentially pipelining work in the background that makes sure dev environments are always ready to go. So when an engineer shows up, you just click a button, you connect your local ID to one of these environments, and suddenly everything looks and feels the same, but it also feels like my ID has just moved into a production-ish environment where I can just start to make calls to arbitrary downstream services, which in the past was only possible in my dev cluster or in a staging environment of some sort.

22:44And we've talked a bit about how we don't want sales folks to pitch us to get on these tools. But given that, you know, you're a former staff engineer yourself, you're now a business leader, you obviously need to pitch other engineering teams on why they should check out your service. What's the approach that DevZero is taking about how to share information and interest developers and getting on board? Yeah, so as we were talking about, engineers do not like being sold or marketed to, right? but I also believe engineers like well-marketed products. There's this overall aspect. I have a problem.

23:27I have discovered a potential solution to my problem. Here is how I have implemented a solution to this problem. And as like vendors, we just have to be at the right place at the right time. Cold calling or cold emailing isn't something that will work to get essentially an engineer onto a product. they just need to be able to find you in the right places, be it a Hacker News or when you're doing a Google search. So, yeah, for sure. Well, if you have any tips on how to make sure we're on Hacker News all the time, we'd love to get the podcast on there more. I need to find someone that knows that trick.

24:03Yeah, I think we all do. Well, fantastic. Debo, thank you so much for coming on the podcast. It's been a great conversation. Where can folks follow the work you're doing at DevZero or check out more about your personal story? Yeah, so I have started writing on Substack. It's called Debo's Newsletter. And in parallel, we are obviously launching a bunch of new features within DevZero, primarily to solve a lot of these pretty complex replication challenges. And these are continually going out of it, usually in two to three week cycles. So feel free to sign up on our website at devzero.io. Take Dev Zero for a spin.

24:44Follow us on LinkedIn, on Twitter. Fantastic. Devo, you should absolutely come write a guest article on why cloud tooling is important for dev teams on our sub stack. So for those of you listening who don't know, we've got about 14 ,000 folks as of this recording who are now subscribed to our Dev Interrupted sub stack. We put out a weekly, a deep dive article on Thursdays And on every Tuesday, we'll release our podcast episode, plus some of the top content we're seeing in engineering leadership. And Debo, we'd love to both feature you on the podcast whenever this comes out. And let's maybe see if we can work together on an article about cloud development.

25:24I think it'd be really interesting to kind of do those in concert. Absolutely. I'd love that. Great. Well, thank you so much for coming on the podcast. And thanks, everyone, for listening. We'll see you next week.

25:41you