In short
Dev Interrupted Podcast Episode Notes
Episode Title
Foresight Over Firefighting: Being Proactive in a Reactive World | Rootly's JJ Tang
Hosts
- Andrew Zigler
- Ben Lloyd Pearson
Guest
- JJ Tang, Co-founder and CEO of Rootly
---
Episode Overview This episode focuses on shifting from a reactive approach in incident response to a proactive one. JJ Tang shares insights on leveraging AI and customer empathy as key components in enhancing incident management strategies, providing a vision for a safer, more reliable digital infrastructure.
---
Key Topics Discussed
- Reactive vs. Proactive Incident Management
- Current State: Many organizations find themselves in a reactive loop, addressing problems only after they occur.
- Proactive Approach: Using AI and customer empathy can help anticipate issues and build a more resilient system.
- Leveraging AI in Incident Management
- Integrating AI: Insights on safely incorporating AI tools in engineering practices.
- Common Pitfalls: Leaders often fail to fully understand AI's capabilities and risks, potentially leading to distrust from customers.
- Importance of Customer Empathy
- Beyond Technical Skills: Engineers should develop a connection with end-users to create products that genuinely address customer needs.
- Cultural Shift: Encourages engineering teams to engage with customer feedback directly, fostering a culture of customer-centricity.
- Challenges of Adoption
- Addressing Skill Atrophy: The impact of AI on developer skills and the divide between leaders and developers in perceiving productivity gains from AI tools.
- Engineering Culture: How creating a culture of learning and experimentation can help teams adapt to new technologies.
- Future of Incident Management
- Emerging Tools: The landscape of incident management will evolve with new tools and methodologies, necessitating a proactive mindset.
- Complexity of Incidents: As systems become more complex, the need for advanced predictive capabilities in incident response will grow.
---
Key Takeaways
- Transparency with AI: Companies integrating AI need to be transparent with customers about its usage and limitations.
- Continuous Learning: Teams should foster an environment where experimentation and learning from failures are encouraged.
- Customer Engagement: Ensuring engineers are in direct contact with customers can lead to better product decisions.
---
Points of Interest
- Windsurf Acquisition: Discussion on OpenAI's acquisition of Windsurf for $3 billion, highlighting the competitive landscape of AI tools in software development.
- Amazon Kuiper Initiative: Insights on Amazon's launch of Kuiper satellites aimed at competing with Starlink in satellite internet.
- AI's Impact on Productivity: The disparity between leadership and developer perceptions regarding AI's productivity benefits.
---
Resources Mentioned
- [Beyond Copilot: What’s Next for AI in Software Development](https://linearb.io/event/beyond-copilot)
- [Survey: Discover Your AI Collaboration Style](https://linearb.io/survey/pbl92vsc50i/bT79ARJ9)
- Article on [Avoiding Skill Atrophy in the Age of AI](https://addyo.substack.com/p/avoiding-skill-atrophy-in-the-age)
- Jen Riggins' article on the divide between developers and leaders regarding AI adoption.
---
Conclusion The episode emphasizes the necessity for engineering teams to embrace a proactive culture that values customer empathy and continuous learning, especially in the rapidly evolving landscape of incident management powered by AI technologies.
---
Follow Us
- Follow Ben on [LinkedIn](https://www.linkedin.com/in/benlloydpearson/)
- Follow Andrew on [LinkedIn](https://www.linkedin.com/in/andrewzigler/)
- Follow JJ on [LinkedIn](https://www.linkedin.com/in/jjrichardtang/)
- [Rootly's Website](http://rootly.com)
---
Support the Podcast
- Subscribe to our [Substack](https://devinterrupted.substack.com/)
- [Leave a review](https://ratethispodcast.com/devinterrupted)
- Subscribe on [YouTube](https://www.youtube.com/c/DevInterrupted)
- Follow us on [Twitter](https://twitter.com/DevInterrupted) and [LinkedIn](https://www.linkedin.com/showcase/dev-interrupted/)
---
Written by AI. May contain mistakes. Listen to the episode to check what was said.
Transcript
Automatic transcript. May contain errors.0:05Welcome to Dev Interrupted. I'm your host, Andrew Ziegler. And I'm your host, Ben Lloyd Pearson. This week, we're talking about OpenAI's acquisition of Windsurf and a new competitor to Starlink's satellite internet. We're also discussing the growing divide between contributors and managers during AI adoption and how folks like you and me can avoid skill atrophy. But Ben, we got to start by talking about Windsurf, right? Yeah, I mean, as much as I identify with that skill atrophy in the age of AI, we can hold off on that one for a moment and talk about this big news with Windsurf. Yeah, so it's been all over the news about OpenAI buying Windsurf for a$3 billion valuation.
0:46Windsurf is a coding assistant tool. It's an IDE that allows a user to do agentic workflows to build software. This is commonly used in a lot of different environments right now by folks experimenting with AI. You may recognize it as a leading competitor to Cursor. The deal which hasn't closed yet would be OpenAI's largest acquisition to date. And as you know, OpenAI is not shy about its acquisitions. In the past, it bought search and database analytics startup Rockset. It's also tried to buy the leading agentic coding IDE cursor before in the past. So OpenAI making this move now acquiring Windsurf, getting a stake in the game for agentic AI encoding, kind of an exciting development in the space.
1:33Ben, what do you think about it? You know, I'm happy to see competition in any tooling in general. I think it's one of the things that really sets American tech companies apart is just how competitive it is. This, I really think, is a simple one. OpenAI, they're just trying to build a moat and they have a lot of money. And in fact, I think this is their attempt to buy a moat rather than build it, as a matter of fact, specifically in the software development space. So you mentioned Cursor. It's obvious they couldn't get Cursor to sell to them. So they went to the next one. Cursor being like the fastest growing company in history based on revenue.
2:09It's going to be some stiff competition, I think, in both directions. You know, neither one of these companies really has a strong moat yet, even though I think we are getting to the stage now where the leaders are starting to sort of establish the norms of this space. You know, when I personally, when I think about agentic coding, I think about cursor rules and a lot of the features that they've built into their platform. And, you know, everyone's going to have their take on how that stuff should work. But right now, cursor is the one that's out there sort of leading the definitions around all of this.
2:43But with that said, I've heard a lot of positive things about Windsurf. I'm going to have to check it out at some point. I know some people that have tried it and have really liked it, even people who know cursor. So, you know, it's just something interesting to check out. And I like that we have a lot of competition in this space. Yeah, I totally agree with the competition has only created a better ecosystem of tools. I've actually tried out Windsurf. I had a lot of luck with it. I think it's a really nice tool. So I think that there's space for, you know, multiple players in this game to create a really great new future of development workspaces.
3:12So I'm excited to see what happens. Yeah, and from surfing the wind here on Earth to surfing the solar winds up in space, let's talk about what Amazon has done this week. Wow, what a transition. So let me launch this story into orbit for you. Amazon has launched its first Kuiper internet satellites, taking on Starlink in the satellite internet space. Kuiper, you know, you might read it as K-U-I-P-E-R. It's a Dutch word related to copper. And it's the name that Amazon has picked to describe its new satellite internet network. In this space, we all think of Starlink as the incumbent and has the predominant mindshare when people think about satellite internet.
3:54But Amazon is now taking on that space and that industry head on with their own offering. And Ben, have you had any experiences with satellite internet before? Yeah, well, first of all, I'm very familiar with the word Kuiper because of my four-year-old who's obsessed with the solar systems. I do actually like the name. You know, for anyone who doesn't know, Kuiper is where all the cool dwarf planets hang out. But, you know, personally, on one hand, I'm just hopeful that we're not headed towards the world in WALL-E, the animated movie where our atmosphere is just littered with space junk all over the place.
4:28I found myself wondering today, when did we decide to just let corporations fill up low Earth orbit with all of these space boats? I don't remember that decision ever being made. But with that said, more data connectivity is a good thing over the long term. And rural areas really stand to benefit the most from this. Areas that have been left behind by a lot of the benefits of the internet age, whether that's in the developed world or even in the underdeveloped world. You know, I've already mentioned my opinions on competition. Like, I think competition is good for services like this. Within some reason, like, there should probably be some regulations over the use of low Earth orbit.
5:07But I've also found myself wondering, with all the recent antitrust things that have been coming down the pipeline, specifically in the U.S. federal courts, how long until Amazon becomes under the microscope for the seemingly vertical monopoly that they're beginning to create? it's definitely an interesting competition space the idea of the world being enmeshed in all of these internet services internet reaching the far flings of the earth and people in rural areas having access to really fast speeds like that can be really transformative for a lot of people in a lot of situations it could even go as far as to save lives or educate people for the future so on the overall i think it's going to be a great development for people but i agree with you that as we launch more things in the orbit, we should think about how we're going to actually maintain this long-term and how things are going to go into orbit.
5:58So if you are listening to this and you have any insights on things entering and coming out of low orbit and you want to connect us with somebody who may be even launching things into that space, we'd love to learn more about those decisions and how things get sent up into the sky because that's a pretty cool application of engineering and there's a lot of technology behind it. Yeah, we've long held the belief on this show, many of us here, that we would love to have somebody who has either been in space or sent things to space on our show, which we've talked to people who have been adjacent to launching things into space.
6:29But if you are out there, somebody who is doing this, we would love to talk to you. Let's get onto the story about atrophy in the age of AI as somebody who often feels like I am maybe starting to become dumbed down because of my GPT usage. This article was full of advice and actions you can take to avoid skill atrophy in the age of AI. It dives into the reality of critical thinking decay and frequent users of AI tools. And folks who are using this on a day-to-day basis, they're probably familiar with this kind of mentality. So I really encourage you to check out this article on Substack by Adi Osmani.
7:05We're going to include it in our news roundup. And some things that really stood out to me was about making sure you throw yourself into the pit of cognitive thinking on a regular basis and you fight through that struggle. It gave you a lot of great ways to re-inject yourself into these situations that you might be guarding yourself from with AI that prevent you from developing skills. Ben, what kind of stood out to you as you read through this article? Yeah, so just a personal anecdote. Last week, I did a little like the most like artisanal content crafting that I've done in a while, you know, as somebody who has definitely adopted AI for a lot of my workflows.
7:40And that was to prepare a conference talk. I'm going to just real quick, I'm going to plug an event that I'm at this week. I'm in Miami at the Code Remix Summit. So if you're there, come hit me up and let's meet and chat about AI or anything else under the sun. I've tried to be more intentional about this like artisanal work, so to speak, at least like on some sort of regular basis, like, you know, once a week, if I can do something myself without about making AI the driving force behind it. Another example is just reading articles all the way through. I feel like this has been a trend really since the internet era.
8:15Our attention spans have gotten so short that it's hard to stay focused when you have a long, in-depth article and actually read it to the end. So I read this article all the way to the end and have been more intentional about making sure that I'm doing that. And there were two signs of over-reliance in this article that really stood out to me because I've felt them. The first is like debugging despair. Like if you're the kind of person that is just skipping reading error logs now and just automatically pasting them straight into AI, it could be a sign that you're over-relying on these tools. And trust me, I'm super guilty of this.
8:50Like I've done it plenty of times, but I've also recognized how limited it is at its ability to do a root cause analysis. So I never let it just be the only analysis. I will sometimes let it be the first analysis or if I get stuck, help me get unstuck. But you know, the sad part is there's actually a tool I just have been trying out in the last week, a brand new AI powered tool. And I really like it, but the thing that kind of stinks about it is it is like entirely built around this concept of like one shot co-pilot, like tell the AI what to do. And if it doesn't do it, you just click the button that says fix the error until it goes away.
9:27Yeah, no, I don't want to be in a situation where I'm unable to get my hands on the terminal and do the debugging with it, or all I can do is just roll the dice on it, hopefully generating the right command. That sounds torturous. Yeah. I mean, the irony is you click that button enough and it often gets there, you know, it often gets to a place. It can, but oftentimes, and I'm sure folks who've maybe been experimenting with this, sometimes it creates really strange workarounds. I've had situations where my application was giving errors and then trying to troubleshoot it. It would just change arbitrary parameters or things about the environment to make the error go away or seem less critical than it was.
10:05So it's always a good call out to understand the patterns that your AI is using to solve problems and that you understand what's going on. And I mean, even cursor can do can have the problem you describe like quite frequently. The other thing that really stuck out to me as a sign of over-reliance is losing your ability to do architecture and big picture thinking. So that's like if you're like accepting code without really understanding the long-term sustainability and quality risks of that code. And, you know, personally, I've been kind of operating under this theory that, you know, a lot of the frameworks that have been built for software development were designed to reduce human complexity.
10:44Like it was designed to help humans navigate the difficulties of creating software. But we're actually entering an era where AI can just abstract away all of that complexity. So how much do we actually need like all of these JavaScript frameworks, for example? So I've gone way back to the basics. I'm like working back in like HTML, CSS, vanilla JavaScript, and just seeing what I can do to orchestrate and control these agentic AI services. So what I'm kind of getting at is if you can feed it strong architecture, like help it make the right decisions, it can be a lot more effective at whatever code you need without the use of all of these dependencies and extra frameworks.
11:23But we'll see how this experiment goes. I actually don't know if it will be successful in the long term. Well, one of the best ways to get to a good, solid architecture or something that is a high quality document that you and a tool can work off of is really understanding the fundamentals. One of the standout pieces for me from this article was not using AI to achieve those fundamentals because struggle is good. And this was a really good reminder to me, and I think folks who will read it, that frustration is a key part of learning. And frustration is what enables you to remember, make connections that are important, but also think of solutions and problems that maybe you're not anticipating yet.
12:01And humans are still way better at this than agents. So you have to take that role when you partner with a tool to do your work in this way. But speaking of frustration, another story we're covering is the growing divide between developers and their bosses over how generative AI is adopted within their organizations. We're really loving this piece from Jen Riggins. It was on Lead Dev in the recent week about frustrating divides between developers and leaders during AI adoption at organizations. And it arrives at this by taking a close look at a survey of over 2 ,000 IT managers and developers last year.
12:38And in this survey, leaders listed AI as the most important technical factor in improving developer productivity and satisfaction. And that's a huge surge in mindshare for that leadership audience. Meanwhile, only a third of developers reported experiencing any significant AI productivity gains. And so what this survey really highlights is the divide still between leaders and doers on how AI is actually impacting their work every day. And this is a cumulative piece that has quotes from folks all across the developer experience and productivity space that speak about this commonly. It also has a quote from yours truly in the article.
13:20I really recommend you check it out because there's so much cool advice in there from smart people that are talking about this. And so, Ben, what stood out for you? First of all, I think there are a few quotes from you, and it's really great to see that you're getting the opportunity to take what we're learning from our community and share it with audiences like over at Lead Dev. Because, you know, I'm a huge fan of them. They do great work over there. So we love being able to share knowledge over there. But I think Ori Karen said it best when he came on Dev Interrupted, where he said that developer productivity is going to decline this year.
13:52And I think upper management is just going to have to accept that. Like there's a lot of disruption that's happening just across all of knowledge work, not just software development. And there's a lot of adaptation that is taking place right now to respond to these disruptions and change adaptation, you know, rethinking how you accomplish your work. That all detracts from what would conventionally be viewed as like productive work for lack of a better phrase. You know, you're not able to focus on shipping new features and moving faster when you're just legitimately trying to redesign how you complete your work.
14:34But I think there's a couple of traits that I'm seeing from organizations who are leading the charge with AI adoption. The first is that they provide really clear guardrails around AI usage. That's training on how to properly use it. That's guidance and resources that help provide context to models that make them do the right thing rather than hallucinating security vulnerabilities. So that's the first thing. The second thing is they're giving space for early adopters to experiment and share learnings with the rest of the company. So at just about any company out there, you've probably got a small percent of your developers who like to use new technology.
15:13They like to experiment. They like to find new ways to do things. They need to have the tools to let them do that. And then the space to experiment and the channels to share those learnings back with the rest of the organization. Developers want to reap like the benefits of AI, but the technology just isn't there for all use cases yet. So you want to let the people who can benefit from it benefit today while teaching the people who are going to benefit tomorrow, you know, what to expect. So, you know, expect productivity losses from this disruption is the point that I'm trying to make. And just be ready to reap the productivity gains that are around the corner.
15:51Like, they're starting to show up today, but they really just haven't fully manifested yet. So get your organization ready for that future. Yeah, so Andrew, who do we have on the show today? Yes, in just a moment, we're bringing JJ Tang of Rootly on the pod to talk about creating an engineering culture built on customer empathy. Stick around. Beyond Copilot, what's next for AI in software development is happening on June 4th and 5th. Join us for a live 35-minute panel featuring past podcast guests from Adnan Ijaz from Amazon Q and Brigida Bokler from ThoughtWorks, alongside experts from Linear B.
16:29We'll explore how leading teams are going beyond Copilot to experiment with agentic AI, measure real impact, and drive meaningful DevEx gains. All registrants get the full recording plus early access to the DevX Guide to AI-Driven Software Development, packed with tools, prompts, and insights from the 2025 AI Developer Survey. Reserve your spot today and stay ahead of the AI curve. Hey everyone, for those of us just joining us, today we're joined by JJ Tang on the podcast. He's the co-founder and CEO of Rootly. And today we're tackling some really cool stuff because JJ is at the forefront of incident response automation, helping teams turn chaos into reliability.
17:13More importantly, the work that he does at Rootly, it helps make our world a safer place by automating incident response for things like mission-critical infrastructure. And today we're going to dive into how leaders can build with shiny new things like AI safely, why customer empathy is kind of an underrated engineering skill, and how teams can go beyond just reactive reliability. So, JJ, welcome to the show. Thank you so much for having me. After seeing all the amazing guests you've had, I feel very special to be here. Oh, no, totally. The honor's on our end. It's really great to have your perspective here.
17:49Like I said, I'm real excited about the topic, so let's just go ahead and jump right on in. Right now, a lot of companies, they find themselves rushing to integrate new things, and maybe they're not stopping to fully consider things like reliability, trust, or usability at scale. And JJ, I know you're always thinking about building safely and not just fast. That goes for all things, not just shiny new things with AI in them. What are some of the common pitfalls engineering leaders face or fall into when integrating new stuff? And how do you think they can avoid them? It's a great question. we serve some of the most mission-critical businesses.
18:28We work with companies such as Dropbox all the way to NVIDIA, but we also have customers, for example, in the Nordics, like 911 call centers. We work with nuclear energy plants and gas stations, and we cover the entire gamut. So for us to be reliable and secure as a platform becomes incredibly important. And one of the principles that we have from an engineering standpoint is we understand And we are in the business of doing the boring things right. So when we started building our on-call product, the first thing we said, this has to be immensely reliable. So we developed this multi-cloud architecture that no one else in the industry had.
19:08And what felt like overkill, we never really had to use it in the purpose that it was built for, but it inspired a lot of trust. So I think there's this aspect of functional reliability and also perceived reliability that also becomes quite important. When we thought about building with AI, we said privacy and safety becomes the most important thing. I think in this world, in particular with AI as companies are adopting it, if you're a vendor that is building AI into your tooling, You have to understand a lot of the customers that you're selling to and you're serving and that you're caring for are also venturing down this path at the same time.
19:51And if both of you view AI as this mysterious black box, neither of you are going to trust each other. And so I think the onus is on the vendor to be incredibly smart with it. The recommendation and the thing that we found to be helpful for us is integrating and building AI into your product is the easy part. I think truly understand, understanding the power and also the dangers and pitfalls of AI is the other piece we spend a lot of time on. And so us as a business, we experiment constantly. We've developed our own rootly AI labs specifically to tinker right on the bleeding edge of what these technologies can do and what they can't do.
20:33From there, we've been able to formulate our own guardrails and truly understand what the capabilities are, then conveying that in of truthful and trusted way back to our customer, I think inspires a lot of trust. And also for us, then suddenly safety and privacy becomes a core differentiator that we love. And, you know, your mileage may vary. And I think that's how you go from being this bolt-on AI product, maybe it's a wrapper for some organizations. There's nothing wrong with that either to being this AI native company that truly understands what it's capable of. Totally. It's more about like having the transparency of understanding its impact and what it's supposed to do, what it can do, what goes into it, what comes out of it.
21:23And, you know, during that process, it's like there's a lot of alignment that has to happen. How do you think that engineering teams can ensure that the solutions and, you know, shiny new stuff that they're using, it actually aligns with their business objectives and it has a measurable impact? Is there a pattern that you see successful teams kind of follow to achieve this? I think for individual teams, it has to be a sustained effort. I think when you view AI as just a singular feature of your product, then you go back to doing what you were doing before. The peaks and valleys become either very high or very deep and can be quite jarring sometimes.
22:04What we've done to ensure that we're constantly thinking in this AI-first mindset and how we build sustainably over time in intelligent ways through our product is every leader of every function inside of our business, we have a bi-weekly meeting where everyone presents new AI tools that they've been using. I would say 80 % of them fall short. We tried some automated PR review tools, some QA tools. We tried some on the RevOps side that unfortunately didn't work out, but had a lot of. Just having proof of concept, just the prototype, flexing out the idea. Exactly. And what that did for us culturally was really important.
22:45It forced leaders to think in this mindset. It also forced us to explore what the latest and greatest was out there. And that naturally propagated itself through the rest of the organization. So when people talk about AI now, it doesn't feel like a novel concept as if we did two years ago. It just feels like a natural extension of our day-to-day workflow. I don't think we've gone as far as, you know, maybe Shopify. I know they came out with a memo recently saying, hey, you're not allowed to request for headcount unless you could prove that AI agents can't do your job. I don't think we aren't there yet as a business, but maybe one day.
23:23Yeah, we talked about that memo on a recent episode, actually. And the picture you're painting here is really helpful because you're showing how creating that culture of learning, creating that space for innovation in a company helps normalize using it as a tool and building understanding of it. It even goes back to your first point about having the transparency and the reliability in what you're doing. Part of that is making sure everyone's equipped with knowledge and you're giving them the space to experiment. In that world where they have the space to experiment and they're trying new things, they're growing as a company, it kind of then starts to butt up against maybe traditional ways that people ship products.
24:00And I wanted to ask about how traditional incident response has typically always been a reactive process for teams. They wait for something bad to happen and then they get out the fire extinguisher and they go find the fire and put it out. But Rootly changes that model. I want us to understand a little more about how, from your perspective, you know, what are the key differences between a proactive and a reactive reliability practice? And what does it mean? Yeah, I think that's a perfect way of characterizing it. Either it's proactive or reactive. And I think it's actually with the unlock of LLMs, can we truly be more proactive now?
24:39You know, we built a very successful on-call business, incident response business that is primarily geared towards helping humans become faster and more consistent at resolving incidents. And, you know, the maturity curve where most customers are, that is what they need. But what the future holds with LLens becomes really interesting because the work that if you think about your smartest SRE, your hardest working, your most, you know, the SRE that's making 700k a year that has 20 years of domain knowledge, their ability to recognize patterns, their ability to sift through large amounts of data that are relevant for your business is incredibly powerful.
25:20And that tribal knowledge is really unlocked by sometimes no one else. And this hero mentality starts developing and they start getting burned out. And with LLMs, what we can do now is harness everything that they can do, then replicate it across hundreds of agents that can simultaneously then do the work for you. So the way that we're solving for this right now is imagine when an alert fires. the first things that you will do is you will go check you know maybe your logs and telemetry and traces and synthetics check datadog you check sentry check you know your github you maybe try to mentally correlate what were some of the past incidents that were occurring and what we're able to do with our agents is well we can sift through large amounts of data instantaneously before that You even have time to flip open your laptop and tell you what the probable root causes.
Read the full transcript
26:17And where this starts becoming powerful is the complexity of incidents is only going to increase over time. I'm sure you guys see it all the time. The adoption of coding assistance has been through the roof, you know, cursor roof. You know, it's one of the fastest companies to reach 100 million. Everyone uses Copile. I was recently in London with a private equity firm, and they have strict board level mandates to adopt coding assistance because they view it as if we can get 5 % more developer productivity, then why wouldn't we do it across, you know, we have a million developers. That's a huge money save for them.
26:54But what these coding assistants and, you know, if people are vibe coding into production right now are new system dependencies that introduce problems and errors and incidents that we did not necessarily anticipate. And even your smartest SRE, that L7 SRE that we were talking about before, does not know those changes are occurring. So on the other hand, you need a new wave of machines to help debug what other machines have done. So you're fighting the, you know, the trend on both sides effectively. And that's where you can get, you know, not only necessitate the need, but where proactivity then becomes important.
27:35What systems like ours will be able to do is actually determine whether or not a human needs to be involved. And the future of that becomes really exciting. On one hand, you're identifying what the issue is. On the other hand, you're able to identify what the right fix is and deploying the right fix. then all the way shifting left into the point of pushing your code into production you can emulate what the impact could be before you even make that change i might be able to say to you andrew hey i'm maybe i wouldn't do that that seems really dangerous because of this and this reason yeah it's like predictive impact in a way and what you're saying is really interesting and it actually threads through something i've been hearing from recent guests on the podcast it's actually quite interesting how this narrative compounds itself.
28:25So to connect to them, you know, we had a recent guest, Tanya Janka, she's a cybersecurity expert, and she really expressed how cyber attacks and incidents are only going to get more and more worse. They get exponentially worse every year with the introduction of AI and agents and this kind of thing at scale, that risk is even higher. You know, these fundamentals become even more important. And we spoke with another recent guest, you know, Sagar Bachu, who talked about API consumption in the future and about how it's going to be 10x as much as it was the day before and how like overnight LLM consumers of services and tools are going to break infrastructure and are going to have profound impacts on how we've built these things.
29:08And now here we are at the culmination of that in that you have to fight that big fire, like those thousand fires that are happening with something that can predict where the thousand fires are going to happen and how you can minimize the impact of that. So if you're taking advantage of all of the benefits of an LLM and AI, you also have to take advantage of all of the protections that you need in that environment. Does that kind of thread through on how y 'all are thinking about it too? Yeah, totally. I think it opens up, you know, selfishly for us, a whole new market of productivity that can occur because a whole new class of incidents will start existing.
29:51And I think, you know, the TAM that LLMs would open up for other software companies will also be quite enormous. But with the good comes the bad to a certain extent. But I think the net positive is still there for most companies and for the space and category as a whole. Yeah. And let's talk about the steps of getting to this proactive model. And I think a lot of it comes down to cross-team collaboration. I'm wondering, you know, what's your perspective on how you see successful teams interact between engineering, product, leadership? How do they work best? I can probably share this best or characterize this best through a few examples that we found particularly helpful.
30:35I think over time, the definition of roles and functions will change. There's a few things that we've done. one introduction was the concept of an ai engineer a generalist ai engineer that we hired and this person's job is to find effectively areas of opportunity where agents can do the work of our team and also be the cultural influence for how other teams can think about adopting and using AI as well. So one of the things that we did was we had a 15 person team of BDRs that we had hired in Denver. And one of the things that we were able to do with an AI engineer was entirely automate how we prospect by using tools like clay and clod and merging them together with a few other things that we're doing.
31:31This person did not have a traditional, you know, computer sounds background. And now we're going function by function and finding the opportunities of where this automation can now exist. We're finding a lot in our own reconciliation processes on the rev ops side of the house. We are finding it a lot on the support side of the house as well. And so I think this concept of where traditionally when we started as a company, engineers felt they were maybe they had a role definition of just, hey, my job is to execute on these tickets and then I go home and do that. The lines are becoming more blurred and we're very much so leaning.
32:11We're letting the role of everyone here to be more blurred. Adam, who is our head of marketing, is running a lot of our self-serve business to ensure that's in a good spot because he has a ton of experience there. That might not be the role of any other head of marketing, but we're totally fine with that. And I think adopting that mentality becomes incredibly important. It has changed how we interview people as well. We don't want to see them as you're only a specialist in a particular area. We want to see that you can be smart and be a generalist in many other areas if you had to be flexed towards it.
32:51Then also culturally, we've done things to allow everyone in the organization, not just customer success and not just sales and not just support, to be much closer to the customer. The tangible examples I can give is every single customer of ours has a shared Slack channel or Teams channel with us, and all of our engineers can access it. Sometimes they're working on a feature request for a particular customer. They're in there directly with the CTO of that customer going back and forth on, hey, does this look good? What would you change? Here's how we're thinking about the next iteration of it and getting feedback.
33:28And that takes a lot of time to deliberately codify, I would say. One of the things in the early days of our business that we did was there's this tendency to just start creating dashboards for the sake of creating dashboards. And I never really understood that. I remember we went through this process of creating these product metric dashboards that measured, if you became a Rootly customer in the early days, you would get a health score, depending on your activity in the platform and a whole bunch of other factors. You got summarized into this number. And this number felt weird to me for the longest time because it ultimately distilled the uniqueness of you as a customer, the challenges and pains that you felt as a human into this number.
34:19Okay, you're a 7 out of 10. What does that really mean? So we went through and we deleted every single dashboard, every single product measurement tool we had. And we said, the only way we are going to learn from customers is we are going to get on the phone with them. And we don't care if you're a$2 ,000 customer or you're a$2 million customer. As far as we're going to get on the phone with you and we're going to uniquely understand what is working and what is not working. And that culturally has set, now we have a bunch of dashboards for what it's worth. That culturally has set the stage for us to be incredibly customer-centric, to be close to customers, engineers interacting with them has changed how we hire, how we evaluate, all of that.
35:04So I think to answer your question, a lot of it has to come tops down. A lot of it you have to think through systematically and codify and you have to find these pockets of where people can really carry the culture and the behaviors that you want as a business that amplify that the best that you can. And oftentimes we get it wrong too. We've done many, many things where we said this is the right move, then two weeks later we said that was the wrong move. it's an iterative process and i like this culture that you're building that focuses on a customer obsession that really gets close to understanding their problems and i want to talk about how that impacts engineering you know for rootly and how customer empathy plays a role there because you know i think it's very typical for engineers engineering leaders to focus specifically on the technical problems and forget about that connection to an end customer so what is customer empathy in that kind of engineering environment look like to you?
36:04Yeah, the thing to say, there are downsides of being too customer centric. There are really great engineers that have passed on us because of it. They don't want to be on the phone with the customer. They don't want to be interacting with the customer every day. They prefer to do hardcore engineering and process and system design stuff. And I think we've made that trade-off because, you know, And when you're not a Salesforce-sized company or Instacart-sized company, you don't have the luxury of a really good decision maker in every turn. I think we rely a lot on, we never A-B test. We always rely on our intuition and instincts to make the right call in a product.
36:47We'd much rather delete what we wrote and be wrong than to run a long, drawn-out experiment. And I think that for us, culturally, it's remained true. And because of all of those things that we do, being incredibly customer-centric is important because we do not trust someone's opinion unless you can advocate and be somewhat the voice for the customer. Because then you don't have someone in that room making that decision that does have the customer empathy will override your decision most of the time because of it. And that becomes incredibly important to us as well, because we want to make, we want all engineers to not just be engineers.
37:34We want them to make product decisions. Our product ultimately serves. We want them to be product engineers, right? And it's serving an end goal, an end customer. Exactly. And so that's something that we found works, at least for us. We'll see if it works in the next five years. But for now, we're continuing down the track. If you ever sign up to be a Rootley customer, a lot of what you interface with is actually with engineers on our team. That's cool. And I'm wondering for engineering leaders who may be listening to this and thinking, maybe I want to try this practice. What are some habits that or some key things that they could put in place for their own team to build that customer obsession, that customer empathy?
38:18I think as an engineering leader, the first thing that you want to do, and where we really started was consistently sharing the context into what they were building. Oftentimes, engineers will get locked into a particular work stream, a particular feature without understanding the larger picture. Because oftentimes, I think we rely on, oh, well, product, that's product's job to, you know, have the overall holistic picture. Does it work for the startup? Does it also work for this enterprise? Is it important? Is it revenue driving? Is it nice to have is like, you know, closing a competitive gap, all of that nuance often gets lost.
38:59It doesn't translate to what engineers do. I would ensure that context exists. It's not as noisy as you think it is. In fact, it's incredibly motivating in good ways and also sometimes not in good ways. You know, you might work on something particular for a customer and you don't end up winning that customer and then, you know, it doesn't feel too great. But, you know, when you do and you can see your work map directly to these outcomes, it triggers something in your mind where, hey, I really like that feel. I want to do more of it. And so all of our engineering leaders, we have a weekly sales meeting.
39:33Our engineering leaders are part of that. So they can distill that to their team. And we post it publicly. We share it in our all hands. There's not a single engineer that works at Rudely that doesn't understand where we are, revenue and customer wise more than an AE here as well. In everything you've guided through today, there's so many interesting new things to consider in the world of incident response. And when we think about incident response, it's a it's table stakes for everywhere, especially large infrastructure, critical infrastructure, things that when they go down, it's not just like, oh, you can't order pizza.
40:12There are services that can, life and limb can be on the line and provide key services and key scenarios. So something I've learned in having these conversations is that founders, they sit on these very unique precipices and they have a very far vantage into how things are moving and evolving. I wanted to ask you in our chat, what would you think is the future of incident management from where you're seeing it now in charge of Rootly? Where do you see it all going? Are there interesting or even scary or kind of like things that keep you up at night? Kind of curious to know. Yeah, the lucky position we get to be in as a business is no matter how good a company gets at their incident management, they always still have incidents.
40:56So we have this fortunate problem where we never run out of supply necessarily. And it's a challenge that affects all verticals and businesses. And I often joke with our customers, they get on the phone with me and then tell me how bad their incidents are and how many they've been having. And I tell them, I said, well, that's great. It just means I can find a way to help you make that a little bit better at the very least. And I think it ties into a lot of what you called out previously. And the world is just going to get more complex, more sophisticated. Tools will naturally evolve. I think there'll be a new class of tools, a new way of working with agents.
41:35There'll be agents that go rogue and do the wrong thing. And those need to be detected and created as incidents. and do humans resolve those? Do agents resolve those? I think it's all to be defined still. I think the complexity and this landscape will change. And I think for us, the work, smart humans will always exist. We will always be the co-pilot to them in many ways. And their work will shift to become this new category of impactful things. They're not going to be on this category of like updating juror tickets and writing status page updates anymore. Their brain is best harnessed to use elsewhere.
42:19And maybe that's using other agents to help agents. And so the world will get more proactively reliable as well because of it. Fascinating. Well, you've given me a lot to really think about here about where it's all going. But before we wrap up, I wanted to ask, you know, where can folks go to learn more about you and what you're doing at Rootly? Yeah, I post a lot on LinkedIn. My mom likes every single one of those posts. I think they're somewhat interesting. I talk a lot about AI and how we build the company and a lot of the things that we're doing on the incident response side. I try to share quite a few photos of my dog as much as possible.
42:58Oh, I love that. I love dog pics on LinkedIn. I'll definitely go check that out. you know, Dev interrupted and we post a lot on LinkedIn too. So I have to continue the chat there. And if you are listening to this and have some thoughts about what we talked about today, please, you know, jump in. Let us know. You're going to be seeing this all over LinkedIn if you follow us. And, you know, based on your predictions of what's going to happen in the future, you know, JJ, maybe we can have you back as like an update on the status quo of incident management, of incident response. We can see what's evolving because I think it's quickly moving.
43:29I love that. Thank you so much for having me. Of course. If you made it this far, you know our loyal listener. First off, thank you. Second off, you clearly really liked it. So be sure to subscribe to the podcast. If you're only listening to this, be sure to check out our sub stack as well and what we post on LinkedIn. And that's it for this week's Dev Interrupted. We'll see you next time.
44:00staying well!
From the publisher
Still stuck in a reactive loop with incident response, only fixing problems after they happen?
JJ Tang, Co-founder and CEO of Rootly, joins host Andrew Zigler to reveal how to shift beyond reactive, leveraging powerful AI and an often-underestimated skill in engineering: genuine customer empathy. Discover how these elements are crucial for navigating the complexities of modern infrastructure and shaping the future of incident management.
JJ explores the forefront of incident response automation, discussing how to integrate shiny new tech like AI safely and why deep customer understanding is key to building trust and reliability. Learn about the common pitfalls leaders face, the cultural shifts needed for proactive reliability, and how teams can make our digital world safer.
Check out:
- Beyond Copilot: What’s Next for AI in Software Development
- Survey: Discover Your AI Collaboration Style
Follow the hosts:
Follow today's guest(s):
- LinkedIn: JJ Tang
- Website: rootly.com
Referenced in today's show:
- OpenAI agrees to buy Windsurf for about $3 billion, Bloomberg News reports
- Amazon launches first Kuiper internet satellites, taking on Starlink
- Avoiding Skill Atrophy in the Age of AI
- Why developers and their bosses disagree over generative AI
Support the show:
- Subscribe to our Substack
- Leave us a review
- Subscribe on YouTube
- Follow us on Twitter or LinkedIn
Offers:
