Redefining Incident Response: Insights from the Chaos Engineer Behind Jeli.io | Nora Jones

0:00I tell orgs, you should actually over-index on calling incidents. A big warning sign is when I talk to a leader and they're like, we don't have incidents. We actually don't have any incidents. And you're like, really? You'd be surprised. I hear this from like very large companies that everyone uses. And that scares. I'm like, I know you have more incidents than anyone else. And incidents are surprises. Incidents are anything that pulls you out of your day that you did not prepare for, right? There are always things to learn from. At DevInrupted, we work to give engineering leaders actionable ways to improve their teams.

0:31That's why we're producing a three-part summer workshop series with Linear B. Each of the three workshops will explore the processes that elite software engineering organizations and executives use to deliver better business outcomes and reduce cycle time by 47 % on average in just 120 days. You'll learn how to assess your current performance and benchmark it against industry averages, streamline processes through automation, and improve business outcomes through resource allocation. Learn from the best and take your team to the next level. Visit our website to learn more and secure your spot today.

1:04We're back on Dev Interrupted, and I'm excited to be joined by Nora Jones. Nora is the founder and CEO of Jelly.io, an incident management platform, and she's also the ex-head of chaos engineering at Slack. Welcome to the show. Thank you so much for having me. I'm excited to be here. I'm really stoked to have you on here because you have a couple of, I would say, almost controversial opinions about how engineering works. One of which is the talk you gave today about leading from incidents and how you think incidents should drive the decision making for a company. Can you expand on that? Yeah, absolutely.

1:39So incidents and unexpected situations are happening all the time. We are constantly getting surprised in our organizations. You know, we have this plan, right? And we set out to do it and we have this feature roadmap. And then all of a sudden a customer is complaining about something. Something doesn't look right. We get this weird spidey sense feeling and we all jump on and we fix it. And then we have to go back to our roadmaps afterwards or back to the things that we had promised to be delivered. And we kind of just leave all that happened there. And there's so much data hiding in that. There's so much investment that we can get out of, you know, this thing we already spent a bunch of money on because it took a lot of time and energy to spend money from the incident.

2:19And there's just there is a lot of value in understanding how people talk to each other during an incident, who they brought in, what they were looking at. Like all that data can be used to understand how your organization works and also help create this this learning culture. So that was kind of what my talk was about today. And then, like you said, I have many controversial engineering. I'm excited. Let's dig in a bit more, though. So I'd love to talk through an example if you have one that comes to mind about how a company can leverage an incident to really learn and grow out of it. Yeah. So I talked about one pretty extensively in my talk today, but I think the real data is hiding in your seemingly innocuous incidents.

3:05You know, the ones where you went, oh, phew, no one noticed that. Like it wasn't a step one. Yeah, we're all good. Right. that's that's actually where you can get the most learnings because emotions aren't as high afterwards when you have a very impacting incident like there was a data dog one recently right and i guarantee you things were emotional internally there afterwards like a hundred percent and if they're not practicing every day you know the seemingly innocuous events the ones that take five minutes the ones that no one noticed like it's going to be hard to glean data from those large scale emotional ones.

3:41And so an example I would take would be find an incident in your organization that didn't take very long compared to your normative standards, but maybe took a lot of people in your org to help fix. And I would ask you, like, did you actually take the chance to learn from that incident, to do a review on it, to interview the people that participated in it? That's an example I would use. And that's one I go over in my talk today. It's like a seemingly innocuous search incident. And then, you know, the team did an incident review afterwards and I'll put it in quotes, but they really just kind of went through a checklist and then got back to work.

4:16And it just it wasn't that valuable to the organization. And so in the talk I give, I kind of take an alternate approach to doing that. Let's talk through that approach a little bit. Do you view the incident report as kind of the key stepping stone to saying, OK, we have to be really extensive here and this is how you start to make these changes? Or how would you approach it? A lot of it is in the cognitive and coordinative work and it's unearthing that. So in every incident you have, you're calling in experts on the situation, right? You can't fix the incident without having some form of expertise.

4:49And so your job after the incident is over is to unearth what made sense for people during those moments, how they figured out what to do, how they learned what to do. And you can do that by asking them certain questions like, oh you know hey Brian how did you get pulled into this incident and Brian can say Joe pulled me in oh how do you tell me how you and Joe know each other oh uh and like it might feel silly asking these questions because you think you know Brian's answers but the thing is Brian wouldn't write this on his own like he wouldn't think to you know and so you're now asking these seemingly silly questions but by documenting this you're able to actually start to learn how your org works and so So Joe can be like, oh, you know, Brian and I worked together on an outage like this a year ago.

5:36Oh, tell me about this outage a year ago. Right. And so then you can start to untangle everything and understand how these patterns are kind of surfacing. And then you can also ask Brian, what were you looking at at this particular time? And he can say, I was looking at this graph I made. And I can say, can you show me that graph? And he might raise an eyebrow at me like, oh, Nora, you know this graph. Like I've seen you use it. But that's not my job in this moment. My job is to like unearth these things and capture them so that other people later on can then learn from this. And like capturing these things, be it in a report, be it in a meeting, be it in Slack.

6:14It doesn't matter. It's like where where does your organization learn? Are you a document culture? Are you a video culture? Asking those questions and sharing those answers in a way that other people can hear and learn, too. And it ends up helping new hires. It ends up helping your leadership teams. It ends up helping your current team. And it just, you know, it ends up growing more experts too. So based on this reporting and logging you're doing and this kind of discovery process to understand where the communication layer is around incidents for your organization, what are the next steps? So I know you mentioned documentation.

6:51How do you enforce folks actually learning from it or encourage that learning from the documentation instead of just having it be buried somewhere? And that's what I was saying before. It really depends on your kind of culture. I've been in cultures that were very meeting averse and they were like, document everything. Everything was in docs. Then I've been in cultures that met all the time and didn't really document anything. And like there's pros and cons to each approach. And I think what you need to do with this is do what feels natural to your culture. Don't try to introduce something totally different, but introduce the way that people are already kind of working, you know?

7:27No prescriptive plan, but instead approach it the way that's going to make sense for your org. I think started a little bit more like started by asking various questions. And on the Jelly website, we have a guide called the Howie that actually like goes through all of this and in a lot of great detail that can kind of give you this this approach. But I would be prescriptive at first. I would start to do it kind of under the radar, like little things like asking questions about having someone pull up their graph, maybe naming names in your incident reviews. Doing stuff like that can help you start to build this kind of culture over time.

8:03But, you know, in a prescriptive plan, I would basically say you should find the responders. You should talk to them one on one. You should ask them various questions about the incident. And your goal when you're talking to them is not to put them on the defensive. It's to make them feel like an expert because they are. And they're an information source for you. Yeah. And a lot of the times people are like excited to tell about their expertise. They are excited to tell you about all the things they experienced in this incident. The trick is on you, on how you talk to them, right? And you're a podcast interviewer.

8:39You're good at this. You're good at making people feel like experts. There's probably all sorts of techniques you use in the way you ask questions, the way you talk to people, the way you research them. It's the same thing with incident interviews. And those go a lot better and you gain a lot more value for them if you take that time to prep. Like, oh, wow, I noticed you were in this incident a couple of weeks ago, too. Like, what was that like for you? And then, you know, they might just be like, it was fine. And you want to start off with a softball question, which is probably what you do in your podcast.

9:07Sometimes, sometimes. And like, but like jumping into those things helps people more comfortable. And when people are more comfortable, they tell you more. And then afterwards, you know, what I would say is amalgamate and aggregate all the things that people told you. And that's that's your report or that's your facilitated meeting or that's your email or your Slack or however your culture works, you know, it's about taking that information and disseminating it afterwards. But that's the prescriptive approach I would take is making sure your goal, I think it goes back to your goal, like your goal with doing this kind of incident review should be to learn, not to report, not to make action items, not to guarantee it will never happen again.

9:53Your goal should just be to learn. And then all those other things kind of fall out of it. So drive the learning culture first and figure out the right way to approach that for your team. Yeah. Instead of trying to force it in iteration immediately and say, oh, well, here's the action item. Let's yeah, let's expand on this. Let's dive in and learn and get people's expertise to your point. And then we should naturally iterate off that. But it's not going to be something you want to force. It's kind of what I'm hearing. Yeah, it's not something you want to force and you actually when when people are just given the space to learn you actually come up with better action items so in my talk today i showed a real post-incident review and like the action items were fix the alerts fix the tests which like that's what happens when you do a shallow incident review and that's what happens when the goal is to report and not to learn it's a bandaid and you could put those action items in any incident that has ever occurred in the history of the software industry.

10:51Right. And so what value is that, you know, and you're actually just wasting cycles and spending more time when you're really focused on action rather than focused on learning and focusing on learning pays dividends in the long run. It helps you build more expertise in your organization. It helps you create an organization that's curious and asks questions rather than keeps information to yourself, it helps you run more efficiently too, right? You don't end up over hiring and over bloating because people aren't aware of how they're all coordinating and you end up focusing on some of the right things.

11:27It also helps you retain your best employees. If people are feeling listened to and have forums to talk about various things in a structured way, it helps you keep them, the right folks that you do want to retain. I totally agree with you about that learning culture, Nora. And it's obviously something that's talked a lot about in a personal sense of like lifetime learning and the value of the importance. But it doesn't always translate to organizations, to your point. Right. And I know, you know, we mentioned some like semi-controversial opinions at the start of this. I know that you have a lot of beliefs about how organizations can take advantage of some opportunities to learn.

12:04And I know one of them is that you believe we need more chaos engineering, which is a great phrase, which is always fun. And let's dive into that. Why more chaos engineering? How does that relate back to, you know, incident manager or anything else? It's a good question. And I think, you know, and I got feedback actually from one of our customers, Jason Copy. He was like, you know, when you originally tried to sell me jelly, I was like, she's the chaos person. Why is she trying to sell me this incident platform? And they're a very happy customer now. But it was like, I think initially a thing that felt different, felt like a departure and incident analysis and incident management is actually an evolution of my thinking around chaos engineering.

12:45You know, I started my career always in risk and always in helping understand incidents and the quality of the situation, depending on the product need at the time. Right. We're always making tradeoffs to hit deadlines. And I think there could be a lot more. I think there could be more beneficial organizations if people were being proactive about learning from incidents and maybe purposefully injecting failure. But my thoughts have evolved over time. Like rather than injecting failure into your systems, you really should be looking at the incidents that you've already had. You know, I saw a lot of people taking that advice I gave like, and I think it was in 2017 to do more chaos engineering, but I saw it in the absence of looking at the incidents they already had.

13:31So they were like injecting chaos to like learn from things and see how things reacted. But the hard part was, it's like, there are a million ways you could inject chaos in your organization. You could do a thousand different things. Are any of those actually going to be useful, right? Is it useful to fail a data center and see what happens? Is it useful to maybe take one of your devs off the on-call rotation that everyone relies on? There's a million different things that could happen, but the question is, is it useful? And how do you know what kind of experiments run. And the way you can learn what kind of experiments run, if it is useful to do that on-call thing, if it is useful to take down a data center, is to look at your previous incidents and to look at, and not just incidents, to look at your surprises, to look at the things that made you go, phew, like those are good situations to maybe inject chaos.

14:21But what I was seeing the industry do was to take that advice and just do any random thing, right, without actually, it was just more based on a gut feeling, which those conversations can be useful, but they're really useful after an investment you already made after something you already spent all this time paying for, which was which is an incident, you know. Was it this evolution of thought around chaos engineering and incidents and what you should be evaluating that led you to found Jelly? Yeah, totally. I got hired at Netflix in 2017 and my title was chaos engineer. And I was building a platform that allowed people to automatically inject failure into production without actually impacting the customer experience.

15:06So we would have like basically two clusters. You know, it was kind of like an A-B test and you would put most of your people into the to the non-experimental cluster. And then you would put, you know, a very small percentage of users into the experiment cluster and you would create a failure. And the hypothesis was always like, oh, if I take this down, it won't impact. It won't impact these users. They'll have the same experience as these users, right? You wouldn't run the test if you thought it was actually going to fail. But obviously, sometimes we learned that, you know, it did fail and they did have a bad experience.

15:40And a Netflix bad experience meant like, oh, I can't press play or I can't, you know, search for my movies or I can't stream something at this particular time. And so we learned a lot about how our users use the system. But that learning ended up being confined to like the team of four people I was on. And none of us were running production systems. We were making code to learn these kinds of things. So dissemination wasn't happening to the rest of the org. Yeah. So it became really important for us to educate people. And then I started thinking, well, the best education is going to be if they do it themselves, we shouldn't be.

16:15I shouldn't be running chaos experiments on search or bookmarks, you know, like it's going to be most useful if they can experience. and see it. And they were all like, I don't even know what to run an experiment on right now. I could, you know, do something related to the code I'm working on right now. And so I started looking at their past incidents and I was like, what was some what was an incident in your head that went badly? And then I started talking to them about it. And my whole goal was to drive them to use my chaos tool more. But then I started realizing that there was so much other data there and how they worked.

16:46And that was sort of like how the evolution came. I went to Slack after I started doing incident analysis at Netflix. I actually spun off from the chaos team and formed a separate team with one of my colleagues, Lauren Hochstein. And then I got hired by Slack around a time when they were IPOing. And when a company has a lot of PR presence, maybe they just raise money, maybe they just launched a feature, maybe they had a big outage, maybe they're hiring a lot, they end up usually having a lot more incidents too. And so I got hired at a time when Slack was having a lot of incidents. And I was running around like, you know, trying to help people learn from incidents, trying to create this dissemination so everyone could learn from it.

17:30And I realized there could be a product in the market for this to really help orgs do this better. So you're a couple of years now in your founding journey with Jelly. And I know you've kind of taken the journey that a lot of software engineers think about, which is, hey, I'm going to go learn at these innovative companies. I'm going to go do big things. And then I'm going to say, OK, here's a problem that I see. Let me jump in and be a founder. Now that I've gained this experience and have this skill set, what have been the learnings for you as a founder? I think one of the biggest learnings is creating this learning organization myself.

18:02Like I've always been creating one from the ground up, you know, as an individual contributor, as a leader of a small team. But now I'm creating it, you know, I'm enabling it from from the top down, too, and like allowing it to be enabled from the bottom up. And so it's a lot of like, I think, putting my money where my mouth is, too, you know, just like making sure I'm doing all the things. And also it's taking into account the phase a company is at. I think that's been a really big learning, too, is like the type of learning from incidents you do is different depending on where your organization is at and your journey to product market fit, too.

18:38How do you think about that differentiation depending on company size or journey? Yeah, I mean, so before we had a product in the market, there were four of us on the team. And there were like certainly surprises and certainly ways we surprised each other, but we had no users, right? And so there was really incidents. And so we had to take some time creating that culture of like asking questions and sharing information and documenting things in a way that people actually wanted to read in absence of user facing issues. So we kind of got to practice it. And then, you know, one thing I usually recommend for folks is to not have the person most involved in the incident run the incident review, because like I was telling earlier, you need to be able to ask them.

19:22It's hard emotionally. Well, and you need to be able to ask them the silly questions like someone that was in the middle of the incident is really bad at telling you what happened, actually. like they can tell you what happened in their own mindset but in order to really get the data out of it that's gonna improve the organization you need someone else asking them questions and so in a perfect world you have someone that was not involved in the incident but when you're a small company hard to do literally everyone's involved in the incident so we had to get really good at like playing these third-party roles taking the person that was like least involved in the incident and also you know really making scheduled time for learning i tell orgs this all the time like it's It shouldn't be easy to schedule over a post-incident review.

20:05Like you should prioritize it because if you're not prioritizing those things, it will come back to bite you. It's not in the next week. It's not in the next two weeks. It's like for years to come, it impacts your culture slowly but surely because you are still taking action on those incidents. But there's been a lot of ways that my mindset shift. I'm not coding anymore. I'm not on the ground anymore. I'm like, I'm leading folks to build this product that I genuinely really believe should be ubiquitous in the industry. Has that transition to not coding on a regular basis been challenging? I started transitioning into that when I was at Slack.

20:44And so I think what is what I miss the most, I do miss getting in the technical weeds. I miss system design. I really liked that. and I miss actually doing long form incident reviews. I actually, I have a lot of fun. I still get that, you know, I get to do it sometimes. I get to do it with some of our customers. Occasionally, they'll let me sit in on an incident review and I'll get to kind of play that role a little bit. Like that can be really fun, but I'm getting really, like it gets really exciting to see those light bulb moments go off for our users now. You know, I'm getting my dopamine hits in my career and like in a different way, you know, I'm not doing it from code or like writing incident reports anymore.

21:27I'm getting to see other folks do it and enabling that to happen, which is it's really it's really cool and different. Are there particular incidents that Jelly has already learned from internally that you could maybe talk about? So, so many. We have them all the time. And I and I tell orgs like you should actually over index on calling incidents. You know, I talk a big warning sign is when I talk to a leader and they're like, we don't have incidents. We actually don't have any incidents. You'd be surprised. I hear this from like very large companies that everyone uses. And I'm like, and that's good.

22:02I'm like, I know you have more incidents than anyone else. And incidents are surprises. Incidents are anything that pulls you out of your day that you did not prepare for. Right. That like those there are always things to learn from. Um, but I say all that, even though we're small and like, you know, we're still newer as an organization, we still have incidents and we, we talk about them all the time. We're doing learning reviews every week right now, actually. And we have one, I think in an hour that I, I'll probably dial into, but, um, we had a unique one this weekend related to, to SVB going down.

22:36That was a, that was a really different type of incident. We're going to get in here. Okay. That wasn't a technical incident, but it was, it was an incident. I had to jump in a situation room. We had to figure out things to do in the moment. It was pretty wild. And then we've had kind of your standard ones as well. We've seen issues in our demo account versus our production account. We've seen issues where core features of our product are not loading. And we take the time to understand how it unfolded. And then everyone on the call, regardless of if they're an engineer, actually starts to understand how the system unfolded.

23:12And who I've really seen it be helpful for is actually customer success because they know when issues are happening, they can actually start to understand how they might have happened and be able to triage a little bit. I like that you've widely defined incidents here. Yeah. You're like, look, our banking system being an issue is an incident. It's not just our own internal technical challenges. It's what are things that are affecting the business? What are things that are affecting our user experience? Can you share a bit about your approach to the situation room and my crisis like that, whether from the SVB incident or something else?

23:47Yeah, I mean, the SVB incident was like very, I think, very different than I would have handled a lot of other ones. But it was also very similar. You know, it was like we were in focus mode. We didn't have time to like pepper things on. And like when you're in the middle of an incident, you have to be very direct. you have to make decisions quickly and you have to make decisions in the absence of the full scope of information and that was what we were having to do this weekend and you know ultimately like with the news on Sunday night you know everyone has had access to their funds but up until Sunday night we didn't know if that was going to be the case yeah and so I was having to make all these contingency plans are we going to make payroll right can we keep our service running?

24:30Do we need to raise a down round suddenly? Yeah. Like how, how, what kind of steps do I want to take in what order? What is up? What is plan A? And then what do you know, what does plan E look like? Right. What's the worst case scenario here and really like plan that out. And I was also like having to communicate with my stakeholders, which in this case was the team, you know, and in usually usually in incidents, it's the team communicating with me or someone from sales. Right. And so it was a really unique and opposite incident. But I was like, hey, here's what we did today. Here's what we know.

25:06Here's what we still don't know. And here's what we're trying to do. I'll update you tomorrow at this time. And so doing things like that, I mean, it was it was nice to like be able to get in that role again and actually just, you know, understand that perspective. And I felt prepared to work in the incident because it was an incident, but it was it was definitely a role reversal for sure. I can imagine that that kind of role reversal in a situation where it's a fairly unique incident. Yeah. Must have been really hectic. Are there any particular learnings you took away from this unique type of incident that you see yourself applying either as a founder or leader moving forward.