Troubleshooting Microservices with Julia Blase

0:00A distributed system is a network of independent services that work together to achieve a common goal. Unlike a monolithic system, a distributed system has no central point of control, meaning it must handle challenges like data consistency, network latency, and system failures. Debugging distributed systems is conventionally considered challenging because modern architectures consist of numerous microservices communicating across networks, making failures difficult to isolate. The challenges and maintenance burdens can magnify as systems grow in size and complexity. Julia Blaze is a product manager at Chronosphere, where she works on features to help developers troubleshoot distributed systems more efficiently, including Differential Diagnosis, or DDX.

0:44DDX provides tooling to troubleshoot distributed systems and emphasizes automation and developer experience. In this episode, Julia joins Sean Falconer to talk about the challenges and emerging strategies to troubleshoot distributed systems. This episode is hosted by Sean Falconer. Check the show notes for more information on Sean's work and where to find him.

1:18Julia, welcome to the show. Thanks, Sean. So nice to be here today. Yeah, absolutely. So I wanted to start off digging into your background a little bit. Can you talk a little bit about your journey into the world of microservices, observability, what led you to Chronosphere, and why you're interested in these issues around troubleshooting? Yeah, absolutely. Well, I started out as a librarian, actually, maybe not the most traditional career path into tech. I worked at the Library of Congress. I got a fellowship there. I went from Library of Congress to actually working at the Smithsonian. I will say it was less like what you think of as traditional librarianship, kind of written word librarianship, and a little more digitally focused librarianship.

2:01I was working with scientists and researchers, and I was helping them store and organize their data so that they could ask questions of it and get the answers they needed, going from information to insight, as I used to say. And then I was working in DC at the time, and I did eventually move over to work at a company called Palantir. It was maybe less well-known in 2014 than it is today. But the reason I moved over is, at least at the time, they really talked about their software as a fundamental tool to help the government do something very similar to what I had been doing as a librarian, right?

2:32That is like understand, organize, analyze their data in a central location with a central toolkit. And I think government agencies faced really similar challenges to those faced by the scientists I had been working with, which is the data was stored in silos and each silo was organized differently and you had different tools to work with each silo. And very few people really had been putting in the manual effort to understand how to work with all those different data silos and get that data together to provide insight. So you can probably see kind of where the through line is, my information to insight role as an individual contributor to going to work at a company where that seemed to be their whole purpose.

3:08And it was really exciting. And I really kind of enjoyed that path from librarianship into tech. While I was at Palantir, I started out in again, that government facing side of the business in a customer facing role. So the first time I actually engaged with observability, I was actually what we would say is high side. I was in a customer secure computing facility. I had been on call. It was late at night and you know, our developers for that software, they weren't always able to get on those government sites, right? They weren't always able to come out there and actually get hands on keyboard to see what was happening when something went wrong.

3:39So they would rely on people like me to kind of sit at the computer, be on the one phone line that could connect to the outside world and be their hands and follow their instructions. So I think my first engagement was, hey, I need you to grep for something that looks like this. And I was like, cool, what is grep? I don't know. So they really walked me through what it means to SSH sin somewhere, what it means to grep, what a log is, what a metric is, how to describe what's on a metric dashboard so that they can kind of guide me through what else to look for to help them diagnose the problem. And it was really interesting.

4:10I really enjoyed engaging with that side of software, and it really demystified software a lot for me, which I appreciated. So as I spent time at Palantir, and as I grew and the company grew, I actually moved into their product org. And that's where I started learning about the difference between sort of monolith and microservices and on-prem infrastructure and containerized infrastructure, because I was working with teams that were doing both, some that kind of started natively building their services in Cates, others that had built services in a monolith, and then were kind of trying to migrate them over to work in a more containerized environment and split that up into microservices.

4:42And it was really challenging, you know, and there were challenges on both sides. And I enjoyed helping people with those and kind of working on those challenges, which brought me to Palantir's central observability team, which at the time was called their signals team. And that was the whole purpose of that team. So on that team, our challenge was to take all that telemetry data from all of the software, whether it was on-prem or commercial cloud or GovCloud or SecureCloud, monolith, microservice, whatever it was, and kind of develop tools and methods to bring that data into a central place where they could use it to troubleshoot issues.

5:15Of course, that did bring me to Chronosphere, I think, pretty naturally. We actually interviewed Chronosphere as a vendor at one point when I was at Palantir in that role. And they were just honestly some of the most transparent, expert, nicest vendors I had ever interviewed. And I was just like, this company gets me. They really understand my problems. They understand my engineer's problems. Palantir at the time, I had been there for six years. It had gone public. I felt like I had kind of reached the end of what I had wanted to do at that company. I was looking for a new role and Chronosphere seemed like a natural fit.

5:45Awesome. Yeah. I mean, I actually think that the path from librarian to product management, you're working in data, working in the world of microservices and stuff like that, is not necessarily that like crazy a path because if you think about some of the work that you know has come from sort of organizing books in structure taxonomies that's probably the original inspiration for like ontologies and things like that which also then leads you to palantir where they're you know big proponents of ontologies and done a lot of work in that space as well it's kind of a lot of the like foundations of what we think about databases and stuff probably came from being inspired from the way that we thought about organizing books in libraries.

6:25Absolutely. I like to say librarians have been organizing data for 2000 years, right? Like these are not new human problems. These are problems that we've always had and we've always tried to develop tools to try and fix. And data is just another form of information, right? It's different in size. It's maybe different in complexity. It's different in kind of the pace of change, but yeah, you're using some of the same foundational principles, like how to store it. What's the best way to store this for access? What is the best ontology and way to organize it? What metadata do we need to index, right?

6:53Like that's the Dewey Decimal System is a way of indexing metadata so that you can find things quickly, right? It's all related, yeah. And for our younger listeners, you'll have to go and do a Wikipedia search on what the Dewey Decimal System is. But in terms of like these problems around data silos, it's kind of interesting. Like we've been working on that problem for probably as long as humans have been writing down and storing information. And that problem, it doesn't seem to be going away. It's like we're actually seems to be getting worse, if anything, because we have more and more data to manage.

7:22What are your thoughts on taking a step back and just looking at that challenge? How do we make progress to kind of breaking down some of these silos? Yeah, I think it really is about understanding the needs that we have from that data. So what questions do we want to ask? Who's asking those questions? How fast do they want to get answers? Because if we start there from kind of a need-based approach to how we want to organize this data and reference across this data, I think we're going to be able to organize and break down those silos faster for the right people. So that's probably also the product side of me talking, right?

7:54I'm always talking about what's the problem we're trying to solve. If the problem you're trying to solve is organize the world's information and put it into a single place so that everyone can use it, that's massive, right? Like you'll be working on that forever. You will never be done. There will always be something. Yeah, Google's been working on that for 30 years. Right? That was their original charter, organize the world's information. Do you go to Google and always find exactly what you need right away? Maybe, maybe not, right? It's a hard, hard problem. So I think you really have to start with like, what's the outcome we want?

8:22And how do we build tailored solutions to work across the relevant data for that outcome? Less of the full-scale approach. We're always going to have all the data in one place. We won't, but we can find better ways to let the right people access it in the right way. What are your thoughts on some of the challenges that things like microservices introduced? Like we had the monolith, three-tier architecture. There's a certain simplicity with that, but of course we run into some scalability, both from just an engineering perspective. If everybody's working on the same code base, it's all part of this large piece of software that gets deployed somewhere.

8:56It can really slow things down. We break it apart, makes us more agile, more nimble. But then we also, as we break it apart more and more, we potentially introduce a lot of challenges from a distributed systems perspective of just like brittle infrastructure for requests and responses. You know, one part of this like complex mixture of dependencies goes down, the whole thing goes down. What are your sort of thoughts about having worked in that space for a while? Yeah, a couple of things. The first one you mentioned is right with microservice. I think one of the biggest problems they introduce is where is the problem coming from?

9:28I'm going to reveal my age again here, but I think in the past, right, if you heard a phone ringing, it was like, well, it's one of three phones in the house because they're all connected by wires to the wall. And that was kind of your monolith focus troubleshooting. You kind of knew where the problems were likely to occur. You had a pretty good understanding of the system. Now, if you have a problem in a distributed system, it's like when I lose my cell phone, right? And I ask the Google to ring my phone to help me find it. And now I have to figure out where that ring is coming from. And it could be anywhere.

9:54It could be the kitchen. It could be in the refrigerator. True story. I've done that. It could be in the car. It could be in a friend's house where I was two weeks ago, right? Right? Like microservices introduced the problem of, is it my microservice? Is it your microservice? Is it my dependency on some other internal team? Is it my dependency on my cloud provider for some kind of autoscaling infrastructure that I don't even understand how I interact with it? I just send the request across the API and hope it works. You just have so many more places where that could be coming from. And I think that is a huge problem with microservices is it introduces a first order problem of where did it actually happen?

10:27Where did it actually start? And I think to add on to that, Not only is that the problem, but you're having to dig through so much more data to find the answer, right? Because you're running containerized infrastructure. The whole point of that is so that you can auto-scale, right? I can scale up when I have more load. I can scale down. It should save me money, should make things easier to deploy, to run, to change over time. That also means that all those deploys and scales and runs and changes are introducing more data, more data volume, more data complexity, right? You're going to have higher cardinality, more interesting facets of data that you're having to kind of rule in or rule out where you're trying to navigate to find out where things are coming from.

11:03And so you have that kind of compounding problem of could be coming from a thousand more places than it used to. I have a lot more data about each of those places to dig through than I used to have. And I would almost say, this is kind of my own conclusion, but looking at those two things, the other problem I think microservices introduces, we rely on fewer and fewer people at our organizations to really understand and be able to solve that problem of where the trouble is coming from. So I think you get this hero, right? And I saw this at Palantir. I still see it at my current employer, although we're really doing our best to get out of this with the different tools we're developing.

11:38And I think we're making progress. But yeah, you have your organizational heroes. And for someone listening, if you're working at one of these companies, like thinking in your head, like, who do I call when the incident's really bad? Probably four names on that list, right? And that's a problem. And I think that's really the ultimate problem that microservices have introduced is you're over-reliant on having the right people in the right incident room at the right time to fix a problem. And that's extremely brittle. What if those three people win the lottery, right? Like now you're out of luck and you don't have anyone who can solve that problem.

12:10And I think that's like a huge risk in companies that are running these microservices today. Yeah, this idea of a hero, for instance, has come up before in other interviews I've had with people who work in this space. So I'm not surprised to hear you talk about that. Like, that is a huge, huge challenge. Like, essentially, you have with an incident, like a bus factor of one, maybe three or three. So how do you fix that? Like, what can organizations do to help solve that problem? Yeah, I think the first thing you have to do is you have to know what data you need and what data you don't. because we talked about that sort of data explosion just now.

12:43Not all of that data is necessary. And especially working at the company I work for now, I see customers consistently reduce the data they actually store, the data they have on hand to solve an incident by 60%. That's a huge number. And that vastly simplifies the troubleshooting process. If you can just get rid of some of that noise so that every time you're facing an incident, you're working with a much smaller, more relevant data set. The other thing organizations can do, I think, is to make that data accessible without requiring expertise in the tool. I was actually talking to a prospect, and he said he used to work at Google.

13:17And he was like, the tools that our SREs built to dig through our data were terrifying, right? You build super complex things that only a few people understand. Or maybe if you don't build things in-house, you purchase something from a vendor, and it requires you to learn a new query language, right? It doesn't have to be in-house to be complicated to learn. If you're having to learn new query languages and you have tools sprawl. You have some in-house, you have some things you purchased from a vendor, right? And now you have to learn each one of those tools. You're just going to keep running into this problem.

13:47So I think understanding what data you need and don't need and trimming it down to just what you need. So you have a strong starting point, reducing your tool sprawl or focusing on tools. Maybe you do still have two or three, but tools that are really easy to walk up and use. You don't have to spend a lot of time learning how to work with the tool, right? The tool is kind of walk up friendly or built for kind of your knock or your novice user. And I think then the other thing is you just have to have those tools that are also built to handle change because your data is going to be changing all the time.

14:16And that's where the expert often is relied on because no one remembers what happened two weeks ago. No one knows how that dependency arose or like why it's calling this thing anymore. So you have to have tools that can help you kind of understand the history and the context of where you are right now so that anyone can walk in and help fix. I don't think heroes are bad. I just think I'd rather invest in heroes as people to really dig into the root cause after the fire is put out, if that makes sense, right? Like let's make it really easy to put out the fire, get back to normal functioning, then bring in those experts when they have a little more time to go deep, let them dig in, like let them ask complex questions, but use them in that way where it's less brittle and less something that you're depending on for your entire business revenue to keep working, right?

14:56Yeah. I mean, I think the challenge with the hero approach is that it's just difficult to scale, right? Yeah. If someone's unavailable or whatever it is, you don't want to essentially be in a situation where you can't solve an incident because someone's on vacation. Right. I feel like in a lot of ways, because of the challenges that microservices distributed systems bring in, there's been a lot of brainpower that's been put into, how do we improve on grep and logs? How do we basically put a nicer software over top of being able to do distributed grep? And I think, you know, you're totally right.

15:30The hero problem, not only is it hard to scale, it burns people out. So, you know, that's another thing that like in general, it's just a human. I don't want to burn out my friends, regardless of whether or not they scale. I don't want to put that pressure on them. But yeah, nicer tools, nicer tools are great. Tools that don't require you learning a query language are great. I think there's also something here about learning to use different types of insights together to give you a holistic view where you can kind of lean in and say, like, what can I learn from my metrics? What can I learn from my traces?

15:58What can I learn from my logs? How do I use those three things together instead of always going straight to grep for logs? Can I actually use my metrics or traces to find a signal earlier, locate the place it is? And then when I grep for logs, I'm grephing through a much smaller portion, right? Like I know kind of exactly what I'm looking for at that point. So it's easier to find. So I also, you know, we think a lot about what are the right purposes of each of these tools? How can we combine those purposes to give you a faster to learn, easier to use experience? So you talked a little bit about this idea of essentially cutting down the amount of data that you're storing so that if you can cut things down by 60%, then it's just easier to essentially deal with that volume of data because you're probably going to have less noise to signal ratio.

16:40How do you do that? Like, how do you determine how to cut out 60 % of this data? Because I feel like when it comes to things like telemetry, logging, monitoring, people are just like, more data is better. You know, I don't know what I'm going to need. So I'll just like keep everything essentially. Yeah, the pack wrap problem is something that we hear customers talk about. Everyone wants to keep everything. A couple of ways we think about this. First of all, we think about what you're already using. What is in a monitor? What is in a dashboard? What are people searching for today with the query tools that they have?

17:08What are your service accounts calling for repeatedly to do whatever financial analysis they might be doing on the back end? That's a great indicator of what data you need. And honestly, that's a small fraction of your data. It's shocking how small a fraction of your data that is. You can also kind of look at that and look at best practices. So like dashboard templates that exist out in the market for monitoring Cates. Those are pretty well understood problems, right? You know, you're going to need your container metrics and you know, you're going to need them to be at this kind of interval and in this kind of summarized way.

17:35That's pretty easy to kind of look at that, look at what your people are using and combine that to get a really good understanding of, Hey, what do we need that we know we need? Because people are looking for it. What do we think we should have? Because this is what the market is saying. Let's put that together. let's see how much of my data fits one of those definitions. And everything that's not in that definition, let's throw it away because there's no utility for it today. Now, I think the other piece of this is I mentioned a data change, what you need might change over time. So you also need tools that let you change what you're collecting or what you're storing very dynamically.

18:06It shouldn't take you, you know, re-instrumentation and redeploy to change what you're collecting. It should be something that you can do from a central location when you start to see something new. So you have a customer, they have a metric, no one's ever used the metric, they drop the metric. Next week, they see a ton of queries against this metric and someone's setting up a dashboard and those users are reaching out saying, hey, I can't find this metric. And you're like, oh, I didn't know you need it. Now, if I have a tool that can just say like, switch it on, you have it tomorrow, people are going to be a lot less worried about dropping data today when they know they can get it back as soon as they need it again.

18:37And I think that's kind of the maybe two-sided way of having this problem is first you look at what people need, trim down to just that. And then as those needs change, have dynamic tooling that lets you sort of adjust that collection to meet what people need right now. Can you talk a little bit about Chronosphere's differential diagnosis and how that potentially relates to this problem that we're talking about? Absolutely. So differential diagnosis is inspired actually by what we saw these heroes doing when we went to talk to our customers and said, what's your process when you call in the hero?

19:06What do they do. And it's based on their diagnostic process. So DDX does basically does what they do with one click. It takes all of the data about the thing that's having the problem, right? So I've narrowed it down to this particular endpoint in this service. And it says, cool, what data do we have about that endpoint in that service? What are all the facets of that data? Let's take those and let's split them up into piles. Let's look at things that are bad and things that are good. And let's compare those two things. So bad, good, you've got errors, you've got successes, you've got really high P99s versus really low P50s.

19:41DDX sort of does all that split up and dimensionality analysis for you and just presents you with the results so that you can start to find outliers, because that's what these heroes are doing is seeing what's unusual about the things that look bad, right? How does that compare to the things that look good? What can I change? Like, how do I make that bad data look like the good data? And DDX does all that for you in one click. I think really the power of this and how this works with all we've just talked about is it's doing that analysis on a really good set of data, right? Like we know this data is relevant.

20:10We know these facets are valuable. We're not doing it on a big pile of who knows what that's probably full of noise. We're doing it on data that's hopefully already high signal so that you can trust the results. I'll also say one thing here. We haven't talked about this yet, but everything we do with DDX is transparent. So we kind of present you with user results. You can choose to act on them very quickly, right? The machine does the pattern analysis on the high-scale data. You bring your human knowledge and context to that and say, yep, that build version, I did forget to deploy that to Japan, real-life example.

20:41I need to go do that deploy and everything will go away. But you also have to trust the system. So anytime you're doing a human-machine partnership like that, you really want everything the machine is doing to be incredibly transparent and verifiable by the human. Because I think we as humans, the first time a system says something looks funny, we say, are you sure? Let me check. I don't know. I don't trust this. So the other way I think DDX works with all that we've talked about is it makes all of its results verifiable by the humans that are actually working with it so that you can trust it over time and learn to maybe let yourself lean on it a little bit more.

21:14Let your novice users bring you conclusions from that because you know, as the expert, you can always go in and verify their results if you're suspicious and it lets you leave the room. It lets you leave the room and kind of leave that diagnostic work to maybe people who are less familiar with the system. So you talked about how some of the inspiration for this kind of comes from like looking at what the hero is doing. Is there also an inspiration from like the world of medicine? You know, if you look at differential diagnosis there, like that's about distinguishing diseases from others that have like similar symptoms.