Prometheus and Open-Source Observability with Eric Schabell

0:00Modern cloud-native systems are highly dynamic and distributed, which makes it difficult to monitor cloud-native infrastructure using traditional tools designed for static environments. This has motivated the development and widespread adoption of dedicated observability platforms. Prometheus is an open-source observability tool designed for cloud-native environments. Its strong integration with Kubernetes and pull-based data collection model have driven its popularization in DevOps. However, a common challenge with Prometheus is that it struggles with large data volumes and has limited cost optimization capabilities.

0:35This raises the question of how best to handle Prometheus deployments at large scale. Eric Chabelle works in DevRel at Chronosphere, where he's the Director of Community and Developer. He also is a CNCF Ambassador. Eric joins the show with Kevin Ball to talk about metrics collection, time series data, managing Prometheus at scale, trade-offs between self-hosted versus managed observability, and more. Kevin Ball, or KBall, is the Vice President of Engineering at Mento and an independent coach for engineers and engineering leaders. He co-founded and served as CTO for two companies, founded the San Diego JavaScript Meetup, and organizes the AI in Action discussion group through Latent Space.

1:16Check out the show notes to follow KBall on Twitter or LinkedIn, or visit his website, kball.llc.

1:35Eric, welcome to the show. Hey, thank you very much. Nice to be here. I'm excited to get to talk with you. So let's maybe start out with a little bit of introduction about you and what brings you here and maybe start touching on Prometheus and Chronosphere. Sure. So my name is Eric Chabelle. I work at the Chronosphere and Observability Company. I have a position that has a pretty long, weird title. So I generally just introduce myself as the director of evangelism. You recognize in a startup, that stuff gets pretty fluid. Every year we want to reset the targets, reset the focus, you know, so things are constantly growing and evolving in different directions and you end up getting more and less and other things under your umbrella.

2:13So I do a lot of stuff around DevRel, basically, in the observability space. It's a good way to put it. Awesome. Well, let's maybe then talk about what is Chronosphere and what is Prometheus, which we came on to talk about. Yeah. As per me, this is the topic we decided to discuss here. I'll kind of twist it in that direction. So ChromeStreet is an observability platform. It's a SaaS offering that ingests open standards. Cloud Native Computing Foundation is something that me and my team are a big part of. Two of us, including me, are ambassadors of the CNCF. We like to help out, contribute, and talk about, and do everything we can to promote open source in the sense that that's, we believe, to be the best way to get started with basically anything.

2:53I've devoted most of my life in the app dev space to it first before I came over to the observability side. So it's a continuation of what I normally talk about. And in relation to Prometheus and OpenTelemetry and Percy's a new project and FluentBit, which is also something that we're very integrated with and a couple of the founder and a couple others work at the Chronosphere. All these things are the standard way of delivering and communicating over telemetry protocols, collecting, storing, querying, that kind of stuff is all integrated into our offering. So anybody that starts out initially at a smaller scale in the open source world, finds themselves growing and scaling and having more and more difficulties managing teams that are growing, that are spending all their time managing the infrastructure and trying to make it all scale.

3:38I find it quite nice to be able to unplug something like that and find a vendor with open standards and is easily able to simulate and recognize the query protocols or the query languages and the things that we use from Prometheus. Cool. So let's maybe dive into what Prometheus is. I saw it described as a cloud native observability platform. What does that mean? Yeah. So Prometheus as a metrics collection project is the best way to do it. That's one of the signals that you would want to collect in an observability platform. There's some very unique things about what they do. So this was originally written back in the day, I believe, by some people like SoundCloud.

4:18And the idea being that it has to be highly performant. It has to be highly scalable. And they want it to be as unobtrusive as possible. So one of the things that it does is it scrapes, which means it goes out and gets the data from an endpoint and doesn't require you to set up things to push data to it. It doesn't involve collectors. It doesn't have agents. It doesn't involve anything like that. They allow you to fine-tune that as you wish as a developer. So you can use auto-instrumentation, instrumentation they call it like you just flip a switch on a java library and it starts spitting out all kinds of java metrics usually not a great experience because java metrics are just a wash list of stuff that's just crazy so you want to get more specific so then you can use that library to trim that stuff down and deploy applications again with the proper instrumentation and they usually expose some kind of endpoint where they publish these metrics on and you set up prometheus to go out there and scrape these endpoints every so many seconds, 10, 5, 15, whatever you choose to do.

5:18You're also dealing with the, you need a backend for this. By default, if you're just playing around with Prometheus, it just does it in memory, but you really don't want to do that in real life. So you have to put some kind of backend storage in there and that's a time series, what you're storing. So normally in a database query, you're, you know, it's a finite set of whatever that you're collecting, right? Select star on whatever table and get this stuff and get a bunch of data. Here, you're basically trying to represent constantly collecting data on a machine. So for example, CPU usage, constantly every second of what's going on is milliseconds or whatever is a little bit hard to store and would probably overload almost any organization's network in a matter of no time, especially when you're doing it across thousands of machines.

6:02So they have a way of sampling the data and then re-representing that when you do queries. So that allows you to do time series queries and figure out what's going on and create the dashboards and all that kind of stuff. It also provides alerting mechanisms. So it has the ability to set up alerts and rules around that kind of stuff that then triggers and sends off and can integrate with things like PagerDuty or send you Slack message or whatever it is that you're using in your org. The query language is also pretty much a standard in the industry. It's called the Prometheus query language, PromQL.

6:33Doing that, you can generate database queries basically against the stored metrics data and create visualizations. And there's an embedded dashboard inside of Prometheus, but that's not really generally how people do that. They tend to query with an external dashboarding tool that used to be centralized an awful lot in Grafana, but there's up and coming projects. The very first visualization and dashboarding project just reached sandbox status this last fall. And it's going to be at KubeCon EU here with one of the Project Pavilion little stands. It's a Prometheus project. and they're basically focusing first on the metrics queering through prometheus instances they're starting to expand into traces and some of the other stuff it's a very young project yeah but that's pretty much the infrastructure what you're going to end up with the prometheus i'd love to dig in on a couple of different pieces there so sure first talking about the sort of piece by piece you talked about one of the things that's distinct about prometheus is It is going out to find the data and things like that.

7:36Can we talk a little bit about that? I saw, particularly in the cloud-native environment, there was some stuff around service discovery and how do you find things within Kubernetes to go and look things up. So how does that all work and what does it enable? So initially, some of the links I sent you, I assume you'll put them in the show notes, is around the workshop that we have. You initially start by doing that statically, just defining where to go, what endpoint, what IP address, where are the targets I'm trying to scrape. And that's fun and games when you have one or two things, but you don't want to be managing a list of thousands of those things.

8:08And let's be honest, Kubernetes is set up to be an environment that you write a description for and say, go out there and run these services. And when certain things happen, react this way. But that's about all the control you have. So you really don't know what their IP addresses are going to be. you don't know where they're going to locate and how many there's going to be. So they have something called a service discovery, and you can integrate various tools that do that. They support several different ones. Zookeeper is a pretty famous one that people know. And what that does is just monitor the space you're running your clusters in.

8:38And so when new nodes, new pods, new containers and stuff come up, it just automatically add those to the list and start looking for endpoints and scrape those also. And generally speaking, you've ensured that those are providing those endpoints. Otherwise, you're going to have a lot of down targets. And yeah, that's kind of how the service discovery works. That makes it all dynamic. And that's much more realistic for a cloud native environment. Yeah. So another thing that you talked about here, and you sort of alluded to, you said these are collecting time series, as contrasted to, you mentioned another project that's doing logging or tracing or things like that.

9:11Can you kind of maybe dive in a little bit on what is the distinction between like time series collection versus what you might get in like a logging or tracing type of observability. Okay. Yeah. So a little bit of the difference between metrics, logs, and tracing. So starting with logs, that's a very standard, it's like text lines, right? Going right down the chain, right? Just a bunch of data that you're storing there and it's log lines. That's pretty normal. Maybe you turned it into JSON with something like Flimbit that parses it into machine readable stuff, but it's still text, pretty normal storage.

9:43Traces is also something that where you're you're setting points in your service calls that you're catching and tracking as it goes by, which is also a pretty straightforward data kind of set that you're putting together. It's not a constant stream of stuff. And the metrics where they have to have a lot of really smart ways to deal with this because of things like cardinality explosions. So if you look at object-orientated programming and you make an object for a person and it has a name and it has a age, has social security number, has whatever an address. These are all metrics with labels. So your person might be the metric and the labels might be all these things underneath it.

10:21And if you happen to put something in there like the IP address from the machine that you're getting this information from, that's a unique thing. And every time this gets sent out, you're getting a unique thing you have to save in the database. That's a cardinality explosion. And the more these things can create that kind of stuff is where you get the mistakes where it gets really hard to create this kind of stuff. And in a time series, you want to be able to have a way to store this kind of streaming data and this massive amounts of data in a way that you can measure it periodically and just connect the lines between those short little periods is the best way I can give you an analogy around that.

10:58I guess then a question becomes, how do you decide when is it you want to use time series collection as compared to sort of full traces or something along those lines? Is it when you start? Well, I mean, don't get too wrapped up in the way it's stored. I mean, that's just time series databases are needed for metrics. They just are. So all of them require this. It's not an option that you can choose one or the other. If you're looking at the signal type that you want to collect, whether I should use metrics, logs, or traces, that kind of has to do with what you're trying to observe and what you're trying to watch.

11:27So if you want dashboards with visualizations on what my infrastructure looks like, you have things like thresholds you're setting and watching for things that get beyond the threshold. When you're looking at resource consumption, when you're looking at network lag and things like that, these kind of measurements only come through metrics. When you're looking to find out where did my calls go through my service network, that's traces. And logs are just dumping information that every developer put into his app. Every application that's running on that machine is logging something, right? It's putting messages in the log that I started up that I'm having trouble, that I'm dying now, or whatever the deal is.

12:08It's three distinct signal types that you're using. Yeah, that makes sense. And each one is probably stored a little different. Yeah. Well, and part of why I'm asking this is I think metrics are much more common in sort of operational monitoring and things like this. And a lot of software developers are thinking, oh, I want my logs so I can debug or trace. So kind of translating that mindset into what happens here. So let's maybe look then a little bit further. You have your metrics. They're stored in Prometheus and you're starting to want to query them or think about them. what types of questions you've already started doing this are best answered by this data and like what does the query language look like to explore them i think you've kind of alluded to it there at the end so i think a lot of devops modern organizations are no longer in that boat where developers are just like i want to see my logs that was when i was a developer that was like 15 years 20 years ago right we used to spend i used to make jokes about this with a couple guys here because i do quite a bit of stuff and flew a bit now and it's a lot of logs and stuff we're were simulating and dealing with.

13:06I used to spend time at the coffee machine because the machines were slow enough to compile the Java code we were writing that you had time to go get coffee and talk about problems you were having. And one of the things we used to discuss quite a bit is like trying to figure out how to make a good Java exception message for the logs. Because some humans reading that downstairs on the third floor, right? That's your ops guy. Now that doesn't work that way anymore. That's not really, really what you're focusing on. And that complete picture of your cloud-native environments are deploying apps and services.

13:33Your team owns a service, and you want to have a visualization of that service and everything about it. And luckily, we have a lot of organizations dealing with things like platform engineering, SRE teams, and all this different stuff that have a really good idea of how they want to do this and provide a developer view, provide an SRE view that's carrying the beeper that has this stuff and know who to call if something breaks. But when you own a service like that, you're not so much digging around and looking at individual metrics, you're having a predefined look at your running object, element, whatever you service that you own.

14:08And that's your landing page. That's your art monitor for what you own. And that's telling you, depending upon what this organization has gone through, they pretty much have a good idea of what they want to know. If I'm the order service, I want to make sure nothing's blocking orders. That's where the money comes in the retail org. So they'll be watching really close the connections to whatever credit card companies they're using or maybe the incoming stuff from the website and something starts slowing down orders and they see that little thing go all the way down like this and maybe they're going to say, hey, you know, it's time to look at our dashboard and they should be able to really quickly find out where's the problem.

14:42Are the orders not coming in? Are they not being processed out? Is the timing out going to the credit card company? You know, that kind of stuff. Yeah. So I think what you end up is with having their own view of their world, right? And that's a combination of all three of those things. not necessarily just metrics or just logs or just trace not saying that that doesn't exist but it's generally you don't want to set up your dashboards with here's my metrics here's my logs and here's my traces and then when sre's beeper goes off he or she is like let's look at a log let's look at no what you're trying to figure out is a high level view you can drill down into to get to the problem so with prometheus then prometheus is still just providing one of these pieces, which is the metrics tracking.

15:24And I think as you highlighted, that's probably why people are then setting up queries to query against it from an external rather than using built-in visualizations because they want to integrate this with their logs, their tracing, things like that. What does that end up looking like for someone who hasn't, for example, written something in PromQL? Like, what is this as a query language? What does it look like? And how does it work? I'm going to say right up front, it's not easy. This is something that a lot of people struggle with, including me when I'm writing labs and digging around and trying to find the right thing to give an example of.

15:57But it's within the workshop that I alluded to. There's about four chapters that are involved with learning PromQL and kind of walk you through all the various aspects of it. There are so many different ways to represent your data. You can do a simple counter. You can do a simple tachometer kind of thing. You can do a graph. You can do histograms, which is, you know, histograms is going in the direction of looking at a graph over time. It's so complex what they're looking at that they're basically bucketing chunks of it. So the first minute is bucketed. The second bucket represents the first bucket plus the second bucket.

16:35And it gives you a different look at your data. You have heat maps. Have you ever watched baseball when they show the pitcher stuff throwing? And what are the pitches that they hit the most? It gets red in the areas where they get the most. you can represent your data or your view of what's going on that way too. You have topologies for your service calls where you're basically showing the network of the services and how the calls are going and the lines get thicker the more calls that go over it. The PromQL gives you the ability to, and some of the tooling embedded right now into the newest Prometheus is really nice where they explain the query.

17:08So you're putting the query together, it has command line completion in there, luckily, helping us find the metrics that are available at the point you're trying to do that. And you can see the result sets. You can dissect them because you're also doing sort of like queries within queries to build up a bigger query. You'll apply things like rate across a query. So to kind of generalize a specific query out over time, maybe you'll only want to look at a small window, only five minutes or only one minute or over 10 minutes. There's so much flexibility in that. And yeah, depending upon what you want to land in.

17:42So generally speaking, say you're a service owner, when you come in there, you have a couple of things that are important to you, probably that it's up. You know, so you'd like to have a green bar when it's up and a red one when it's not, right? Or yellow when it's degrading or things like that. There'll be a threshold when it starts degrading. You're not allowed to get to a certain point and then it's a problem. Things like that are the very high level look. And then when things start going wrong, you have to be able to dig down into it. So if it's a specific service, maybe I want to go look at the traces, see one of those topology kind of graphs and see that, hey, this one is not getting anything.

18:12Click on that and I can go down and look at the trace, and then maybe I can look at the log for that specific thing. And we like to talk an awful lot about observability being the pillars in the old days, where they talked about logs, metrics, traces being three pillars. For quite a few years now, we've been talking about a chronosphere that we think it's aphasis. I think we should speak a business language. It's much more complex than an individual tool. And I like to tell the story that it's just like you're driving around in an old car you just restored. You're really, really proud of this old car, and you're driving along, and all of a sudden the temperature of the engine starts going up and it starts getting kind of weird how it's driving.

18:47You're like, oh, I'm close to the mechanic I use. Let's whip in here quick and see what he says. You pull up and it's getting worse. It sounds bad and you get out and he comes running out with his, he goes, oh, come over here and look at this. And he starts opening up a toolbox and showing you all the tools. Meanwhile, you look outside the window and your car is overheating, catching on fire, and he's in here talking about these great tools he's got. So we like to talk about the phases of observability where you want to know as fast as you can what's wrong. And then you want to be able to triage it as quickly as possible, preferably fix it with remediation.

19:18And you want to be able to go back and look at like root cause analysis and find what the long-term solution is for this thing. And to do that, we don't care whether it's two metrics, one label, three traces and half a log of mine. It doesn't really matter as long as I can get to the answer as quick as possible and fix the problem. And I think that's pretty much modern observability in a nutshell, where Prometheus has a very important role in helping collect, manage, and display these kind of stories. Let's maybe, if we can, make it a little more concrete by going through an example. So let's say, I don't know if we want to use a car example or something like that.

19:57You have a service, you're monitoring. Maybe let's take it from ground up. First, what do you need to configure to get Prometheus starting to track things in there? Is there a thought process around what needs to be tracked or everything or it's there by default? Then how do you design those queries that are going to give you that dashboard that tells you, oh, shoot, my car's on fire. What do I need to do here? And then take it through that route. Right. So for Prometheus itself, there's exporters, for example. So let's say that your service is a node service, Node.js. It's very easy to turn on a node exporter.

20:30It creates an endpoint and generates a bunch of node type of metrics that are pretty standard. You're able to get quite far with just doing that. And what's nice about that is you don't have to redeploy anything. There's no code changes involved. You target the service, you set the exporter on, and you start watching what's going on. Once you start collecting data, you now have all the generic metrics around things like CPU usage, what the memory usage is. I don't know what specific things are in this exporter, but if it was a Java one, it's keep sizes and all that kind of stuff. Everything that you normally would be able to monitor around something like that without having to design it yourself.

21:05So the people that write these exporters and manage these exporters and maintain them are trying to make life as easy as possible for people who just want to kickstart it and see what they like. That's either way how you usually start just so you can see what's available. And then you can start trimming stuff down if it's too much for you. Because one of the things that gets out of control really quickly is it's very easy to collect a lot of data. And if you're doing this in the actual cloud, in and out with your data is water through the pipe is money, you know. And so if you don't have the ability to see it, which is one of the big things that Chronosphere is good at, is using your control plane to see your data coming in and out and tell you when you're not using your metrics.

21:44It turns out that across our customer base anyway, that on average, 60 % of the metrics are not used, that are collected. And that's not a bad thing at this point in time. It's just a big chunk of your bill you don't really want to be paying, right? And so the first thing you start looking to do is how can I trim this down or at least stop it from being stored. You do want to ingest everything because you might need it in the future. But it's not a problem to not pay for storage if you're not using it. If it doesn't show up in a dashboard, it doesn't show up in a query and an alert or anything like that.

22:14No ad hoc queries from the user. Nobody's touching it. What are you doing? So that's kind of what you run into really quickly when you use a standard exporter or a standard library that would just spit out everything. But for us developers, that's a nice way to start, right? And then you say, okay, I only want to know CPU. I only want to know how much memory it's using. I only want to know whatever. And so then you go back and you start instrumenting it using code, going actually in there and saying, I want this, this, and this, and the rest I don't need. Redeploy the thing. There you go. Now we have it trimmed down to just what I want.

22:46And that's the stuff that you're querying to create your dashboards. So I'm saying, I want to see a chart that shows me how much memory consumption is going on in the last five minutes or the last 10 or in the last day or whatever it is. By default, it might be in the last hour, but you can expand that and drill down in it, cut and slice and dice any way you want inside that stuff. So that's kind of the evolution you go through to get to something to make you happy. And trust me, when you start getting somebody carrying the beeper, you're going to start seeing things in there that also include documentation in your dashboard.

23:15So there might even be playbooks or runbooks that they have. when certain things happen, go down this path and do this and call this person, alert that person, look for a feature flag that got changed. Maybe there was a new deployment. Oh, definitely reverse that stuff. That kind of stuff. Yeah, that makes sense. Okay. So just to make sure I'm understanding lifecycle, first, you build this exporter, which you probably can start with just a package off the shelf. That's sending a bunch of data. Prometheus is going to then start tracking that in time series. You look at that and you can start building your dashboard by sending these queries against it.

23:47And I think one of the things you highlighted there that is kind of interesting and might be worth exploring is like what time series enables is querying over ranges. And I did see there looked like there's a core distinction between you can do sort of instantaneous moment in time queries, and you can also do kind of range queries. Maybe we want to look at what differences there are there. You studied hard. Do what I can. So, all right, you build your dashboard on that. Maybe you set up some alerts on that. Question I actually have is, is there any difference between the queries you're doing for dashboard versus alerts does prometheus have native alerting support are you querying that on it's it's a separate binary that you can install but it definitely has an alert manager interesting part about this is and i ran into this quite a bit when i first built the workshop is you tend to forget like it's in memory but it's also writing it to a little file system in your directory there so every time i would like adjust something and restart and things like that.