Observability is your profit center now | Honeycomb’s Christine Yen

0:04Today, we have a very special guest joining us, Christine Yen, the CEO and co-founder of Honeycomb. Christine has been at the forefront of the observability movement since before it was even really called that. And drawing from her early days building analytics at Facebook and more, she spent her career figuring out how to separate signal from noise, which is the biggest battle that I think we're all fighting right now. How do we understand these deeply complex systems that we're creating? So today we're going to dive deep into the state of observability because in an era where AI coding and AI agent is shipping code faster than humans are even really reading it and understanding it, we're in an impossible battle.

0:50And traditional monitoring tools are bursting at the seams. the techniques that we used yesterday might need to be flipped on their head. So we'll explore what this means for developers and the pressing conversations the industry needs to be having right now. But particularly, we're going to explore how the worlds of observability and data science and engineering and product are all colliding into this one big new thing. Christine, welcome to the show. Thank you so much for having me. I'm excited to be here. We're real excited to have you. Honeycomb is a big, long-time friend of the Dev Interrupted show as well.

1:27Your co-founder, Charity, has been on here many a time, even predating me. She's probably been on more episodes than even I, the host, have in some regards. There was a period of time in my early days where that was certainly the case. So it's really great to have you here and get your perspective because I'm a really big fan of all of the thinking and the leadership that's coming out of Honeycomb right now. And I really want to dive into the things that you're seeing and you're understanding from your role as a CEO of this company of really looking at the landscape and how it's been changing kind of like right underneath you.

2:02You're not an AI native company. You were talking about understanding technology before all of this transformation happened. So what is like highlighted right now for you in this big wild world and how it's getting shaped by agents and how you think observability is fitting into that? I think one of the things that strikes me, you started this off by touching on data science. And when I think about observability, there's many definitions out there. I just think of it as trying to make sense of what your systems are doing by using data, right? Which is a very broad definition. And when you think about what mechanically that requires, it does sound like a lot of very parallel industries, business intelligence, data science, product analytics.

2:54There's all these adjacent things where we're trying to do very similar things. And I've always been surprised historically how high and thick the walls between these adjacent disciplines tend to be, how little idea sharing there tends to be. And I'm seeing, and I imagine we can talk a little bit more about that later if you want, but I'm seeing a version of that happening with talking about agents and agentic development, whether it's I have Claude helping me write code or I'm trying to build my customer facing agents. Like, people are really, really trying to talk about this as if it's something entirely new and entirely different.

3:37And we leave all of our traditional software things over here and we're doing all of our AI things over there. And maybe it is because we've been doing this since before ChatGPT. But I look at it and I'm like, guys, it's software. Like, it's weird software. It's non-deterministic software. It's more autonomous software than we're used to. And, you know, there's a lot of the shape of how we serve that software has changed, right? We don't have clean request response cycles anymore where, you know, you have clients talking to servers and no states shared in between. The shape has changed, but it is software.

4:17And it's interesting to see different organizations sort of working through that idea and figuring out for themselves which ideas to carry over from how they serve software systems, how they think about reliability of that software, how they think about observability, and where they're like, no, truly our practices from the past do not serve us in this new world. And we need to take this sort of step change evolution in our practices. That's definitely something we've heard a lot from leaders on the show in terms of really what AI does is it makes more dramatic and more visible the problems in an organization that are deeply maybe like around those rituals and those silos and those things that cause the information to not flow correctly.

5:05And so really what it does is it highlights issues that were always there, makes those issues way more stressed and way more strained. And what you see in these teams are they come together and in this like, you know, the rubber is hitting the road with AI as fast as possible. They're building a race car, but it's falling apart on the racetrack. Why is that? And they're, for the first time, opening up the hood and fixing that one particular thing that had been, like, really a huge crutch that the organization had found workarounds for or wasn't the biggest obstacle of the time. But now it was what was causing the race car to fall apart.

5:41So because of that, like, you got, like, people shining light on things that always should have mattered. Like what you're saying, like, the underlying discipline didn't change. just the things that we need to carry forward might need to be in a different arrangement. You might need to repack your suitcase, but you're still taking the same thing, you know? I think that piece is super interesting, right? One of my level moments years ago now with AI was realizing that I was at some event and someone was talking about using AI to generate bedtime stories for their kid. and how their six-year-old was actually really good at being like, I want a bedtime story about dragons and witches and my left sock, and I want them to go on an adventure together, right?

6:33And hearing them talk about this, like, what has been hard about communicating with humans forever? Well, it's been about communicating expectations and assumptions up front, being explicit about what you need and your intent. And when I think about engineering practices, when I think about observability, there is something really exciting. Honestly, when I'm talking to a customer about observability and their practices, one of the most valuable conversations I can have with them and then they can go have with their teams are the conversations outside the tool themselves. It's what does delivering great service to our customers look like?

7:10What are the signals that really matter to the business where we should page an engineer if something looks off? It's probably not every little, you know, it's probably not CPU hits 70 % utilization, CPU hits 75, CPU hits 90, CPU hits 95. Okay, let's step away from that pattern. It probably is conversations like if we are an e-commerce platform and our checkout rate drops and sustains that drop for some amount of time, right? These conversations that are at a higher level of what matters and what is good are ones that I see teams having all the time now because AI forces those conversations up front.

7:56And it is necessary or important for using AI code generation. And it is increasingly, I think, should be, it both should and is happening, part of using tools like Honeycomb. And so there's, you know, AI taketh, AI one hand taketh, one hand giveth. Giveth. I don't know. There's some really fun shifts happening in how we use our tools and how we do what we are here to do, which is deliver great service to our customers. exactly really what the technology has done for many teams is allowed them to close a loop that was always open where there were people and processes and idea of this is what our north star this is what we're aligned to we're going to push and we're all going to do really hard and the product people are going to do the product stuff and the design people are going to do the design stuff and we're all going to move this wagon along somehow right but at the end we're all going to then evaluate it and understand it and be like where did we land what's this good Was this bad?

8:59Go through our goals as a team. Now, because all of that can also flow into a system, a probabilistic system that can do a lot of high-impact things at scale, because all of that information can flow into that same system. But that same system needs to be aligned to rewards and what you're saying, like objectives. And like this is, you know, this machine that can do 100x, 1000x more of this analysis impact than maybe the individual running it could. They now need to know where to point that bazooka very, very accurately in terms of their time and their token cost. So it really causes like what you're calling like this transformation, you know, that happens within companies where they have to have those conversations a lot sooner.

9:50and then they're better for it because everyone is now like, oh, wow, we're really aligned. And me and all of my agents are also aligned and I don't have to exist in this fuzzy space of trying to convince them otherwise. And so that's really what like the big unlock I think that's happening. And observability is definitely the key to closing the loop. Observability is really what is handing the information and allowing that loop to be closed. And so that's why, like, for me as a go-to-market engineer, like I build agents, I work with them constantly, and they're only made possible by observability.

10:25Observability is a key underpinning for how they work because I won't even begin to build them until people can tell me what they have to do. And so, and then you work backwards from, you know, observability from there. So it's like there's a really great advantage that you get to be in to understand and how all of these teams are handing that information across that loop. Well, thank you for setting me up so well. Yeah, I mean, people love to talk about AI as like, again, it's this AI problem. It's so new. It's so different. And yes, but for that verification piece, for what AI needs to be able to do anything, you know, there are a lot of people who are like, AIS series will be the second coming.

11:07and I never need to do this work again. What do those AIS agents need? Well, they need data. Everything falls down to a data problem. Everything falls down to where are those agents getting the information about what is normal, what is not normal, what's happening, what should be happening. And all of that is driven by observability tooling. And I think it is both really exciting to just like crack open this world of what is possible and what can we automate and what can we make faster with humans driving and all of this. And also, not to go back to my whole it's all just software, but reducing it to a data problem means that there's like two things that really need to be true of the thing that's providing the data to your agents.

11:54And it's the thing needs to be really fast, especially at scale as we produce more and more telemetry than ever before. and it needs to be able to support, it needs to be able to provide the agent as much context as the agent can possibly use to figure out where to look in the system and what to investigate. And that context means really rich schemas or really rich data formats so that you can provide it information about, you know, if you're that e-commerce platform, how your checkout process is doing. Exactly. It's like all of those things become those little bridges that are closing the loop and they all are existing within this observability space that you work within.

12:40And there's something that also that is happening for these teams. This really kind of, it has a side effect that's also then relevant for you in the observability world. And that's that because, you know, AI agents are doing a lot on the knowledge working side. AI coding, AI coding agents, AI in the CLI, in the IDE, they are also churning out huge amounts of work, both in like durable, in the cloud environments that are running 24-7 at scale. We're all seeing these autonomous like coding factories and groups that are doing this at scale, but then also on the machine. So there's more code than ever before.

13:17GitHub is literally bursting at the seams and they're posting a report about it all the time. And if you look at the charts of every Monday morning of everyone opening up their laptop and starting clawed at 8 a.m. Pacific time, it's just like staggering. And so we're in this world where there's so much code that that code base itself kind of becomes not much of a source of truth. And this is something that I heard you say when you were recently at HumanX and you spoke there briefly about this. And there was this clip on YouTube that we'll include in the show notes as well. So our listeners can go check it out.

13:49We've called out this like really great observation that because there's so much more code, like that code is not like as important as what's the impact of that code. And that flip has been like really hard to do testing around to understand. But like, how do we build it and understand our software's impact? So I wanted to dive more into like what you've thought about that since you've said it. I stand by it. But early on in talking about observability, someone started talking about the mental model that we have of our systems. And I was like, that is a great phrase. because anytime we are trying to map our code base to what is happening in reality, where it's passing through our mental model of like, I'm going to read the code, put it in my head, and then try to figure out, you know, is what I'm seeing in production aligned with what it's supposed to be doing or is it different?

14:46And that was always hard. And that was always something that I think as systems got more complicated, like just... left, increasingly left at the realm of reality. But now, as you say, or I guess, as I said, the rate of change and the volume of code is just so much that any mental model we try to build is going to be immediately obsoleted. There's a blog post that I loved that I still have open in a tab by Chad Fowler titled, Production is a Compiler Input. And I love that sentiment, right? That And as we're moving so fast, part of how we maybe should have always worked, but now certainly should be working, is taking these production signals and having that be a part of how the agent is making decisions about how to write code going forward.

15:39What real world conditions are being seen and should be factored into making sure that the next set of changes that are pushed out ensure a great experience for your users. Yeah, that's the challenge right now is compressing that gap that bridging that fuzzy context. Fuzzy context is a lot of what I call this, like where humans like you and I, like we go to meetings or we have chats and tasks and threads and emails and a lot of contexts that like you could argue in all sorts of different ways that an AI could access. But could they sophisticatedly understand the relations of those, the timing of them?

16:17This becomes much more of a challenge. So because the consumption of that fuzzy context is really difficult, it becomes like this. And because unlocking that is the key to scaling an engineer's impact or a team's impact beyond an individual and to orchestrate further with agents. because of that, the observability tool also then becomes like the learning and teaching mechanism by which the human and the agent figure out how to do that dance. I'm curious like how you've seen, I know at Honeycomb that y 'all are very agentic and you exist in an agentic world, like how you've seen the abilities of working with observability to actually then transfer directly into like engineering competence around being able to orchestrate and work with the tools.

17:05Yeah. I think that we, I think I take pride in is how thoughtful the Honeycomb team is about where the human goes in the loop, how to not attribute skills, how to, you know, we have some really great processes and actually blog posts about how we do things like incident reviews to really focus on learning rather than, you know, slap a bandaid on it, move on. And I think that that really carries through how we're trying to build Honeycomb out as a product for our customers. I think that there, I think I have liked to say since the beginning is that debugging is inherently a collaborative process, even if it is just present you, past you, and future you.

17:50Right? Because past you wrote something, made some decisions that present you is trying to figure out and you're trying to like mitigate, you know, the problem for a particular day. You're hedging for future you. Yeah. And there's this question of, again, I used this word in this talk track before AI made it a thing. But those different versions of you are passing context between each of you. And you're trying to figure out, okay, how did I debug this last time? And what signals are interesting? I think those questions, again, even talking in a pre-AI world, say you're onboarding a new team member.

18:24knowledge, like there's such a role that observability tools can play in that sort of knowledge transfer. The naive approach of this is like, hey, you're the new engineer on the team, here's my dashboard that I use, you should use it too. But I think that there's a lot more, and this is something that Honeycomb's always tried to bake into the fabric of our, to mix metaphors, to weave into the fabric of our product, this knowledge of other people on your team and sort of the trails that they've worn into the grass and how to surface those sort of investigative cues. Now in an agentic world that we have these incredible building blocks that can try to do some of these investigations for us, it becomes like a really exciting thing to explore.

19:17again like a naive approach of this is oh document everything that your humans have ever done put it in a run book and then feed it to the agent to follow like that's possible we think much more interesting is like okay when two humans or you know when you're actively in an incident or you're in some sort of investigation incident implies an urgency and severity that I don't think applies to everything that's interesting worth investigating anyway so say you're in an investigation how can agents connect dots that you might not have been able to? How can agents help you explore multiple hypotheses at a time?

19:52How can, and then how can a human teammate look at what your agents are doing to maybe spot trends or provide some context? What if they bring their own agent? What does that look like? What does the collaboration surface then become if you're all, you know, does it come out of Slack? is Slack the primary collaboration surface. By the time this podcast is released, we will have announced some pretty interesting points of view on what that surface area should look like and what that flow should look like. And it's sort of just the beginning, but it is off the back of years and years and years of thinking about this collaboration.

20:34And I continue to be bullish on humans. I think that humans are always going to bring some context and judgment and lateral thinking that LLMs aren't going to be capable of. But LLMs certainly raise the floor. LLMs certainly help write down and document things in a way that a tired version of you can at least come back later and reference. And I think it is going to be really interesting to see the range of solutions out there over the next few years as vendors explore the surface area of where to encapsulate and shield a user from. I see a lot of vendors out there making magical promises to their users about never having to do X, Y, Z again, and where to make those boundaries more porous because there is value to building up knowledge and building up skills in the humans and humans learning how to use these tools maximally effectively.

21:37Right. There's kind of like a - Totally ramble, but I love this area. No, I mean, I love that. That's really, that's okay. So that's like what you just hit on the very end there. That's like a lot of the danger and the glossing over of just kind of like mainstream kind of consumerism, I think around AI, like, oh, it's this black box that does these things for you. And it's just so great. And you never have to worry about this again, but that's not really the case. It's an opportunity to sharpen a really like specific and personalized tool, but that's going to take work and that's going to take understanding what you even want out of it in the first place because you can't offload that level of engagement in the problem.

22:15And so I loved how you've called out of it enables this collaboration. That's what observability does rather it enables this collaboration between the developers, between their agents, between the context. It allows this transfer that is really crucial that not a lot of other tooling allows us to do. This transfer of context. Yeah, it's like how you orient yourself in a system. Yeah, exactly. Any observability tool, you're coming in, you're like, what does normal look like? What does normal look like? The collaboration piece is not a thing that all observability tools try to answer, and that's something.

22:48No, no, the collaboration piece is not. Some tools are definitely going to be more focused on as much signal noise for analysis as possible. It's not really digging into the impact, Did this hit this objective that we had around it? That's much more subjective. And that's kind of like it becomes its own evolved domain of that, right? And you're kind of, you're hitting at like something I wanted to ask about kind of like in this world where there are all of these signals. And you have these amazing abilities to transfer context and skills up between the agents and down into the, or up into the humans and down into the agents and sideways.

23:29You end up in this world where there's a lot of noise. There's a lot of data. There's a lot of information. And for me as a builder, as a worker, as a knowledge worker, as somebody in the space, there's a lot of signals. So then it becomes like, how do I then cut through that noise and find the highest impact things that I can leverage, that the people around me can leverage? Or how do I just keep it all from drowning out? So I'm curious, from your perspective, how you've seen observability where it is more of a fire hose, how people can then find the learnable moment from it. Yeah. I mean, I think you're asking the, how do I know what matters?

24:14Yes. Right. And that's, it's true. When you're like in it, when you're just like handed a giant pile of telemetry and you have to go sift through it, that problem in the abstract sounds terrifying. And for most people, the challenge of investigating a system is not a single contained moment in time. It is a longer period of time where you have some influence over the system. You have some ability to say, hey, I'm going to admit this telemetry because this telemetry will be useful for investigating. You have the ability to do an investigation and then you have the ability to feed it back. again in the naive case this has resulted in a lot of people building it you know emitting some telemetry building a dashboard having an incident having as a post-mortem build more dashboards and then then you end up in a sort of vicious cycle where you just have too many dashboards i always try to reframe it for our users and customers about having a virtuous cycle around improving your telemetry instead of building dashboards and artifacts right dashboards are I think dashboards are, frankly, also going to become obsolete in an AI world.

25:24If you interact with most of these tools via their MCPs and your own agent, who cares what dashboards are pre-canned? You're always going to ask possible questions. Which, again, points it back to a telemetry problem. Cool. Now that we're all looking at the telemetry, what fields can we add? How will we know what matters? Well, if you start thinking about it from a data point of view and you think about what does it mean to deliver great software, your boss or your boss's boss is probably going to start saying something like, well, our high -priority customers need to have great experience in the platform.

26:00Great. How do you define high-priority customers? How do we capture those bits and store it in our telemetry? What does it mean for them to have a great experience? What parts of the product are really important to be low error, low latency things? What are the contracts that have been implicit in how we're building our system that might not be expressed as a simple red metric? An example with Honeycomb, we are at our core a data processing system. um we an internal metric that we look at really closely and have always looked at really closely is how long does it take between the time that a some some blob of telemetry hits our api to when it is queryable that's something that touches across the whole system that is something that directly impacts your user experience if you're trying to query something that you just sent in and you want a really real-time response.

26:59These questions around what matters, then drive how do we capture what matters, store it to query, and then build this flywheel of making sure that we can capture all the bits that might be useful. One last caveat here. I imagine a lot of your listeners right now are rolling their eyes. Ah, another observability vendor telling me to capture all the data that serves their business model so well. You know, I've experienced, I've tried to throw, I've accidentally thrown an IP address into my metrics and seen things explode and had people get mad at me. Like, you're not going to catch me again, Christine.

27:39And for those folks, if you're feeling that, I really encourage you to interrogate how that observability vendor stores your data because, again, to reduce everything to a data problem, you know, for an engineering audience, it is not so hard to imagine that any data tool is only as good as the data store. And we are really in the middle of, I think, sort of an inevitable transition from the previous generation of ways that we used to store telemetry data in like a log store over here and a metric store over here and a trace store over here. towards sort of the next generation of columnar stores that are meant for really fast analysis, that are meant for you to send lots of metadata that you might need in a way that is really cost-effective in Honeycomb, effectively free.

28:39And this is the shift from the tools that have those silos that rely on pre-ergregation, cardinality limits that you've run into in in practice, and then you read about them in documentation after the fact when you're trying to figure out why your cluster fell over. A lot of the problems in the previous generation around data fidelity and data quality and context are things that this next generation, Honeycomb included, have really taken into account and built for. So for anyone in the audience rolling your eyes, feeling deeply skeptical about observability vendors telling you to capture richer data, take another look because things are changing.

29:22Things have changed. At the top there, you gave us a really powerful strategy for when you're drowning in data and signals and you're trying to understand what am I supposed to be learning from this? You need to interrogate. You need to flip the script actually. And you need to start asking questions because by asking all of these uncomfortable investigative and really deeply like core to the consumer and the product and the audience of your product. Really asking those questions is the only way that you're going to then figure out what is the signal that I should be measuring to finally find something in the sea of noise that you and other people can align to that then you can turn into, you know, nom nom nom delicious agent food and feed to them and get them on the same page.