New Relic and Agentic DevOps with Nic Benders

0:00Observability emerged from the need to understand complex software systems and involves tracking metrics, logs, and traces so engineers can detect and diagnose problems before they affect users. However, modern applications often encompass hundreds of services, containers, and dependencies, generating more observability data than dashboards and alerts alone can effectively surface. New Relic is a leading observability platform with a history that spans the full arc of modern software operations. Today, they are working to apply AI to move observability beyond passive monitoring toward active intelligence, where systems can surface what matters, reduce alert noise, and ultimately take autonomous action before problems reach engineers or users.

0:47Nick Benders is the Chief Technology Strategist at New Relic, where he has worked for 16 years. In this episode, Nick joins Lee Acheson to discuss the evolution of observability from dashboards and alerts to AI-driven intelligence, how LLMs and statistical tools work together to surface meaningful signals from massive datasets, the emerging challenge of observing AI systems themselves, and what the rise of AI means for the future of software engineering as a profession. This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization.

1:29His best-selling book, Architecting for Scale, is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments. Lee is the host of his podcast, Modern Digital Business, produced for people looking to build and grow their digital business. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com and see all his content at leeatchison.com.

2:08AI and observability. How exactly do they work together? My guest today is Nick Benders. Nick is the chief technology strategist for New Relic, one of the major observability platforms that is now focused on AI. And Nick is also a personal friend of mine. So Nick, welcome to Software Engineering Daily. Thanks Lee. It's great to be here. Now you and I go back a long time from New Relic days, early in the New Relic days. As a matter of fact, we've just spent a little bit of time talking about some of those earlier days. But can you catch listeners up on what you've been doing since those early days in New Relic and what your role as chief technology strategist really involves?

2:48Absolutely. So when I think about New Relic's journey, and really in some ways the industry's journey, back when we started at New Relic, we were very much solidly in this instrumentation era. Like the thing that we sat down at our desks and tried to figure out every day was how can we instrument more of the systems that matter to people? Oh, well, we started with Ruby. How do we get Java? How do we get.NET? How do we get Python? How do we get into the browser or onto mobile apps or add a new library? And pretty soon, we were instrumenting so many things that there was more data than we could deal with.

3:24And so it moved from this instrumentation era into this data platform era. And for New Relic, that shift was around 2013, 2014, when we introduced NRDB. NRDB gave people a way to ask questions of a system that you didn't know you needed to ask. So all the data goes into it, and then after the fact you're like, oh, where are my slow queries coming from? Oh, well, that's mostly a test system. Exclude that test system. Where are the rest of the slow queries coming from? Can you break that out by country? And so this type of interactive questioning system that powers dashboards, it powers just kind of an interactive data explorer, it powers alerts, there's all these things you can do with a data platform.

4:09But that was over 10 years ago now and what we've seen is the ability to have all the data and to put it all in a place so you can ask any question is no longer enough because you have so much data, you don't even know what to ask from it. And so instead of just being about the ability to ask something or to make a dashboard out of anything, you need a tool that tells you what are the questions you want to ask. It tells you what are the things you want to look at. And so that's that shift from the data platform era into this intelligence era. And intelligence, everybody jumps on immediately. You're like, oh, it's AI.

4:49I'm like, yes, AI is a piece of intelligence, but it's also about product design. It's about the way that we use something, having those built-in opinions, those flows. Because when somebody sits down at a tool, they don't want to just see a prompt and say, oh, I can dashboard anything. Great. They want answers. They want to know what's important in their system. And so that's that intelligence shift. That's the era we're in now. I'm going to talk about this a little bit later, but also the intelligence era won't last forever. And it may already be bringing to a close as we need to move into an action era as an industry and for New Relic.

5:30And so New Relic's journey since those early days has been getting into each of these, kind of pioneering it, figuring out what has to be done, and then asking the question of what has to be done next. As the chief technology strategist, that's where I come in. I've been with the company now for 16 years, just working with every piece of the system. And I've been using observability tools for 30 years now. And back from when we used to call it monitoring, it was just ping. Right. And the question isn't real. Where are we as much as where are we going? And that's my job. Got it. Got it. That makes sense.

6:11So I was going to ask you, what's the one biggest change that occurred in your mission? But it sounds like there's really two big changes. There's a change from instrumentation to data and then from data to intelligence. And are those really the major changes in New Relic that have occurred over time? I mean, in New Relic product. Obviously, there's a million things that go on in any tech company. But I think, yeah, I think that realization originally, back a decade ago, that the secret was to have a strong data platform and that you had to be able to ask anything of your data platform at any scale.

6:50And then more recently, just a few years ago, that realization that we've built this industry, like I said, I've been using these tools for decades, that dashboards that we build today are fancier, they're easier to build, like they monitor different things. But fundamentally, they're not that different than dashboards I built in the 90s, where you sat down and you're like, I don't know, I'm going to put a dashboard together. It's going to monitor, here's how much free memory we have. Here's my CPU usage. Here's my network usage. I did it in TKTCL then instead of just doing it in NURQL in a browser.

7:28The difference is instead of having three or four graphs, we have 300 or 400 graphs. Yeah, oh yeah, right? And people want to build these charts. But there's no widget I can make that's small enough to actually watch everything in the system. And when that struck us as a company, we realized dashboarding and alerting as the way you do observability has reached its conclusion. You can't go forwards anymore with a, I'm going to make fancier dashboards or tinier widgets, or I'm going to give people more tools to set alerts. Nobody wants to set alerts. Nobody wants to build dashboards. They want to know answers.

8:03They want to sit down and know what's going on in their system. And if dashboards and alerts are a tool, you can use that tool. But it's not really people's objective. And that, to me, that's that change that's occurred to us a couple years ago. And I think you're going to see all across the industry especially with these AI-powered capabilities. So I remember back in the early days in New Relic, in the early NRDB days in particular, we talked a lot about machine learning and the value of alerts generated from some magic machine learning algorithm that did magic things and said, oh, this is a problem because such and such pattern exists.

8:43And we used that as, we were very clear. I remember standing up at conference after conference saying, this isn't AI. This is not AI. This is something else. I forgot exactly the term he used. He had a term that he used for it. But basically, it was machine learning is really all it was. So what's changed? Obviously, machine learning isn't the core of what you do anymore from an intelligence standpoint. What really changed? Why did this change occur? Yeah, that's a good one. I like to think of those techniques as falling into three buckets. Obviously, it's all computers, it's all just software.

9:25There's no magic here. But when we look at simple things like you're going to do some type of component analysis on a signal, or you're looking for baseline deviations, you've got mathematical formulas, you apply them. The parameters are static, even if the math is fancy. So I said the first category of this is like, this isn't AI, it's just math. This is just statistics. You can look it up, you can execute it on it. And you'll see a lot of functionality that's like that across every product that tells you when there's a significant deviation or things like that. Then there's machine learning where we're taking those parameters and instead we're defining like hyperparameters.

10:04We're saying, oh, hey, could you tune these baseline alerts so that there's only one alert per period or kind of make these hyperparameters work out? There's some simple algorithms there. And that made up a lot of what people thought of as ML ops or these AI functionality up until about two, three years ago. And everything just got clobbered by the return of neural nets. Neural nets, obviously, are not a new technology. We've been kicking them around in the CS department since the 1960s. but suddenly they started to work with that transformers architecture and some of these other optimizations and infinite money poured into it.

10:47Now we have neural nets that work. And so I think of the neural net-based systems as being what we currently call AI, although maybe five years from now we'll feel differently. And those are more complicated and less predictable than machine learning-based systems, which have a more constrained set, but they tend to not actually use any neural nets. And those are simpler, but a little bit less predictable than static statistical systems, just like using math. A good product is going to have all three. I think that if your view of AI is, I send everything to OpenAI and they send me back answers, you get a lot out of that.

11:24That's cool. But it's not the right tool for every job. There's a place also for the more traditional machine learning approaches and even for just good old-fashioned statistics. I think that as a company and as an industry, we're exploring where those boundaries exist and how to tie them together. Today, the neural net and especially the foundation APIs, so calling OpenAI, calling Gemini, calling Anthropic, that's the boss. That's the piece of the system that's in charge. But the tools that it has access to in an intelligent system should include lots of traditional ML and statistical tools. So, you know, it's kind of easy to imagine this transformation from the static to the ML to the AI when you think about like an individual stat.

12:15Like I have a server and it's got memory usage, right? And the simple analysis, 80%, right? Above 82%, send me alert. Fine. You hit 80%, you get 50 alerts an hour and you try and deal with the noise. What an ML did then is say, well, 80 % is not the right number. Let's make it 86.2 and we'll leave it there for five hours. And then we'll make it 82.1 and we'll make those adjustments as we go and get rid of more of the noise. And that's great as well too. But what the AI, you know, the LLMs of the world now are trying to do is say, well, you know what, this looks kind of odd. What does this mean for this pattern to occur.

12:56And maybe I should alert on that. These are the growth patterns that you occur. But I think one of the big values that I'm not sure yet we're seeing it yet, but I think could be coming, and I'd like your opinion on this, is the value of LLMs in the breadth of data versus, you know, tell me everything about this piece of data, but instead say, here's a whole ton of data. In my mind, observability, not monitoring, but observability, is really about making a complex system understandable. That's really what you're trying to do. And what's better at making a large complex system easier to understand than an LLM?

13:38That's one of the things they're very, very good at, is summarize and explain what's going on in a large system without a human having to go through and analyze it bit by bit and understanding everything that's going on. I don't see that yet, but I see it coming. What do you see of the role of LLM in this large-scale system understanding model versus just the, yeah, this is anomalous data model? Yeah, I mean, I completely agree with you. I think that that's the key step to move from that data era to the insights era, is that you can't make dashboard widgets small enough to watch everything that's important in your system today.

14:17We have these Kubernetes clusters that have hundreds of nodes. We're running thousands of pods. We have many, many clusters. There's so much data in the system today. And a human can't possibly search through it all, nor can you pre-write ahead of time alert rules or build dashboards ahead of time that will show you everything that matters. The thing that matters might be a signal you've never looked at before, But you could see a major excursion on one particular metric that correlates with this failure on the user side. And you say, ah, that's what I was looking for. So that's the place where you want to combine those statistical tools and the LLM tools.

15:01Searching across huge amounts of data is tricky for LLMs. I'm not going to tokenize a petabyte of data and then feed it into an LLM. I mean, not certainly without making Anthropic even richer than they are. But what I might do is perform a statistical analysis, look for anomalies using well-understood methods, then take those anomalies, take those timing ranges, and send that into my reasoning system and say, hey, are any of these interesting? So the missing pieces to do this, like if you wanted to sit down and we're going to build this this afternoon, and we're going to vibe code up, whatever, our future system.

15:43Like, what do I need? I need lots of data in, but I need structure to the data. I need to understand, well, what metrics are about what systems? How do the systems relate to each other? It has to be all temporally organized so that I know when something happens. But I also need it to be spatially organized. Like, the user is on this browser, it's calling this service. This service relies on these other services. They rely on these infrastructure, they rely on those. Because you want to draw across that graph when you are performing, whether it's like a root cause analysis of a failure, or even a predictive fault where you say, what's the linkage between response time and database?

16:26Like, how do I piece this together? I think that there's a ton of work that can be done there with general purpose LLMs. I think that you need a lot of tooling. when I say tools in this case, I mean like MCP tools to give it that statistical capability. So it's where you need that statistical understanding and the ability to feed relevant context into an LLM. LLMs are fabulous at summarizing and they have huge context limits on some of these systems now. You know, 100 ,000 tokens, million tokens. But the data world we live in is billions. And so we've got to go from the billions down to the hundreds of thousands or thousands in order to make it processable.

17:30the way it is, so agents look in the wrong place and deliver bad outputs. Then you spend time correcting, turn after turn. Unblocked is the context layer your agents are missing. It synthesizes your PRs, docs, slack, and tickets into organizational context that agents actually understand, so they make better plans, write higher quality code, use fewer tokens, and require fewer correction loops. If you are running Claude Code, Cursor, or any agentic workflow, Unblocked is worth a look. Get a free three-week trial at getunblock.com slash sedaily. TurboPuffer is how companies like Anthropic, Cursor, Notion, Atlassian, and Ramp ship their most ambitious search features.

18:13TurboPuffer is a serverless vector and full-text search engine built on object storage. It's up to 95 % cheaper than traditional search databases and just as fast. With TurboPuffer, you can index and search 50 million documents at 10 millisecond P90 query latency for less than$100 per month. Head to turbopuffer.com slash SED to get your first month free. So one of the things that this kind of sounds like to me is like back in the early days of monitoring, the problem was alert fatigue, right? We could get all sorts of alerts and all sorts of things. We could have all sorts of notifications of all sorts of anomalies, but we just got so many of them that we ignored them.

18:55It's essentially what you're, the type of thing you're talking about is now a system that can take all of those alerts and not be fatigued by them and find the patterns that exist and find what's really going on. Is it that simple or is there a lot more to it than that? I think it is occurring at two levels. So, you know, it's funny you said, you know, in the early days, it's still, this is one of the top problems I hear from every customer and from every team I talk to is they said, when something goes wrong, we do a retro and what happens at the end of the retro? Every retro ends kind of the same way.

19:28And it says, oh, well, one of the things that we should do to not repeat this incident is we should establish alerting on monitoring for X, Y, and Z so that we can spot it earlier next time. And by the time you run that forwards for a few years, there's an alert for everything in the universe. And I remember Aaron Bento, who we used to work with, gave a fabulous talk one year on how additional alerts does not improve responsiveness. Because what it does is it trains the users to instead, you get an alert and you're like, I'm going to give this a minute and see if it resolves. And what you've done there is you've delayed your time to response.

20:08So noisy alerts corrode that response time even though teams think it's going to add. So one place that we can use intelligence is we can be consuming those alerts and trying to determine if they're serious or not before signaling a human, right? Great, you can look at that and say, well, what's the things that every on-call engineer knows? If I just see the one blip, I'm like, oh, that's kind of funny. If I get like six alerts from six different systems, I better run to the keyboard. So, okay, an AI can do that, an ML system can do that, it's not rocket science. Another thing we can do is we can actually look at the setup of the alerts.

20:48Are a lot of your alerts useless? Do they always go off at the same time as something else? Or are they never actionable where they always come and go? They're not associated with real downtime. Let's look across the whole system. Let's tune some of those down so we can improve the human configured alerts. But both of those are kind of after the fact. The real root of this is, why do we even have these damn alerts? Like, what are alerts for? We create the alerts because we are afraid that something is broken in the system and we don't know about it. Can we get to the root of that issue instead using these techniques and say, well, when I see an anomaly, I'm going to evaluate it and figure out, is this interesting?

21:30If so, is it actionable automatically? Can I just roll something back and notify the user? Is it actionable, but it needs a human? I should grab a user now. Or is it interesting, but not yet over that threshold and I should kind of like feed it to my accumulator and then wait for a few other signals to see what's going on. I believe that in the near future, when a user sets up their observability solution, which as you correctly point out, should be called an understandability solution because nobody actually wants to observe. You want to understand. When I'm setting up those observability tools, I don't want to go through and have to configure a lot of alerts.

22:08I want to say, here's my signals that matter the most. These are people using the system. These are the source of truth for whether it's working. Walk through it yourself and tell me when something's going to break before it breaks. And sometimes this sounds like really pie in the sky. Oh, you're going to have that HAL 9000 telling you that it's like sensor array is going to fail in two days. But a lot of this is actually really straightforward. It's like the same stuff that we do today as humans. We say, oh, well, here's a CPU threshold that we're going to set before it's a problem. Here's a memory threshold that's going to set before it's a problem.

22:43But then we are responding to those on human scale time and not on machine scale time. And if we can detect it, respond on machine scale, and correct it, it stops looking like an on-call rotation and starts looking like other systems that we consider so boring that they're barely even technology. Like when programs crash and they get restarted automatically. That was like, when did we start doing that in the 90s? when Kubernetes says, oh, well, you know, that node went away, put a pod back up. That's self-healing. But it's just so nuts and bolts, we don't even give it credit anymore. We're like, oh, well, you know, Kubernetes just took care of that.

23:23All of our systems should be like that. So many things that require a runbook today are going to be bridged over into just something that you find out about. Like, you know, when I lose nodes and my pods get rescheduled, I don't get paged, I don't even get emailed. It's just a thing that happens. And I think that a lot of incidents that exist today, they belong in that category. They should be things that I don't get paged. Maybe I get emailed about it. Maybe it's just a thing that shows up in the log. It's like, oh yeah, and then there was a problem and we had to deal with it. Kind of like customer support of the olden days.

23:56Did you try turning it off and back on again? I mean, that's really what we are getting to is the model where literally doing a run book equivalent for a problem is what things should I turn off and restart? This one, this one, and this one. Did that solve the problem? Nope. Let's try this one. Okay. That solved the problem. Done. And that's, you know, the cloud did a lot of that. Kubernetes did a lot of that. Both of them working together really was the core of that change. But that's really the way most of our infrastructure works nowadays, right? Is even our networks, right? The networks causing problems, you know, we'll reboot our network, right?

24:32And I can't imagine doing that in the past, but you can imagine doing that now. So you're basically saying that all can be automated so that it just happens naturally. And AI obviously being a central part about that. I think you mentioned cloud, which is a great example of this. With cloud, especially with these container orchestras like Kubernetes, even system management software, there's so many things that we expect will just take care of themselves. And we were able to do so because those problems became really well-defined. And we said, okay, these are all the things that can happen. So if this happens, then this.

25:11And what are we left with? Our operations engineers, our SREs say, well, the parts of the system that can't be automated are, it's like, I got to pay attention because if we ran out of IPs, I had to get called in because there was an IP blockage. Or there was a certificate issue upstream and I had to go and jostle it. There's a bunch of stuff that's kind of fuzzy. You would never put it in a runbook. If we had to sit down and build a piece of software that looked at every possible case, we would finish out our lives covering a tiny percentage of all the things that can go wrong. But if you can have a reasoning system that has a fuzzy interface between the structured data, and that's what LLMs do, and that's what these neural net systems do, is they say, well, this looks pretty similar to this.

25:59Then we can start grouping problems into things we know how to fix and we just take action on. Things that we believe are interesting, but they're not critical yet. And so we'll just kind of put it on a list for humans. We say, oh, I've created a ticket for you. That certificate's going to expire. And things that are an emergency and do require human intervention because we don't know what to do. And then we're paging people. But we can take, I think, a lot of what people believe today is just unavoidable work and move it into that automatable bucket. And we've redefined because of this what qualifies as toil and say, oh, you know, a lot of things that we thought of as human unique are actually just automatable toil.

26:40Let's go automate it. That's an interesting take because one of the first things that I come to is that means we'll need less human toil in order to maintain our systems. But on the other hand, our systems become more and more complex. And so ultimately the amount of toil that we go through ends up being the same or maybe increasing still. But we're now more dependent on these things to make things happen, which is the normal space. That's the way things work. That's the way things grow and the way things expand. That's really the case, right? AI is going to make this easier for us so we can do harder things really is the way it works.